Compare commits

...

91 Commits

Author SHA1 Message Date
1445043649 84de632b19 chore(scripts): add verify_stage0.sh — systematic Stage 0 verification
Adds a single script that walks the 5 verification layers in order
and emits pass/fail counts at the end:

  static   — boundary scans + full pytest (baseline ≥ 3250 passed,
             ≤ 18 fails matching known caplog flake set) + ruff lint
  paths    — .deer-flow/users/ should be empty (or absent);
             workspaces/{wid}/threads/{tid}/... layout in place;
             exercises `make migrate-paths DRY_RUN=1`
  rds      — PG reachable; alembic at 0003; service_accounts +
             api_keys + external_users tables present;
             idx_api_keys_active partial index has
             "WHERE revoked_at IS NULL" predicate; workspace_id is
             NOT NULL on thread_meta / runs / feedback / run_events;
             UNIQUE(workspace_id, thread_id) on thread_meta
             (requires DATABASE_URL; skipped if unset)
  runtime  — curl /health on the Gateway (requires `make dev`;
             skips downstream e2e if unreachable)
  e2e      — register two users via /api/auth/register, capture each
             session's csrf_token, create one thread per user,
             cross-access GET + DELETE both return 404 (per PR6
             "404 not 403" contract), same-workspace GET returns 200

Each layer is independently runnable: `./verify_stage0.sh paths rds`.
With no args runs all five.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:54:02 +08:00
1445043649 6090b9e5b1 docs(impl): PR8 implementation note + STATUS update (Stage 0 收尾)
Adds `pr8-headless-api-schema.md`:
- Scope summary (3 schema-only tables for Stage 1 headless API base)
- Per-task commit table (T8.1-T8.6)
- Acceptance: 9 new tests + 3 tables auto-created + FK behaviour + partial index DDL pinned
- Architecture decisions (key_prefix global UNIQUE rationale, String scopes not PG text[], external_users workspace_id redundancy, no Repository class until Stage 1, T8.6 reverse invariant)
- File structure index (new vs modified)
- Live smoke commands (RDS \dt + \d+ api_keys for partial index)

Updates STATUS.md:
- One-line status: Stage 0 工程层面收尾 (PR1-PR8 全合)
- 8-PR status table: PR8 row marked merged with commit range and impl note link
- Test baseline: PR8 末 3250 passed + 31 skipped + 18 flake; +163 new tests over Stage 0
- Skipped/deferred 行 加 PR8 RDS live smoke 项
- Next-step suggestion 翻新:6 个 live verification 用户跟进项 + Stage 0 退出 Go/No-Go 工程门已满足 + Stage 1 可启动方向
- PR8 经验回顾段:Inline + 严格 TDD,3 张表互相独立,T8.6 反向 invariant 锁定历史坑

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:18:18 +08:00
1445043649 f803f393d3 test(persistence): PR8 T8.6 — verify all 3 PR8 tables auto-created
Adds tests/test_pr8_metadata_registration.py: opens a fresh SQLite
engine via init_engine() and asserts inspect(conn).get_table_names()
contains service_accounts, api_keys, and external_users. Guards
against an ORM row class being added under deerflow/persistence/* but
accidentally left out of deerflow/persistence/models/__init__.py —
which would leave the table un-provisioned at startup and surface as
a confusing "no such table" later in Stage 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:14:14 +08:00
1445043649 6f806ff4a4 feat(persistence): PR8 T8.5 — ExternalUserRow ORM + 2 tests
Adds deerflow/persistence/external_user/{__init__,model}.py with
ExternalUserRow:
  - id: UUID36 PK
  - workspace_id: FK workspaces ON DELETE CASCADE (redundant with SA's
    workspace_id but stored directly to speed workspace-scoped queries
    that span multiple SAs)
  - service_account_id: FK service_accounts ON DELETE CASCADE
  - external_id: String(128) — caller-supplied X-External-User-Id
  - display_name: String(128) nullable (admin UI only, not auth-relevant)
  - metadata_json: JSON nullable=False default {} — plan tier / region /
    custom tags
  - created_at / last_seen_at (UTC)
  - UniqueConstraint (service_account_id, external_id)
    name=uq_external_users_sa_external — the same external_id may be
    reused under a different SA, but is upsert-unique under a single SA

T8.5 tests:
  - test_unique_service_account_id_plus_external_id: second row with
    same (SA, external_id) raises IntegrityError
  - test_cascade_on_service_account_delete: deleting parent SA removes
    all external_users rows under it

Registered in deerflow/persistence/models/__init__.py — all three PR8
tables (service_accounts / api_keys / external_users) are now wired
into Base.metadata.create_all().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:12:43 +08:00
1445043649 52e9999a61 feat(persistence): PR8 T8.4 — ApiKeyRow ORM + 3 tests
Adds deerflow/persistence/api_key/{__init__,model}.py with ApiKeyRow:
  - id: UUID36 PK
  - service_account_id: FK service_accounts ON DELETE CASCADE
  - key_prefix: String(16), UNIQUE — global uniqueness preserved even
    after revoke so audit logs never reference an ambiguous prefix
  - key_hash: String(128) sha-256 hex (plaintext returned only once at
    create time)
  - name / scopes / rate_limit_rpm / expires_at / last_used_at /
    revoked_at — all nullable or default-providing
  - created_at: UTC
  - Index idx_api_keys_sa (service_account_id) — list keys for an SA
  - Index idx_api_keys_active (key_prefix) WHERE revoked_at IS NULL —
    partial index, dual-dialect via sqlite_where + postgresql_where,
    shrinks the hot-path lookup index by excluding revoked keys

T8.4 tests:
  - test_unique_key_prefix_enforced: column-level UNIQUE blocks two
    rows from sharing key_prefix (active or revoked alike)
  - test_active_index_declares_both_dialect_where_clauses: schema
    introspection confirms idx_api_keys_active has both
    `dialect_options.sqlite.where` and `dialect_options.postgresql.where`
    set to `revoked_at IS NULL` — guards against accidental loss of the
    dual-driver hint when the Index is edited later
  - test_cascade_on_service_account_delete: deleting the parent SA
    removes all api_keys rows

Registered in deerflow/persistence/models/__init__.py.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:10:32 +08:00
1445043649 bb7289781e test(persistence): PR8 T8.2 + T8.3 — ServiceAccount cascade / restrict
T8.2 test_cascade_on_workspace_delete: seed user → workspace → SA,
delete workspace, assert SA row is gone (ondelete CASCADE on
workspace_id FK).

T8.3 test_restrict_on_created_by_user_delete: seed user → workspace
→ SA, then attempt DELETE FROM users WHERE id = created_by, assert
IntegrityError raises and the SA row survives (ondelete RESTRICT on
created_by FK).

SQLite enforces FKs because the engine's connect-listener turns on
`PRAGMA foreign_keys = ON` for every new connection — see engine.py
init_engine().

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:07:16 +08:00
1445043649 1fb07e48e6 feat(persistence): PR8 T8.1 — ServiceAccountRow ORM + insert smoke
Adds deerflow/persistence/service_account/{__init__,model}.py with
ServiceAccountRow:
  - id: UUID36 PK
  - workspace_id: FK workspaces ON DELETE CASCADE
  - name: String(64)
  - role: String(16) default "member" (Stage 0 lone value; Stage 2 RBAC)
  - identity_mode: String(16) default "collapsed" — three states
    "collapsed" / "external_passthrough" / "both"; Stage 1 API key
    auth layer branches on this to decide whether each call writes
    an external_users row
  - status: String(16) default "active"
  - created_by: FK users ON DELETE RESTRICT (must hand off / delete
    SAs before removing their creator)
  - created_at / updated_at (UTC, onupdate)
  - Index idx_service_accounts_workspace (workspace_id, status)

Registers in deerflow/persistence/models/__init__.py so the engine's
side-effect import path picks it up for Base.metadata.create_all().

T8.1 test: insert_smoke writes a row through a session, reads it back,
asserts each column round-trips. SQLite ephemeral DB per test via
tmp_path, no Postgres required at this layer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 14:06:21 +08:00
1445043649 b0bf033f15 docs(impl): PR7 implementation note + STATUS update
Adds `pr7-ci-boundary-scan.md`:
- Scope summary (allowlist + AST scanner + TYPE_CHECKING exemption + self-tests + CLAUDE.md docs)
- Per-task commit table (T7.1-T7.5; T7.4 is the revert-after-prove injection drill — no commit by design)
- Acceptance: RED->GREEN cycle, 9 self-tests, T7.4 inject/revert, 3241 passed + 31 skipped + 18 caplog flake
- Architecture decisions (TYPE_CHECKING exemption vs allowlist inclusion, toml path-list shape, pytest-not-separate-CI-step, scope kept narrow, plan-draft vs ground-truth allowlist)
- File structure index (new vs modified)

Updates STATUS.md:
- One-line status moves to PR7 merged; Stage 0 now only PR8 left
- 8-PR status table: PR7 row marked merged with commit range and impl note link; commit count notes T7.4 has no commit by design
- Test baseline line: PR7 末 3241 + 31 + 18 (PR6 末 3214 + 30 + 17, +27/+1/+1 explained)
- Next-step suggestion switches to PR8 (parallel-eligible since PR4); PR7 inline-mode retro added

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:59:22 +08:00
1445043649 d8b13afc59 docs(backend): PR7 T7.5 — document workspace boundary scan in CLAUDE.md
Extends the existing "Boundary check" section with two new entries:

- tests/test_workspace_boundary.py — AST static scan for direct imports of
  langgraph.checkpoint.* (and the third-party postgres/sqlite packages)
  outside tests/boundary_allowlist.toml. Notes the TYPE_CHECKING exemption
  and points readers to `app.gateway.deps.get_checkpointer` as the
  intended path; documents the "append to allowlist in the same PR"
  contract for new legitimate importers.
- tests/test_workspace_boundary_self.py — self-tests guarding the scanner
  from silent-empty regressions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:54:31 +08:00
1445043649 d0f1877070 test(boundary): PR7 T7.3 — self-tests for the scanner
Adds backend/tests/test_workspace_boundary_self.py (9 cases) to guard
against silent-empty regressions in the boundary scanner: every claim
the scanner makes is reproduced against a synthetic .py snippet written
to tmp_path and fed to `collect_runtime_checkpoint_imports`.

Coverage:
- direct `from langgraph.checkpoint.* import X` -> flagged
- bare `import langgraph.checkpoint.memory` -> flagged
- third-party `langgraph_checkpoint_postgres` / `..._sqlite` -> flagged
- `if TYPE_CHECKING:` block -> skipped (Name form)
- `if typing.TYPE_CHECKING:` block -> skipped (Attribute form)
- nested-under-`if True:` TYPE_CHECKING block -> skipped (parent walk)
- unrelated imports (`os`, `langgraph.graph.state`, `app.gateway.deps`) -> ignored
- string literal containing the target module path -> ignored (not an import node)
- syntax-broken file -> returns [] (parser exception swallowed)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:52:36 +08:00
1445043649 ba30d14041 feat(tests): PR7 T7.2 — AST boundary scanner for langgraph.checkpoint
Adds backend/tests/test_workspace_boundary.py: walks every .py under
backend/ (excluding tests/, docs/, build artefacts), parses with ast,
flags any ImportFrom/Import that targets langgraph.checkpoint.*,
langgraph_checkpoint_postgres or langgraph_checkpoint_sqlite outside the
allowlist loaded from tests/boundary_allowlist.toml.

Skip rules:
- Imports inside `if TYPE_CHECKING:` (or `if typing.TYPE_CHECKING:`)
  blocks are exempt: they do not enter runtime so cannot bypass the
  boundary. Verified against deerflow/agents/factory.py which only uses
  BaseCheckpointSaver as a parameter annotation.

The scanner walks parent pointers up from each import node to detect
the enclosing TYPE_CHECKING guard, rather than checking only direct
parents — handles nested guards correctly.

RED proof (empty allowlist): 14 violations across the 4 ground-truth
runtime importers (threads / async_provider / provider / worker).
GREEN (with this PR's allowlist): single test passes in ~2s.

The collector function `collect_runtime_checkpoint_imports` and
`scan_violations` are kept at module scope so T7.3's self-test can
exercise them on a synthetic temp file in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:50:24 +08:00
1445043649 1a6ccc9aaa feat(tests): PR7 T7.1 — boundary scan allowlist toml
Adds backend/tests/boundary_allowlist.toml that names the four current
legitimate runtime importers of langgraph.checkpoint.*:
  - app/gateway/routers/threads.py (empty_checkpoint for thread init)
  - packages/harness/deerflow/runtime/checkpointer/async_provider.py
  - packages/harness/deerflow/runtime/checkpointer/provider.py
  - packages/harness/deerflow/runtime/runs/worker.py (empty_checkpoint)

The allowlist replaces plan's draft (thread_runs.py / gateway/app.py)
with the ground-truth grep: both of those reach the checkpointer through
`app.gateway.deps.get_checkpointer` DI, so they need not be listed.
factory.py imports BaseCheckpointSaver only under `if TYPE_CHECKING:` —
the scanner (next commit) exempts type-only imports automatically.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-14 13:50:12 +08:00
1445043649 4c0b4fab48 docs(impl): PR6 implementation note + STATUS update
Adds `pr6-routes-paths-workspace.md`:
- Scope summary (routes + repos + Paths + middleware + migration script + lifespan + T5.11)
- Per-task commit table (T5.11 + T6.1..T6.15)
- Acceptance checklist with the 64 net-new test count and 17 flake breakdown
- Architecture decisions (404 not 403, check_access signature, three-state semantics, path shape, fallback workspace)
- Live smoke command list for the migration follow-up
- File structure index (new vs modified)

Updates STATUS.md:
- One-line status moves to PR6 merged, T5.11 cleared as PR6 deliverable
- 8-PR status table: PR6 row marked merged with commit range and impl note link
- Skipped/deferred rows: T5.11 struck as completed, new PR6 T6.15 live-smoke row added
- Next-step suggestion switches to PR7 (CI boundary static scan); PR8 noted parallelisable
2026-05-13 18:09:42 +08:00
1445043649 87ea715c2a feat(persistence): PR6 T5.11 — ORM workspace_id nullable=False
Flip the 4 business ORM models (ThreadMetaRow, RunRow, FeedbackRow,
RunEventRow) to ``workspace_id: Mapped[str]`` with ``nullable=False``.
PR5's alembic 0003 already enforces NOT NULL at the DB layer; this
aligns the ORM-driven ``create_all()`` path (dev / tests) with the same
invariant so a new install ends up at the post-0003 schema without
running alembic.

Test fallout absorbed:

- `tests/conftest.py` autouse seed now produces a fully consistent
  pair: user row (default_workspace_id = test-workspace-autouse) plus
  the workspace itself. The PR5 backfill script's "users without
  default_workspace_id" query no longer picks the fixture up. Insert
  order is user → workspace → UPDATE user, walking around the chicken-
  and-egg FK between `workspaces.owner_id` and `users.default_workspace_id`.
- `tests/test_backfill_workspace_id.py` adds a file-scoped autouse
  fixture that temporarily flips `column.nullable = True` for the four
  business tables (production correctness comes from alembic 0003;
  the script's own job is exactly to fill rows between 0002 and 0003
  so its tests need that transient state to be representable). Its
  `_init_engine` also deletes the autouse seed rows to match the
  "fresh DB" model the tests assume.
- `test_thread_meta_workspace_filter::test_create_workspace_none_bypasses`
  renamed to `test_create_workspace_none_rejected_by_orm` and asserts
  the new IntegrityError on explicit None — write paths can no longer
  bypass workspace scope.
- 5 `test_workspace_context` tests + the auth-middleware reset test
  get `@pytest.mark.no_auto_workspace` so they keep testing the
  unset-contextvar path.
- `test_workspace_repo::test_list_by_user_bypass_returns_all` switches
  to membership assertions instead of strict equality since the
  autouse fixture surfaces under `user_id=None`.

3214 passed, 30 skipped; the remaining 17 are the documented
pre-existing caplog ordering flakes (all pass in isolation).
2026-05-13 18:06:53 +08:00
1445043649 c5c66ccbfc feat(gateway/make): PR6 T6.14 — migrate-paths target + lifespan warning
`app/gateway/app.py:_check_path_migration_pending` runs at lifespan
startup and emits a single WARNING log when `{base_dir}/users/` still
has content — the operator's cue to run the new migration script. The
warning is best-effort (silent on permission errors) so it can never
escalate into a gateway boot failure; reads from the legacy tree keep
working via the `user_id` branch of `Paths.thread_dir` until the
migration runs.

The root Makefile gains a `migrate-paths` target that wraps
`scripts/migrate_paths_to_workspace.py`. Two opt-in env knobs:
- `DRY_RUN=1`           — pass `--dry-run` (preview only, no writes)
- `DEFAULT_WORKSPACE=…`  — claim un-assigned users under this workspace id
   (defaults to `legacy_workspace` to align with PR5's orphan-row bucket).
`help` documents the new target.

3 tests cover the warning's three states (legacy dir with content, dir
missing, dir empty).
2026-05-13 17:50:47 +08:00
1445043649 56f2c8d873 feat(scripts): PR6 T6.12 + T6.13 — migrate_paths_to_workspace.py + tests
`scripts/migrate_paths_to_workspace.py` walks the PR4 per-user filesystem
layout and rewrites it under the PR6 per-workspace dimension:

- `{base}/users/{uid}/threads/{tid}/` → `{base}/workspaces/{wid}/threads/{tid}/`
- `{base}/users/{uid}/memory.json`    → `{base}/workspaces/{wid}/users/{uid}/memory.json`
- `{base}/users/{uid}/agents/{name}/`  → `{base}/workspaces/{wid}/users/{uid}/agents/{name}/`

`{wid}` is read from `users.default_workspace_id` in the local sqlite
DB; users without one fall back to `--default-workspace`, defaulting to
`legacy_workspace` (the same bucket PR5's row backfill uses for orphan
rows so the file-system and database alignments stay consistent).

Pre-existing destinations route the legacy copy to
`{base}/migration-conflicts/workspace-migration/...` for manual review;
empty source dirs are cleaned up. The script is idempotent and supports
`--dry-run` (writes nothing, still logs the full report).

9 tests cover: thread / memory / agent rewrite, dry-run no-write, the
fallback workspace path, conflict diversion, post-migration cleanup of
empty `users/` dirs, and the missing-DB / missing-table degraded
modes.
2026-05-13 17:48:49 +08:00
1445043649 2d3b546bf9 feat(agents): PR6 T6.11 — ThreadDataMiddleware switches to workspace layout
`ThreadDataMiddleware.before_agent` now reads
`get_effective_workspace_id()` and routes the per-thread directory tree
through `Paths.sandbox_*_dir(thread_id, workspace_id=...)`, producing
`{base_dir}/workspaces/{wid}/threads/{tid}/user-data/...`. The legacy
user-id-only layout is no longer written by this middleware; the
migration script in T6.12 will lift any pre-existing `users/{uid}/...`
trees into the new shape.

In no-auth dev mode the contextvar is empty so
`get_effective_workspace_id` returns `"default"` and writes land at
`workspaces/default/...` — the layout invariant ("threads always live
inside a workspace") holds without a real auth setup. `thread_data` now
also exposes `user_id` and `workspace_id` so downstream middlewares
(sandbox, memory, etc.) can read them without re-resolving the
contextvar themselves.

4 new tests cover: contextvar workspace → expected path,
`no_auto_workspace` falls back to `default`, eager mode creates the
right dirs, and the `get_config` fallback still routes through
workspace. Existing 4 thread_data middleware tests stay green.
2026-05-13 17:45:06 +08:00
1445043649 f013fc1a65 feat(paths): PR6 T6.9 + T6.10 — workspace-scoped Paths
`Paths` learns a new top-level filesystem dimension. Precedence in
descending order:

- workspace_id given (PR6+):
  `{base_dir}/workspaces/{wid}/threads/{tid}/...`
  `{base_dir}/workspaces/{wid}/users/{uid}/memory.json`
  `{base_dir}/workspaces/{wid}/users/{uid}/agents/{name}/...`
- user_id only (legacy user-isolation): unchanged
- neither (very legacy): unchanged

API: every thread/sandbox/host/ACP helper now takes
`workspace_id: str | None = None` alongside the existing `user_id`,
plus `Paths.workspace_dir()` and `_validate_workspace_id()` for safe
id substitution. `resolve_virtual_path` likewise routes through the
new precedence. The migration script in T6.12 will walk the old
`users/{uid}/threads/...` tree and rewrite to the new shape.

15 new tests (workspace shape, validation, sandbox dirs, ACP,
user-memory/agents under workspace, traversal defence) pass; 46
existing path tests stay green.
2026-05-13 17:43:51 +08:00
1445043649 a7ecd76e0a test(boundary): PR6 T6.8 — cross-workspace isolation e2e (4 cases)
End-to-end coverage for the workspace boundary:

- GET /api/threads/{tid}    from workspace B → 404
- DELETE /api/threads/{tid} from workspace B → 404
- PATCH /api/threads/{tid}  from workspace B → 404
- Positive control: same workspace → 200

Wires `MemoryThreadMetaStore` (real impl, not a mock) behind a
stub-authed FastAPI app. The check_access call inside
`@require_permission` returns False on cross-workspace and the
decorator converts to 404 — proving the boundary holds at the HTTP
boundary, not just the unit level. Run stream/wait are skipped here
(they spin a background worker); their guard goes through the same
decorator path so unit coverage in `test_require_permission_workspace`
is sufficient.

`_StubAuthMiddleware` gains `override_user_contextvar=True` so
cross-user / cross-workspace tests can drive both contextvars from the
stub. Default stays off — the autouse user fixture continues to own
the contextvar for legacy tests whose routes resolve filesystem paths
via `get_effective_user_id()`.

86 router / boundary tests stay green.
2026-05-13 17:40:56 +08:00
1445043649 f6a922921b test(routers): PR6 T6.7 — POST /api/threads workspace_id integration
`create_thread` in `app/gateway/routers/threads.py` already relies on
the AUTO sentinel default for ``ThreadMetaStore.create``, so once T6.1
landed the workspace_id was implicitly carried from the contextvar.
This commit adds the integration coverage: TestClient + stub auth
middleware + ``MemoryThreadMetaStore`` round-trip showing the persisted
row carries the right ``workspace_id`` under two distinct workspaces.

No production code change — the route wiring was correct, but the
guarantee was previously only proven at the unit level.
2026-05-13 17:35:27 +08:00
1445043649 0456606dc1 test(authz): PR6 T6.6 — @require_permission workspace_id propagation tests
The decorator path was already updated in T6.4 to read
`get_effective_workspace_id()` and forward it as the third positional
to `check_access`. T6.6 closes the loop with dedicated coverage:

- `_StubAuthMiddleware` and `make_authed_test_app` now accept an
  optional `workspace_factory` so router tests can drive the active
  workspace contextvar end-to-end through the FastAPI middleware stack.
- 5 new probe tests assert: cross-workspace → 404 (not 403), same
  workspace → 200, the third positional reaching `check_access`
  carries the contextvar id, no-context tests (marked
  `no_auto_workspace`) fall back to "default", and read-style routes
  (`require_existing=False`) thread workspace_id too.

75 existing router tests (artifacts / runs / threads / uploads /
suggestions) stay green.
2026-05-13 17:33:02 +08:00
1445043649 b4fa3bf12a feat(persistence): PR6 T6.5 — Run/Feedback/RunEvent repos workspace_id
`RunRepository` (put / get / list_by_thread / delete), `FeedbackRepository`
(create / get / list_by_run / list_by_thread / list_by_thread_grouped /
upsert / delete / delete_by_run), and `DbRunEventStore` (put / put_batch /
list_messages / list_events / list_messages_by_run / count_messages /
delete_by_thread / delete_by_run) all accept
`workspace_id: str | None | _AutoSentinel = AUTO`.

- Write paths stamp `workspace_id` from the contextvar (same shape as
  the existing `user_id` stamping). For event writes the soft-read
  `_workspace_id_from_context()` mirrors `_user_id_from_context()` so
  background worker writes without a contextvar leave the column NULL —
  consistent with PR5's nullable-during-backfill stance.
- Read paths get an extra `WHERE workspace_id = :wid` clause when the
  resolved value is not None.

7 new tests (3 RunRepo + 2 Feedback + 2 RunEvent) prove cross-workspace
reads see zero rows. 92 existing run / feedback / event tests stay green.
2026-05-13 17:29:03 +08:00
1445043649 05be7f9ad0 feat(persistence/authz): PR6 T6.4 — check_access takes workspace_id
`ThreadMetaStore.check_access(thread_id, user_id, workspace_id, *,
require_existing)` is now a three-positional method. Cross-workspace is
denied unconditionally — even when the row exists and `user_id` matches
— so the decorator layer can convert the False into a **404** and never
leak the existence of a thread across tenants. Inside the workspace, the
existing legacy semantics still hold (NULL `row.user_id` stays
"shared in workspace", `require_existing` still gates the missing-row
path against ghost-row re-targeting).

`@require_permission(owner_check=True)` in `app/gateway/authz.py` now
reads the active workspace from `get_effective_workspace_id()` (set by
PR4 AuthMiddleware; falls back to "default" in no-auth dev) and passes
it through. The existing 404-not-403 mapping is unchanged.

Existing positional callers in `test_thread_meta_repo.py` and the
permissive mock in `test_threads_router.py` were updated for the new
arity. 58 thread_meta / router / memory tests stay green; 90 auth /
uploads / suggestions tests stay green.
2026-05-13 17:23:38 +08:00
1445043649 28ad6c2b0b feat(persistence): PR6 T6.3 — search/update_*/delete workspace_id sentinel
Every remaining ThreadMetaStore method gains a `workspace_id` keyword
mirroring `user_id`:

- `search()` adds WHERE workspace_id and (for the memory impl) folds it
  into the BaseStore filter dict.
- `update_display_name` / `update_status` / `update_metadata` / `delete`
  no-op if the row lives in a different workspace. The SQL helper
  `_check_ownership()` was widened to do both checks in one pass.
- `MemoryThreadMetaStore._get_owned_record()` likewise takes both ids.

5 new isolation tests prove writes from workspace B against workspace A's
thread are silently dropped (no row mutation when the caller re-reads
from workspace A). 38 existing thread_meta / owner / memory-store tests
still pass.
2026-05-13 17:20:40 +08:00
1445043649 296a4f1950 feat(persistence): PR6 T6.2 — ThreadMetaRepository.get filters by workspace_id
`get()` accepts `workspace_id: str | None | _AutoSentinel = AUTO` and
moves the workspace check into the SQL WHERE so cross-workspace lookups
short-circuit without loading the row. `MemoryThreadMetaStore._get_owned_record`
gets the same filter for parity.

The user_id check stays as a post-load comparison (preserves the existing
shared-row semantics where row.user_id IS NULL means "everyone in this
workspace"). Cross-workspace always returns None, never the row.

3 new tests cover the three states: in-workspace get returns the row,
out-of-workspace get returns None even when user_id matches, explicit
workspace_id=None bypasses (migration / CLI). Existing 22 thread_meta
tests stay green.
2026-05-13 17:19:00 +08:00
1445043649 361e653d37 feat(persistence): PR6 T6.1 — ThreadMetaRepository.create workspace_id sentinel
`create()` now accepts `workspace_id: str | None | _AutoSentinel = AUTO`
on both the SQL and in-memory implementations (and the abstract base).
AUTO resolves via `resolve_workspace_id()` from the workspace contextvar
that PR4 AuthMiddleware sets; explicit None bypasses for migration paths;
explicit str overrides the contextvar.

Test infrastructure:
- conftest gains an autouse `_auto_workspace_context` fixture mirroring
  the existing user fixture. Opt-out via `@pytest.mark.no_auto_workspace`.
- A SQLAlchemy `after_create` listener on `Base.metadata` seeds the
  matching `test-workspace-autouse` + `test-user-autouse` rows whenever
  `init_engine` runs `create_all()`, so the FK from threads_meta to
  workspaces resolves. Alembic migration tests bypass create_all and are
  unaffected, keeping real FK constraints under test.
- `test_thread_meta_workspace_filter.py` covers the three AUTO / explicit
  / None paths.

3 new tests pass; 115 existing thread_meta/run/feedback/run_event/owner
tests stay green.
2026-05-13 17:17:56 +08:00
1445043649 430f4a1132 docs(impl): PR5 implementation note + STATUS update
11 PR5 commits: alembic 0002 (nullable workspace_id + FK + composite
index on 4 business tables) -> backfill_workspace_id.py (3-step
idempotent) -> alembic 0003 (pre-flight refuse + NOT NULL + UNIQUE).
18 new tests (9 alembic 0002+0003 + 9 backfill, including dry-run and
no-users error). 3150 passed + 30 skipped + 16 pre-existing caplog
flake (unchanged from PR4).

T5.11 (ORM nullable=False) deferred to PR6 — flipping the ORM-side
NOT NULL would break 6 INSERT sites whose repository signatures get
workspace_id wiring in PR6 anyway. DB-level invariant is already
enforced by alembic 0003 along the production path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:22:19 +08:00
1445043649 30f2bd0084 test(persistence): alembic 0003 — NOT NULL + UNIQUE coverage
Three SQLite cases + Postgres twins: happy-path upgrade after
populating workspace_id, refuse-with-NULL-leftover raises RuntimeError,
and a UNIQUE(workspace_id, thread_id) IntegrityError reproducer. The
UNIQUE test uses a scratch table because the production schema's
PRIMARY KEY on thread_id would shadow the new UNIQUE index. Index
truthiness asserted instead of bool-equality so SQLite (int 1) and
Postgres (True) both pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:16:24 +08:00
1445043649 73d0b7017b feat(persistence): alembic 0003 — workspace_id NOT NULL + UNIQUE(wid, tid)
Refuses to upgrade if any of the 4 business tables still has NULL
workspace_id rows (pre-flight count then RuntimeError pointing the
operator at scripts/backfill_workspace_id.py). Then op.batch_alter_table
flips each column to NOT NULL and adds the threads_meta
(workspace_id, thread_id) UNIQUE index that PR6 routers will rely on.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:15:07 +08:00
1445043649 8a03abac75 test(persistence): backfill --dry-run end-to-end no-write proof
Snapshots workspace / membership / users.default_workspace_id row
counts before backfill(dry_run=True), runs the orchestrator, asserts
no counts changed. Pins the dry-run report's per-step semantics:
Step 1 counts candidates without populating defaults, so Step 2's
JOIN reports 0; Step 3 picks up all NULL business rows untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:13:16 +08:00
1445043649 def45dd0c6 feat(persistence): backfill Step 3 — orphan rows -> legacy_workspace
_ensure_legacy_workspace creates the nil-UUID anchor (slug=legacy)
owned by the platform admin (or oldest user as fallback). Raises a
clear error if the DB has no users at all so we never silently create
an orphaned workspace. Step 3 UPDATEs each table's remaining
workspace_id IS NULL rows to LEGACY_WORKSPACE_ID. Orchestrator wires
ensure-then-loop between Step 2 and Step 3. 3 new tests: orphan
fan-out, no-users error, end-to-end orchestrator with mixed owned +
orphan rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:12:26 +08:00
1445043649 56f6572086 feat(persistence): backfill Step 2 — UPDATE 4 tables from users.default_workspace_id
Per-table correlated subquery UPDATE (portable across SQLite + Postgres).
Filters workspace_id IS NULL AND user_id IS NOT NULL so already-tagged
rows and orphan rows are skipped. Dry-run mode counts via a JOIN, never
writes. Returns rows-updated for the orchestrator report. 2 new tests:
single-user 4-table fan-out + multi-user isolation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:10:49 +08:00
1445043649 e6bb220979 feat(persistence): backfill Step 1 — workspace per user without default
For each user with NULL default_workspace_id, generate a base slug
from the email, walk past collisions/blacklist via next_available_slug,
create the workspace + owner membership, and set default_workspace_id.
Idempotent: candidates list is filtered by IS NULL, so re-running on a
populated DB is a no-op. 3 new unit tests (creation, idempotence,
blacklist-walker behaviour).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:09:16 +08:00
1445043649 ad322543d0 feat(persistence): backfill_workspace_id.py — script skeleton
argparse with --dry-run, async orchestrator that loops the 4 business
tables, and three step stubs (Step 1: users -> workspace creation;
Step 2: UPDATE tables from users.default_workspace_id; Step 3: orphan
rows -> legacy_workspace anchor at the LOCK'd nil UUID). Step bodies
are filled in T5.5-T5.7 alongside their tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:07:44 +08:00
1445043649 d3361dba59 feat(persistence): 4 business ORM models — nullable workspace_id
ORM-side mirror of alembic 0002. threads_meta gets the matching
(workspace_id, user_id, updated_at) composite index so fresh
create_all() dev databases land on the same shape Alembic produces.
Pre-existing test_alembic_default_workspace_id pins to revision
0001 explicitly since 'head' now includes 0002.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:06:08 +08:00
1445043649 4e26ec9884 test(persistence): alembic 0002 round-trip on SQLite + Postgres
Mirrors test_alembic_default_workspace_id pattern: synchronous bodies
(env.py uses asyncio.run internally), pre-PR5 schema bootstrap with the
minimal columns the migration touches, upgrade/downgrade assertions for
the workspace_id column + FK + composite index on all 4 business tables.
Postgres cases auto-skip when Docker daemon is unreachable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:04:21 +08:00
1445043649 a732697855 feat(persistence): alembic 0002 — business tables nullable workspace_id
ALTER threads_meta / runs / feedback / run_events to add nullable
workspace_id String(36) + FK to workspaces (ON DELETE CASCADE) and a
composite (workspace_id, user_id, updated_at) index on threads_meta for
the workspace-scoped thread-list query. Downgrade reverses all four FKs,
columns, and the index.

Stage 0 PR5 step 1/2; revision 0003 flips NOT NULL after the backfill.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:04:13 +08:00
1445043649 44f84800e9 docs(impl): PR4 implementation note + STATUS update
Records the 14-task PR4 delivery (T4.1-T4.14), final test baseline
(3136 passed / 26 skipped), key design decisions that diverged from
the plan draft (TokenPayload optional fields, ensure_default_workspace
helper shape, slug walker treating blacklist as taken), follow-up
items, and a copy-paste live smoke command list for the user to
verify make-dev end-to-end.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:43:04 +08:00
1445043649 5c7753c0b8 feat(auth): lifespan _ensure_admin_user backfills missing workspace
Pre-PR4 admins have users.default_workspace_id NULL. Without this
backfill they would log in successfully but immediately bounce off the
workspace gate (T4.7) because their token cannot encode a wid. The
lifespan hook now resolves the admin user and calls
ensure_default_workspace (idempotent — no-op if the user already has
a workspace), so the post-upgrade boot makes the admin usable.

Renamed routers.auth._ensure_default_workspace → ensure_default_workspace
to make it importable across modules.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:39:09 +08:00
1445043649 91846a201e feat(auth): /auth/me returns workspaces[] with id/name/slug/role
GET /auth/me now responds with the new UserMeResponse: the existing
user fields plus default_workspace_id and a workspaces list that
joins WorkspaceRepository.list_by_user with the caller's role from
WorkspaceMembershipRepository.list_by_user. Stage 0 every user has
exactly one entry there, but the shape is forward-compatible for
Stage 2 multi-workspace memberships.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:37:29 +08:00
1445043649 634e5119e1 feat(auth): login + change_password re-issue JWTs with wid + role
After PR4 every protected request demands wid. Three JWT-issue paths
must encode it; T4.8/T4.9 covered /initialize and /register, and this
commit closes the remaining two:

- login_local calls _ensure_default_workspace (idempotent — returns
  the existing default_workspace_id when set) so users who pre-date
  PR4 are backfilled at login time, and signs the new cookie with
  wid + role='owner'.
- change_password does the same, ensuring a password change does not
  strip the workspace claim and lock the user out on their next
  request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:35:06 +08:00
1445043649 84701730da feat(auth): /register auto-creates default workspace + owner membership
Mirrors T4.8: the regular registration path now invokes the same
_ensure_default_workspace helper so every new user lands with a
1-person workspace (owner) and a JWT carrying wid + role='owner'.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:32:39 +08:00
1445043649 a657d17995 feat(auth): /initialize auto-creates default workspace + owner membership
initialize_admin now seeds a 1-person workspace immediately after the
admin user is created: WorkspaceRepository.create(name, slug, owner_id)
+ WorkspaceMembershipRepository.add(role='owner') + writes the new
workspace id back to users.default_workspace_id. The session JWT is
re-issued with wid + role='owner' so subsequent requests pass the
T4.7 workspace gate.

Mechanical pieces:
- SQLiteUserRepository row<->user mapping now includes
  default_workspace_id (sql update_user too) so the column persists.
- workspace.sql.SLUG_BLACKLIST is now public (was _SLUG_BLACKLIST) and
  the registration helper treats blacklisted slugs as "taken" so the
  walker steps past reserved names like "admin" instead of crashing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:31:53 +08:00
1445043649 a06e88d58b feat(auth): workspace slug helpers — auto_slug_from_email + next_available_slug
auto_slug_from_email maps an email's local-part to a schema-valid base
slug (lowercase, [+_.]→'-', alphanumeric-and-hyphen, clamp 32 chars,
fallback to user-{token_hex(4)} for pathological inputs).
next_available_slug walks the {base, base-2, base-3, ...} sequence
against a caller-supplied async exists_check until it lands on a free
slot, truncating base when the suffix would push past 32 chars.

Pulled forward of T4.8/T4.9 because both /auth/initialize and
/auth/register need it. Lives in app.gateway.auth (not persistence)
since "email → slug" is a registration-time concern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:27:42 +08:00
1445043649 2145d36744 feat(auth): AuthMiddleware injects workspace ContextVar from JWT wid/role
deps.get_current_user_from_request now stashes the decoded payload on
request.state.auth_payload so AuthMiddleware can populate the
workspace_context ContextVar without a second decode. Reset is paired
in the same try/finally as user_context to keep teardown atomic.

Also adds:
- auth.models.ActiveWorkspace — minimal proxy that satisfies the
  CurrentWorkspace protocol (id + role only).
- auth.models.User.default_workspace_id — surfaces the DB column added
  in T4.4 so the eventual /auth/me payload can reference it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:25:43 +08:00
1445043649 b4bef65079 feat(auth): decode_token rejects legacy 4-field JWTs as WORKSPACE_MISSING
After PR4 every JWT must carry wid (workspace_id). decode_token now
returns the new TokenError.WORKSPACE_MISSING when the signature is
valid but the payload lacks wid; expired tokens still report EXPIRED
first so /auth/refresh logic stays correct. AuthErrorCode gains a
matching WORKSPACE_REQUIRED for middleware to surface to clients.

Updates 13 existing test sites that issued tokens without wid to pass
workspace_id="ws-test" + role="owner", reflecting the new contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:22:10 +08:00
1445043649 54cb94c30f feat(auth): JWT TokenPayload accepts wid + role
TokenPayload gains optional wid (workspace_id) and role claims;
create_access_token accepts them as keyword-only args and only
encodes them when provided. Existing tokens and existing callers
keep working unchanged — the contract that protected requests
must carry wid is enforced by middleware (PR4 T4.6/T4.7), not by
the JWT type system.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:17:55 +08:00
1445043649 54762f491c feat(persistence): UserRow.default_workspace_id
Adds a nullable FK column to mirror alembic 0001. On fresh deployments
metadata.create_all() will create users with this column; existing
deployments rely on the alembic migration to add it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:15:05 +08:00
1445043649 917d8fbeaf test(persistence): alembic 0001 round-trip on SQLite + Postgres
Verifies alembic upgrade 0001 adds users.default_workspace_id (with FK
to workspaces) and that downgrade cleanly removes it. Runs on both
dialects because the migration uses op.batch_alter_table for SQLite
ALTER compatibility.

Tests are intentionally sync — alembic's command layer is sync and
env.py calls asyncio.run(); running under pytest-anyio would deadlock
on the inner event loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:11:11 +08:00
1445043649 8efbb2f9e5 feat(persistence): alembic 0001 — users.default_workspace_id
First Alembic revision for the DeerFlow application schema. Adds a
nullable users.default_workspace_id column with an ON DELETE SET NULL
FK to workspaces(id), so newly-registered users can be routed back to
their default workspace on next login without consulting memberships.

Uses op.batch_alter_table for SQLite ALTER compatibility (sqlite is
still a supported dev fallback).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:08:44 +08:00
1445043649 d98498b705 docs(impl): T4.1 — confirm alembic baseline is not needed
alembic heads/history both empty, versions/ dir empty: 0001 can be the
first revision with down_revision=None. alembic_version table will be
auto-created on first upgrade head. doctor.py needs no new check.

Also records T4 preparation grep findings on _ensure_admin_user current
behavior (open issue #3 resolved).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:07:34 +08:00
1445043649 c70c6594de docs(impl): STATUS — record LOCK sign-off and origin push
Two unblocking events on 2026-05-12: team signed off all 7 irreversible
schema decisions, and docs branch was finally pushed to origin (38 commits,
via SSH-over-443 to bypass local proxy). PR4 is now unblocked.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:02:33 +08:00
1445043649 a592319e3c docs(impl): STATUS update after PR3 merge
PR3  merged 进 docs branch(7 commits, T3.1-T3.10)+ RDS live 验证通过
(11 张表含 workspaces/workspace_memberships,partial unique on owner 索引
建出来)。测试基线 3134 passed + 25 skipped + 0 failed。

下一个 PR4(auth 改造)plan 推荐 Inline 模式(跨 jwt.py/auth_middleware.py/
routers/auth.py 多文件耦合紧)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:34:05 +08:00
1445043649 dda8264057 docs(impl): PR3 implementation note + acceptance checklist
记录 Stage 0 PR3 的实施落点、LOCK 决策、跟进项与 6 commit 列表。

验收:3134 passed + 25 skipped + 0 failed(PR2 末 3087 + 47 新 ws_context/
ws_repo/membership_repo 测试 + 2 PG-only skipped)。RDS live 验证通过:
workspaces / workspace_memberships 表 + partial unique on owner 都建出来。

T3.6 (membership ORM) 提前到 T3.4-T3.5 (workspace repo) 之前,因为
WorkspaceRepository.get/list_by_user 会 JOIN memberships。

Stage 0 PR3 T3.10. PR3 完成。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:18:19 +08:00
1445043649 3313047ff1 test(workspace): partial unique on owner — Postgres twin test
新建 test_workspace_partial_unique_postgres.py,2 个 @pytest.mark.postgres
test,用 PR1 的 testcontainers postgres_url fixture 验:
  - test_partial_unique_on_owner_enforced_on_postgres:第 2 个 owner
    IntegrityError(SQLite twin 在 test_workspace_membership_repo 已覆盖)
  - test_multiple_admins_allowed_on_postgres:多个 admin/member 不触发约束

为什么 PG 单独写:sqlite_where vs postgresql_where 是两份 DDL;SQLAlchemy
能 emit 不代表 PG 真的执行。本测试 pin 两边语义一致。

本地无 docker daemon → SKIPPED;CI workflow backend-postgres-tests 会实跑。

Stage 0 PR3 T3.8。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:15:25 +08:00
1445043649 36ffe2713f feat(persistence): WorkspaceMembershipRepository + 8 unit tests
新建 backend/packages/harness/deerflow/persistence/workspace_membership/sql.py,方法:
  - add(workspace_id, user_id, role, invited_by=None) 校验 role ∈
    {owner, admin, member}(Stage 0 仅写 owner,schema 已支持其余)
  - remove(workspace_id, user_id) → bool(rowcount > 0)
  - list_by_user(user_id) 按 joined_at desc
  - list_by_workspace(workspace_id) 按 joined_at asc
  - get_role(workspace_id, user_id) → str | None
  - change_role(workspace_id, user_id, new_role) → bool

MembershipValidationError 自定义异常(role 非法)。

8 test 覆盖:
  - add → get_role round-trip
  - remove 命中/未命中返回 True/False
  - 同 workspace 第 2 个 owner 触发 IntegrityError(partial unique)
  - 同 workspace 多个 admin 不触发(Stage 2 forward compat)
  - CASCADE: 删 user 自动清理 memberships
  - list_by_user 按 joined_at desc 排序(多 workspace)
  - role 校验拒绝 'viewer'
  - change_role 命中改值 + 未命中返 False

注:owner 转让需要事务内两行原子 swap(先把现 owner 改 admin,再把新 owner
插 owner),放 PR4+ auth router 内做带权限检查的版本;本仓储不暴露
transfer_ownership 方法以保持单一职责。

Stage 0 PR3 T3.7。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:13:52 +08:00
1445043649 a40df03521 feat(persistence): WorkspaceRepository + 23 unit tests
新建 backend/packages/harness/deerflow/persistence/workspace/sql.py,方法:
  - create(name, slug, owner_id, *, workspace_id=None, status='active')
    UUID v4 自动生成;强校验 slug 格式(regex ^[a-z0-9](-?[a-z0-9])*\$ + 3-32
    长度)+ slug 黑名单(25 个保留字)+ status 枚举
  - get(workspace_id, *, user_id=AUTO) JOIN workspace_memberships 做成员校验;
    user_id=None 显式 bypass(迁移/admin)
  - get_by_slug(slug) 不带成员校验(path-based routing 用:先 slug→workspace_id
    再到 route handler 里查成员)
  - list_by_user(*, user_id=AUTO) 列 user 所属所有 workspace
  - update_status / delete platform-admin 操作,不带成员校验

WorkspaceValidationError 自定义异常(slug 格式 / 黑名单 / status)。

23 test 覆盖:
  - CRUD smoke + get_by_slug missing
  - 重复 slug → IntegrityError
  - 8 个 invalid slug pattern(短/长/大写/空格/破折号位置/连续破折号/下划线)
  - 6 个 blacklisted slug
  - status 状态机 + 非法值拒绝
  - delete CASCADE 到 memberships(SQLite FK PRAGMA 已开启)
  - get/list 成员过滤(user-A 看不见 user-B 的 workspace)
  - list user_id=None 显式 bypass

全部在 SQLite ephemeral DB 上跑(< 1s)。partial-unique 双驱动验证留给 T3.8。

Stage 0 PR3 T3.4 + T3.5。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:11:28 +08:00
1445043649 d2d2d29c34 feat(persistence): add WorkspaceMembershipRow ORM model
新建 backend/packages/harness/deerflow/persistence/workspace_membership/
{__init__.py, model.py},注册到 Base.metadata。

WorkspaceMembershipRow schema(来自 workspace-schema-design §2.2 锁定版):
  - workspace_id String(36) FK workspaces.id ON DELETE CASCADE — PK part
  - user_id String(36) FK users.id ON DELETE CASCADE — PK part
  - role String(16) NOT NULL(Stage 0 仅写 'owner',schema 允许 'admin'/'member'
    为 Stage 2 RBAC 准备)
  - invited_by String(36) FK users.id ON DELETE SET NULL nullable(Stage 2
    invitation 流程用)
  - joined_at DateTime(tz=True)

索引:
  - idx_workspace_memberships_user (user_id, workspace_id) — 倒查索引让
    /auth/me 列 user 所属 workspaces 走索引
  - idx_one_owner_per_workspace partial UNIQUE on workspace_id WHERE role='owner'
    —— 一 workspace 严格 1 个 owner;sqlite_where + postgresql_where 双驱动
    并存(实测两边都识别)

Stage 0 PR3 T3.6(提前到 T3.4-T3.5 之前;WorkspaceRepository.list_by_user
等会 JOIN 这张表)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:09:19 +08:00
1445043649 8323bf68d2 feat(persistence): add WorkspaceRow ORM model
新建 backend/packages/harness/deerflow/persistence/workspace/{__init__.py, model.py}
+ 在 persistence.models.__init__ 注册 WorkspaceRow 让 Base.metadata.create_all
能自动建表。

WorkspaceRow schema(来自 workspace-schema-design §2.1 锁定版):
  - id String(36) PK(UUID v4 字符串,与 users.id 对齐)
  - name String(64) NOT NULL(显示名)
  - slug String(32) NOT NULL UNIQUE(URL 标识,正则 ^[a-z0-9](-?[a-z0-9])*$)
  - status String(16) default 'active'(active/suspended/deleted)
  - owner_id String(36) FK users.id ON DELETE RESTRICT(删 owner 时阻拦,
    必须先转让所有权)
  - created_at / updated_at DateTime(tz=True)

每列 + 表都带中文 comment(沿用 commit 9ff79055 的 convention)。

Stage 0 PR3 T3.3。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:07:30 +08:00
1445043649 f63089aea8 feat(runtime): add workspace_context module + tests
新建 backend/packages/harness/deerflow/runtime/workspace_context.py,仿
user_context.py 的 API 形态:

  - CurrentWorkspace Protocol(要求 .id: str + .role: str)
  - _current_workspace ContextVar + set/reset/get/require
  - DEFAULT_WORKSPACE_ID = "default" + get_effective_workspace_id(fallback
    友好,不抛错;用于文件系统路径)
  - AUTO 哨兵 + resolve_workspace_id 三态(AUTO/str/None)

与 user_context 的区别:CurrentWorkspace 额外要求 .role 字段,让 Protocol 同时
约束"workspace 是哪个"和"caller 在该 workspace 内的角色"(Stage 0 只见 'owner',
Stage 2 RBAC 打开 admin/member)。

backend/tests/test_workspace_context.py 16 个 test,覆盖:
  - 4 个 set/reset/require 行为
  - 3 个 Protocol structural check(接受 .id+.role / 拒少 .role / 拒少 .id)
  - 4 个 get_effective_workspace_id(含 UUID → str 强转)
  - 5 个 resolve_workspace_id 三态(AUTO/AUTO 无 ctx 抛错/explicit str/explicit
    None/AUTO 强转 str)

Stage 0 PR3 T3.1 + T3.2。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:04:49 +08:00
1445043649 cbbb83a706 docs(impl): STATUS update after PR2 live verification on RDS
PR1 + PR2 已 live 验证:远程 Aliyun RDS (PostgreSQL 17.9) 上 9 张表
(DeerFlow 5 + LangGraph 4) 全部 create_all 成功。

记 3 个 PR2 follow-up commits(不在原 plan task list 但 live 必需):
  - testcontainers image 16→17 对齐 RDS 大版本
  - serve.sh auto-add --extra postgres 防 uv sync 卸 asyncpg
  - async_provider 剥 +asyncpg dialect 前缀让 LangGraph saver 能解析 URL

更新"一句话状态" + 标 PG live 验证(区分 testcontainers 路径仍未实跑)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 18:01:22 +08:00
1445043649 39f8e117e8 fix(checkpointer): strip SQLAlchemy dialect prefix before passing to LangGraph
DeerFlow 用同一个 \`postgres_url\` 喂两条路径:
  - SQLAlchemy 引擎(需 \`postgresql+asyncpg://\`)
  - LangGraph AsyncPostgresSaver(psycopg 直连,需 libpq 风格 \`postgresql://\`)

PR2 把 .env / config.example.yaml 默认 URL 改成 \`+asyncpg\` 形态后,LangGraph
saver from_conn_string 会被 psycopg parser 直接抛 ProgrammingError
'missing "=" after "postgresql+asyncpg://..."'。

修:在 async_provider.py 传给 AsyncPostgresSaver 之前用正则剥掉
\`postgresql+\\w+://\` → \`postgresql://\`。一行修复,让同一个 URL 同时满足
SQLAlchemy 和 LangGraph 两条路径。

实测:make dev-daemon 起 gateway 成功,远程 RDS 上 9 张表(DeerFlow 5 +
LangGraph 4)全部 create_all 出来。

Stage 0 PR2 follow-up(不在原 plan task list 内,是 live make dev 触发的)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:54:57 +08:00
1445043649 bdef6c6b9f fix(serve): auto-install postgres extras when config.yaml selects postgres
scripts/serve.sh 在 \`uv sync --quiet\` 前 grep config.yaml 探测
\`database.backend: postgres\`;若是则追加 \`--extra postgres\`,否则保持原状。

修一个 PR1+PR2 没覆盖到的隐患:serve.sh 每次 dev 启动都跑 uv sync,会把
之前手动 \`uv sync --extra postgres\` 装的 asyncpg 卸掉,导致 gateway
启动时 ImportError。

Stage 0 PR2 follow-up(不在原 plan task list 内,是 live 验证时发现)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:35:23 +08:00
1445043649 7f17cb8fee chore(test): bump testcontainers Postgres to 17-alpine
\`make doctor\` 实测远程 Aliyun RDS = PostgreSQL 17.9。把
backend/tests/fixtures/postgres.py 的 testcontainers 镜像从
postgres:16-alpine 调到 postgres:17-alpine,与生产对齐。

同步把 STATUS.md "RDS 大版本对齐" 项标 done。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:29:16 +08:00
1445043649 ad22242ecc docs(impl): add Stage 0 progress dashboard (STATUS.md)
新增 docs/multi-tenant-redesign/03-impl/STATUS.md 作为 Stage 0 唯一的"现在
到哪了"权威来源:

  - 8 PR 状态表(PR0-PR2  merged,PR3-PR8 🟡 pending)
  - 用户必须跟进的事(live PG 实跑 / RDS 大版本对齐 / push origin / LOCK
    review / 远程 RDS 密码轮换)
  - 跳过/推迟的子任务(T1.10 docker-pending、T2.7 acked、T2.8 optional
    skipped、T2.9 backend/CLAUDE.md 待补)
  - 即将遇到的开放问题(PR4 alembic baseline / _ensure_admin_user 现状 /
    PG 大版本)
  - PR3 模式选择 trade-off(Inline vs Subagent-Driven)
  - 维护规则:完成 PR 后必更新;新 session 首先读本文件

填补 plan/impl notes 之外的空白:plan 是静态的,impl notes 是 per-PR
快照;STATUS 是跨 PR 的执行状态视图。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 16:44:09 +08:00
1445043649 1112a1971b docs(impl): PR2 implementation note + acceptance checklist
记录 Stage 0 PR2 的实施落点、关键 LOCK 决策、跟进项与 7 commit 列表。
验收:3087 passed + 23 skipped + 0 failed(vs PR1 基线 3085 + 2 新
default-backend 测试)。

Stage 0 PR2 T2.10. PR2 完成。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:23:16 +08:00
1445043649 c53295dfee docs(readme): add Database backend section in Quick Start
Quick Start 加 Step 3 "Database backend (Stage 0+ defaults to Postgres)":
  - 说明 config.example.yaml 默认 postgres + DATABASE_URL 写 .env
  - 给本地 dev 起 docker compose postgres 一行命令
  - 提示 make doctor / make dev preflight 行为
  - <details> 折叠 SQLite fallback 说明(offline dev 用)

Stage 0 PR2 T2.9(README 部分;backend/CLAUDE.md follow-up)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:20:26 +08:00
1445043649 745a33e05d chore: T2.7 acknowledgement (covered by T1.8)
Plan T2.7 = "setup_wizard.py 推荐 postgres,引导填 DATABASE_URL"。
T1.8 commit eae01901 已经实现:

  - 加 "Use Postgres? (y = postgres, n = sqlite)" 问答,默认 y
  - 选 y 时引导填 DATABASE_URL,可留空稍后填 .env
  - writer.build_minimal_config(database_backend='postgres') 输出
    database.backend=postgres + postgres_url=\$DATABASE_URL

无新代码改动;本 commit 仅为 task tracking 完整性。

Stage 0 PR2 T2.7(acknowledged-only)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:17:51 +08:00
1445043649 83b680b2ea feat(check): postgres preflight in scripts/check.py
新增 check_postgres_preflight() —— make dev 链 (check.py → serve.sh) 的最后一道
preflight:
  - config.yaml 不存在 → silent skip(让 setup_wizard 引导)
  - backend != postgres → silent skip
  - DATABASE_URL 未设 → FAIL with hint
  - postgres 设了但 host:port 3s socket 不通 → FAIL with 启 docker 提示
  - 通则 OK + 显示 host:port

不在 serve.sh 里加:Makefile 已经把 check.py 串在 serve.sh 之前,FAIL 会
自然阻断启动;避免 bash + python 两处实现 PG 探测。

doctor.py 的 check_database 是事后诊断(make doctor);本 check 是事前
preflight(make dev/start)—— 互补。

Stage 0 PR2 T2.6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:17:19 +08:00
1445043649 3e62a0f6ef feat(docker): gateway depends_on postgres healthcheck (dev only)
dev compose gateway 段加 depends_on: { postgres: { condition: service_healthy } },
确保 postgres 通过 pg_isready 后 gateway 才启 uvicorn。

prod compose 不动:production 用远程 RDS(不在 compose 内),无 postgres
service 可 depend;docker-compose.yaml 不加此 depends_on。

YAML 通过 docker compose config 校验。

Stage 0 PR2 T2.5.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:13:44 +08:00
1445043649 d312bdf968 test(config): pin default postgres backend + sqlite regression
新建 backend/tests/test_default_database_backend.py 两个测试:

  - test_explicit_sqlite_backend_still_works (T2.3)
    显式 database.backend=sqlite 仍生效;防 PR2 改默认后 SQLite 用户
    悄无声息 regression
  - test_config_example_default_backend_is_postgres (T2.4)
    直接读 on-disk config.example.yaml,断言 database.backend == 'postgres'
    且 postgres_url == '\$DATABASE_URL'(不允许硬编码凭据)

Stage 0 PR2 T2.3 + T2.4.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:12:28 +08:00
1445043649 7d3d3560ad feat(env): activate DATABASE_URL with local-dev default
.env.example DATABASE_URL 行从注释改为活值(指向 docker compose 起的本地
postgres)。同时加 RDS 示例注释作为参考。psycopg2 风格 URL 升级到
postgresql+asyncpg:// 与 SQLAlchemy 异步 dialect 对齐。

Stage 0 PR2 T2.2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 04:24:32 +08:00
1445043649 404135a16a feat(config): default database backend to postgres
config.example.yaml database 段:postgres 成为活跃默认(postgres_url=\$DATABASE_URL),
SQLite 改为注释掉的 fallback 块(offline dev 用)。bump config_version 9→10。

为什么默认改 postgres:Stage 0+ 已要求 ALTER 4 张表加 workspace_id;
SQLite 加列后再迁 PG 是返工。Stage 0 没有生产数据,迁移阻力最小。

Stage 0 PR2 T2.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 23:03:52 +08:00
1445043649 85a14f4c05 docs(impl): PR1 implementation note + acceptance checklist
记录 Stage 0 PR1 的实施落点、关键 LOCK 决策、fixture 用法示例、跟进项与
10 commit 列表。验收:3085 passed + 23 skipped + 0 failed(无 regression),
PG smoke 在 docker daemon 起来后实跑(CI workflow 已配)。

Stage 0 PR1 T1.10. PR1 完成。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:58:21 +08:00
1445043649 a33b46b4af ci(postgres): add Postgres workflow running @pytest.mark.postgres tests
新增 .github/workflows/backend-postgres-tests.yml,独立于现有
backend-unit-tests workflow:
  - GitHub runner (ubuntu-latest) 自带 docker daemon
  - testcontainers 在 runner 上起 postgres:16-alpine 容器
  - uv sync --extra postgres-test 拉 testcontainers + asyncpg + psycopg
  - pytest -m postgres 仅跑 PG 标签的 4 个 smoke test (+ 后续 PR 增量)

为什么独立 workflow 而不是合到 backend-unit-tests:
  - 现有 fast 测试不受 PG container 启动延迟影响
  - 可独立 fail-soft(早期 Stage 0 rollout 期间需要时加 continue-on-error)
  - fork 用户不强制承担 docker infra 成本

Stage 0 PR1 T1.9.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:55:53 +08:00
1445043649 eae0190184 feat(wizard): add Postgres backend question to setup wizard
setup_wizard.py 在 Step 3 (Execution) 之后新增一个 Database 问答:
"Use Postgres? (y = postgres, n = sqlite)",默认 y(Stage 0+ 推荐 PG)。
选 y 时引导填 DATABASE_URL(可留空稍后写 .env);DATABASE_URL 进 .env,
config.yaml 写入 database.backend=postgres + postgres_url=\$DATABASE_URL。

writer.py build_minimal_config 加 database_backend 参数,postgres 时
覆盖 base_config 的 database 段;默认 sqlite 时沿用 base_config 行为
(继承 config.example.yaml 的 sqlite_dir 等)。

minimal pattern:不新建 wizard/steps/database.py,inline 在 main 里加 1
问答 + writer 加 1 参数。后续如果需要更复杂数据库选项再升为完整 step 模块。

Stage 0 PR1 T1.8.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:55:19 +08:00
1445043649 e6f5ba53bd feat(doctor): add Database section probing configured backend
scripts/doctor.py 新增 check_database():

  - sqlite → OK + 显示 sqlite_dir
  - memory → WARN(数据非持久化)
  - postgres → 解析 database.postgres_url 中的 \$DATABASE_URL,asyncpg
    实际连接 + SELECT version() → OK with server version;连接失败时
    FAIL 给出可执行 fix(启 docker compose postgres / 检查 DATABASE_URL)
  - 未知 backend → WARN

Database section 插在 LLM Provider 与 Sandbox 之间。SQLite 部署不会触发
PG 探测(不破坏现有 dev 体验)。

Stage 0 PR1 T1.7.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:52:26 +08:00
1445043649 8f480cd76f test(persistence): postgres smoke tests for init_engine + ThreadMetaRepo
新建 backend/tests/test_postgres_smoke.py,4 个 test 用 postgres_url fixture:

  - test_postgres_url_creates_isolated_database (T1.4)
    asyncpg 直连验证 fixture 落到 test_<hex> DB
  - test_postgres_url_isolates_between_tests (T1.4)
    手动起第 2 个 DB 比对,证明每 test 隔离
  - test_init_engine_postgres_creates_tables (T1.5)
    init_engine('postgres', postgres_url) 跑完后 information_schema
    出现 5 张现有表(users/threads_meta/runs/feedback/run_events)
  - test_thread_meta_repo_postgres_round_trip (T1.6)
    ThreadMetaRepository.create + get 完整 round-trip,验证 PG 上仓储行为
    与 SQLite 对齐

全部 @pytest.mark.postgres 门控;Docker 不可用时由 fixture 跳过。本次本地
docker daemon 未起,4 测试 SKIP;CI workflow(T1.9)将带 docker service
触发实跑。

Regression:跑全套 `pytest tests/` 3085 passed + 23 skipped + 0 failed
(基线 3086 passed + 18 skipped;diff = 4 新 postgres skip + 1 env 相关
live test 偶发 skip)。

Stage 0 PR1 T1.4 + T1.5 + T1.6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:50:10 +08:00
1445043649 31361e2dcf test(fixtures): add testcontainers Postgres fixture for Stage 0
新建 backend/tests/fixtures/postgres.py,提供两层 fixture:
  - postgres_container (session 级):用 PostgresContainer 起 postgres:16-alpine
  - postgres_url (function 级):每 test 一个 ephemeral DB,teardown 时
    pg_terminate_backend 清掉残连后 DROP DATABASE

为什么 per-DB 而不是 per-schema:asyncpg + SQLAlchemy 不通过 URL 传 search_path,
per-DB 一次 ~50ms 开销可接受,让 test 代码不感知 schema。

Docker 不可用时 fixture 自动 pytest.skip 而非 error,dev 环境无 Docker
仍能跑其余 3086 个测试。

注册 pytest mark `postgres`、把 tests/ 加 sys.path 让 pytest_plugins
按 `fixtures.postgres` 路径解析(不加 tests/__init__.py 避免干扰
现有 pytest 发现行为)。

Stage 0 PR1 T1.3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:44:02 +08:00
1445043649 ae4ea46be2 feat(docker): add postgres service to dev compose
新增 postgres:16-alpine service + named volume + healthcheck,作为 Stage 0+
默认 DB backend。生产可通过 DATABASE_URL 指向远程 RDS 时跳过此 service。
端口 / 用户 / 密码 / 库名都走 env var 覆盖(POSTGRES_USER/PORT/...
默认 deerflow/5432/deerflow_dev)。

YAML 通过 `docker compose config` 校验;live healthcheck 待 docker daemon 起后由
T1.3 testcontainers 路径覆盖。

Stage 0 PR1 T1.2.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:40:12 +08:00
1445043649 fab85b14a6 build(deps): add postgres-test optional extra for testcontainers
新增 postgres-test extra(harness + backend),引入 testcontainers[postgres]
4.14.2 用于 Stage 0 PR1 的 PG fixture。复用现有 postgres extra 的
asyncpg/psycopg/langgraph-checkpoint-postgres,pytest-asyncio 沿用 dev group。

Stage 0 PR1 T1.1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:38:50 +08:00
1445043649 a74b88a45c docs(superpowers): add Stage 0 multi-tenant foundation master plan
8 PR 串成的 stage-level 拆解(Postgres 切换 + workspaces schema + auth
扩字段 + 入口路由强校验 + CI boundary + headless schema 预留)。每个 PR
含 commit-sized TDD task list 和验收清单。执行时各 PR 独立 review
checkpoint。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:33:25 +08:00
1445043649 89fe54cc07 docs(multi-tenant): 加入汇总索引、跨文档一致性修订、Postgres 切换前移到 Stage 0
* 新增 README.zh-CN.md 汇总索引:ADR 状态表 + Stage 0-4 业务目标 / 技术路径 /
  验证方式 + Stage↔ADR 对照矩阵 + 不可逆决策一览 + 用语映射 + FAQ + 阅读路径
* 新增 workspace-schema-design.zh-CN.md(Stage 0 schema 锁定版)一并入库
* 7 份 ADR 顶部加"代码命名"映射行(tenant_id ↔ workspace_id)
* ADR-002 §1 加分期落地提示,明确 K8s 推迟到 Stage 3
* ADR-005 §5"第 1/2 阶段"补出与 rollout Stage 2/3 的映射
* ADR-007 §4 加 /api/v1/ 反向链接;§8 加 tid → wid 字段名映射
* headless-api §0/§7 把"SaaS + on-prem 双主线"改为"SaaS 主线、schema 兼容 on-prem"
* phased-rollout 去除重复的"Go/No-Go 进入 Stage 2"段

Postgres 切换从 Stage 1 提前到 Stage 0:Stage 0 已要 ALTER 4 张表加
workspace_id,先 SQLite 再 PG 是纯返工;Stage 0 没有生产数据,迁移阻力最小。
同步调整 phased-rollout / headless-api / phase-0-plan / workspace-schema-design /
README 中的时间盒(Stage 0: 3-4→4-5 周;Stage 1: 10-15→8-13 周)、不可逆决策
清单、PR 顺序、轨道前置依赖。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 21:39:58 +08:00
1445043649 9ff790554d docs(persistence): 给 ORM 表/字段补充中文注释
通过 SQLAlchemy 的 comment= 给 5 张持久化表(users / threads_meta / runs /
run_events / feedback)的所有字段以及表本身加上中文注释,便于读代码、
生成文档与未来切到 Postgres 时直接落库为 COMMENT ON。

SQLite 引擎本身不支持 COMMENT ON,运行时不会改变 .schema 输出。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 18:50:17 +08:00
1445043649 ecc9339ede docs(multi-tenant): 加入 Headless API 改造轨道(Pattern A/B)+ Stage 1 双轨修订
新增 headless-api-track 文档,把 DeerFlow 作为后端服务的两种集成
pattern 全部纳入设计:

- Pattern A(业务 backend 代理):业务系统 backend 用 API key 调,
  浏览器走他们的 backend。覆盖 IM channels + server-to-server 集成。
- Pattern B(自研 web 浏览器直连):业务 backend 调
  /api/v1/auth/exchange-token 换 5-15 min 短期 JWT → 浏览器拿 JWT 直连
  含 SSE。核心 4 件事:exchange-token endpoint、ServiceTokenAuthBackend
  (AuthMiddleware 第三条路径)、workspaces.allowed_origins + CORS
  中间件、SSE 跨域验证。
- 不做 widget / iframe(纯 API)。

数据模型:service_accounts / api_keys / external_users 表(schema 在
Stage 0 末加上不阻塞);身份模式三态(collapsed / external_passthrough
/ both)按 endpoint 分支。

Stage 1 改为三轨并行:付费 SaaS / Pattern A / Pattern B;时间盒从
6-10 周延到 10-15 周。Pattern B 依赖 Pattern A 完成,建议 Stage 1
末 1-2 周做。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 23:29:46 +08:00
1445043649 fd8d0d637d docs(multi-tenant): 加入 Stage 0 代码地图
把 Stage 0 必做项落到具体文件 + 行号,覆盖 7 个子系统:
auth、user 数据模型、thread 入口路由、ThreadDataMiddleware + 路径系统、
仓储访问模式、setup / 注册流程、CSRF + ContextVar 注入。

文档结构:每节有【关键文件 + 行号】、【关键函数 / 类】、【当前数据流】、
【Stage 0 改动锚点】;外加改动影响面总览(每条 Stage 0 必做项映射到
具体代码位置)、不可逆决策落点、推荐阅读顺序(11 个文件,1.5-2.5h
粗读)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 23:29:27 +08:00
1445043649 35ae97d0a6 docs(multi-tenant): Stage 2 加入 per-user skill 覆盖与 config 设计
补充团队 workspace 内不同成员的 skill 区分模型:

- 启用状态:workspace 级默认 + per-user 覆盖
  (final_enabled = user_override ?? workspace_default)
- skill 私有配置(API key 等):per-user 强制隔离 + KMS 加密
- 上传权限:仅 owner / admin,成员只能 enable/disable + 填自己 config

PR 顺序新增第 8 步,依赖 RBAC + KMS 已就位。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 22:47:05 +08:00
1445043649 e78ed687a6 docs(multi-tenant): 加入按规模分期的落地方案(02-rollout)
基于已收敛的目标客户画像(个人用户为主 + 小团队 / 中心化 SaaS /
freemium)把 7 份 ADR 拆成 5 期落地:

- Stage 0:workspace 模型立起来 + auth 收紧(不可逆决策集中在此)
- Stage 1:Postgres + Quota 必落(freemium 不上 quota = 信用卡递给攻击者)
- Stage 2:DeerFlow 表 RLS、KMS、ObjectStorage S3、付费分层
- Stage 3:K8s sandbox + BYO key + audit DB 拆分
- Stage 4:SSO / custom domain / per-tenant DB(按 enterprise 客户合同驱动)

关键取舍:① Stage 0 末就把 workspace_id 列加到 SQLite,避免 Stage 1
切 Postgres 时再补;② Quota 比 ADR-003 原稿提前一档到 Stage 1;③ K8s
sandbox 推到 Stage 3,AioSandbox + 出网白名单 + 资源限额撑到几千用户。

时间盒:最小可付费 4 个月(Stage 0+1)/ 风险可控增长 8 个月(+Stage 2)
/ 全功能 14 个月(+Stage 3)。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 22:28:55 +08:00
1445043649 8bdd308ea0 docs(multi-tenant): 据审计对齐 ADR-002/003/004/005 与 phase-0 计划
非 spike 驱动的对齐改动:
- ADR-002/004/005:头部加现状提示 + 链接审计报告(沙箱出网/资源缺位、
  token_version 已存在 MembershipCache 全新建、ObjectStorage/7 表/KMS
  全部从 0 起)
- ADR-003 LLM 计费:修正 TokenUsageMiddleware 当前只 log 不持久化的描述;
  补充 create_chat_model sync→async 改造的连带影响说明
- phase-0 计划:新增 §3.5 底座先行(Postgres 测试夹具 / ObjectStorage
  Protocol / KMS 抽象 3 件并行做),时间盒 2 → 3 周;ADR-006/007 摘要
  对齐;DoD 加底座骨架检查项

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 21:03:32 +08:00
1445043649 27c4f14233 docs(multi-tenant): 加入 ADR 审计 + spike,并据其修订 ADR-001/006/007
新增两份评审产出物:
- adr-vs-code-audit:7 份 ADR 与现状代码的差异核对,标注每条假设是否成立
- adr-spike-langgraph-postgres:实测 langgraph-checkpoint-postgres==3.0.5
  注入能力,确认不存在 connection_factory 参数,且 psycopg_pool 自带的
  configure callback 不是 per-acquire hook

据 spike 与审计修订三份 ADR:
- ADR-001 数据隔离:LangGraph 表改为应用层强校验 + threads_meta unique
  约束兜底(不再挂 RLS、不 ALTER 表);hook 点从 AssistantsCompat 修正
  为 threads.py + thread_runs.py
- ADR-006 运行时与渠道:§2.1 完全重写为应用层强校验;MCP OAuth token
  从"无持久化进程内存"直接做加密 DB;channel store binding 改为新建
  channel_bindings 表
- ADR-007 路由与前端:删除 Better Auth 假设(前端实际无此依赖),改为
  扩展现有 auth/jwt.py TokenPayload 加 tid/role 字段

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 21:03:16 +08:00
1445043649 dce5e9598b docs(multi-tenant): add ADRs and phase-0 plan for multi-tenant redesign
Add 7 ADRs and a phase-0 plan covering the multi-tenant redesign of
DeerFlow, plus an architecture-overview snapshot of the current state.

ADRs:
- 001 data isolation: row-level tenant_id + Postgres RLS, including
  LangGraph-owned checkpoint tables (subquery RLS or column upgrade path).
- 002 sandbox isolation: K8s namespace + gVisor + NetworkPolicy default-
  deny, threat model and pod spec defaults.
- 003 LLM key & billing: hybrid platform/BYO with pessimistic reservation
  to handle the "ghost token" overflow on the last call, plus a usage
  category split for memory/title/summarization charges.
- 004 RBAC: two-level (owner/admin/member), JWT-with-role + 30s LRU
  cache for reads, strict DB lookup for sensitive writes, token_version
  bump as the single revocation path.
- 005 storage topology: Postgres (structured) + S3-compatible object
  store (large objects) + emptyDir (ephemeral); explicit treatment of
  the extensions_config.json migration's downstream effects.
- 006 runtime & channel tenancy: per-tenant MCP cache, dual-track skills
  loader, sandbox provider routing by namespace, internal LLM call
  billing, IM channel-to-tenant binding model.
- 007 routing & frontend: path-slug URL form, JWT-only API auth,
  TenantProvider, hard-reload tenant switch, Better Auth integration.

These docs are decision records; no code changes are included.
2026-05-08 23:45:33 +08:00
128 changed files with 15694 additions and 335 deletions
+5 -2
View File
@@ -38,8 +38,11 @@ INFOQUEST_API_KEY=your-infoquest-api-key
# GitHub API Token
# GITHUB_TOKEN=your-github-token
# Database (only needed when config.yaml has database.backend: postgres)
# DATABASE_URL=postgresql://deerflow:password@localhost:5432/deerflow
# Database (Stage 0+ default; required when config.yaml has database.backend: postgres)
# Local dev — start with: docker compose -f docker/docker-compose-dev.yaml up -d postgres
DATABASE_URL=postgresql+asyncpg://deerflow:deerflow_dev@localhost:5432/deerflow
# Remote RDS example:
# DATABASE_URL=postgresql+asyncpg://USER:PASSWORD@HOST:5432/DBNAME
#
# WECOM_BOT_ID=your-wecom-bot-id
# WECOM_BOT_SECRET=your-wecom-bot-secret
@@ -0,0 +1,56 @@
name: Postgres Tests
# Stage 0 PR1 · runs the @pytest.mark.postgres subset against a real
# Postgres backend (testcontainers spawns postgres:16-alpine on the
# GitHub runner's docker daemon).
#
# Kept as a separate workflow from `Unit Tests` so:
# - the existing fast unit test loop is unchanged
# - PG tests can fail-soft during early Stage 0 rollout if needed
# (set continue-on-error: true on the run step)
# - infra cost is opt-in for forks
on:
push:
branches: [ 'main' ]
pull_request:
types: [opened, synchronize, reopened, ready_for_review]
concurrency:
group: postgres-tests-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
backend-postgres-tests:
if: github.event.pull_request.draft == false
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- name: Checkout
uses: actions/checkout@v6
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: '3.12'
- name: Install uv
uses: astral-sh/setup-uv@v7
- name: Verify Docker daemon (testcontainers needs it)
run: docker version
- name: Install backend dependencies (with postgres-test extra)
working-directory: backend
run: uv sync --group dev --extra postgres-test
- name: Run @pytest.mark.postgres tests
working-directory: backend
# -m postgres selects only postgres-tagged tests; testcontainers
# spins up postgres:16-alpine as the fixture's session-scoped
# container. ~5-10s container startup + per-test DB CREATE/DROP.
run: PYTHONPATH=. uv run pytest -m postgres -v
+3
View File
@@ -60,3 +60,6 @@ config.yaml.bak
/frontend/playwright-report/
.gstack/
.worktrees
skills/gstack
skills/superpowers
CLAUDE.md
+9
View File
@@ -31,6 +31,7 @@ help:
@echo " make start-daemon - Start prod services in background (daemon mode)"
@echo " make stop - Stop all running services"
@echo " make clean - Clean up processes and temporary files"
@echo " make migrate-paths - Migrate legacy users/ tree into workspaces/ layout (DRY_RUN=1 to preview)"
@echo ""
@echo "Docker Production Commands:"
@echo " make up - Build and start production Docker services (localhost:2026)"
@@ -144,6 +145,14 @@ clean: stop
@-rm -rf logs/*.log 2>/dev/null || true
@echo "✓ Cleanup complete"
# Lift legacy per-user paths into the per-workspace layout (PR6).
# Pass DRY_RUN=1 to log the migration plan without writing.
# DEFAULT_WORKSPACE=<wid> claims un-assigned users (defaults to legacy_workspace).
migrate-paths:
@cd backend && PYTHONPATH=. uv run python scripts/migrate_paths_to_workspace.py \
$(if $(filter 1 true,$(DRY_RUN)),--dry-run) \
$(if $(DEFAULT_WORKSPACE),--default-workspace $(DEFAULT_WORKSPACE))
# ==========================================
# Docker Development Commands
# ==========================================
+32
View File
@@ -202,6 +202,38 @@ That prompt is intended for coding agents. It tells the agent to clone the repo
</details>
3. **Database backend (Stage 0+ defaults to Postgres)**
`config.example.yaml` ships with `database.backend: postgres` and `postgres_url: $DATABASE_URL`. Set `DATABASE_URL` in `.env`:
```bash
DATABASE_URL=postgresql+asyncpg://deerflow:deerflow_dev@localhost:5432/deerflow
```
Start the local Postgres dev container:
```bash
docker compose -f docker/docker-compose-dev.yaml up -d postgres
```
Or point `DATABASE_URL` at a remote RDS / Cloud SQL instance.
`make doctor` will report the configured backend, attempt an asyncpg connection, and surface actionable fix hints. `make dev` preflights Postgres reachability before starting services and aborts if `DATABASE_URL` is unreachable.
<details>
<summary>Offline dev (SQLite fallback)</summary>
If you prefer no Postgres, edit `config.yaml`:
```yaml
database:
backend: sqlite
sqlite_dir: .deer-flow/data
```
SQLite is preserved as a valid backend for offline development. RLS / multi-node features (Stage 2+) require Postgres.
</details>
### Running the Application
#### Deployment Sizing
+2
View File
@@ -102,6 +102,8 @@ Regression tests related to Docker/provisioner behavior:
Boundary check (harness → app import firewall):
- `tests/test_harness_boundary.py` — ensures `packages/harness/deerflow/` never imports from `app.*`
- `tests/test_workspace_boundary.py` — AST static scan that forbids direct imports of `langgraph.checkpoint.*` (and the third-party `langgraph_checkpoint_postgres` / `langgraph_checkpoint_sqlite` packages) outside the allowlist in `tests/boundary_allowlist.toml`. Everywhere else must obtain a checkpointer via `app.gateway.deps.get_checkpointer` or the harness `deerflow.runtime.checkpointer` factory. Imports inside `if TYPE_CHECKING:` blocks are exempt automatically (they do not enter runtime). When a legitimate new importer is genuinely needed, append its path to `boundary_allowlist.toml` in the same PR
- `tests/test_workspace_boundary_self.py` — self-tests for the scanner above (9 cases over synthetic `.py` files) guarding against silent-empty regressions
CI runs these regression tests for every pull request via [.github/workflows/backend-unit-tests.yml](../.github/workflows/backend-unit-tests.yml).
+47
View File
@@ -108,6 +108,23 @@ async def _ensure_admin_user(app: FastAPI) -> None:
admin_id = str(row.id)
# Stage 0 PR4 backfill: pre-PR4 admins have no default_workspace_id.
# Create their personal workspace + owner membership on next boot so
# they can log in and pass the workspace gate without hand-rolling
# SQL. Idempotent — ensure_default_workspace short-circuits when the
# column is already set.
try:
admin_user = await provider.get_user(admin_id)
if admin_user is not None and not admin_user.default_workspace_id:
from app.gateway.routers.auth import ensure_default_workspace
ws_id = await ensure_default_workspace(admin_user)
logger.info("Backfilled default workspace %s for admin %s", ws_id, admin_id)
except Exception:
# Don't fail startup if backfill stumbles — the user can still
# log in (login_local calls the same helper on its hot path).
logger.exception("Admin workspace backfill failed (non-fatal)")
# LangGraph store orphan migration — non-fatal.
# This covers the "no-auth → with-auth" upgrade path for users
# whose existing LangGraph thread metadata has no user_id set.
@@ -158,6 +175,34 @@ async def _migrate_orphaned_threads(store, admin_user_id: str) -> int:
return migrated
def _check_path_migration_pending(app: FastAPI) -> None:
"""Warn the operator if the PR4 legacy user-isolation layout still has content.
PR6 routes every new write into ``{base_dir}/workspaces/{wid}/...`` via
``Paths``. Pre-PR6 installations have data at
``{base_dir}/users/{uid}/...`` that needs ``make migrate-paths`` to lift
it under a workspace. We emit a warning at boot rather than crashing so
the gateway keeps serving (reads from the legacy tree still work via the
user_id branch of ``Paths.thread_dir``), but with a loud signal to run
the migration script.
"""
from deerflow.config.paths import get_paths
legacy_users = get_paths().base_dir / "users"
if not legacy_users.exists():
return
try:
has_content = any(legacy_users.iterdir())
except OSError:
# Permission or transient FS issue — don't escalate; lifespan must succeed.
return
if has_content:
logger.warning(
"Legacy per-user layout detected at %s. Run `make migrate-paths` to lift it under the per-workspace layout (PR6).",
legacy_users,
)
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"""Application lifespan handler."""
@@ -174,6 +219,8 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
config = get_gateway_config()
logger.info(f"Starting API Gateway on {config.host}:{config.port}")
_check_path_migration_pending(app)
# Initialize LangGraph runtime components (StreamBridge, RunManager, checkpointer, store)
async with langgraph_runtime(app):
logger.info("LangGraph runtime initialised")
+4
View File
@@ -21,6 +21,7 @@ class AuthErrorCode(StrEnum):
PROVIDER_NOT_FOUND = "provider_not_found"
NOT_AUTHENTICATED = "not_authenticated"
SYSTEM_ALREADY_INITIALIZED = "system_already_initialized"
WORKSPACE_REQUIRED = "workspace_required"
class TokenError(StrEnum):
@@ -29,6 +30,7 @@ class TokenError(StrEnum):
EXPIRED = "expired"
INVALID_SIGNATURE = "invalid_signature"
MALFORMED = "malformed"
WORKSPACE_MISSING = "workspace_missing"
class AuthErrorResponse(BaseModel):
@@ -42,4 +44,6 @@ def token_error_to_code(err: TokenError) -> AuthErrorCode:
"""Map TokenError to AuthErrorCode — single source of truth."""
if err == TokenError.EXPIRED:
return AuthErrorCode.TOKEN_EXPIRED
if err == TokenError.WORKSPACE_MISSING:
return AuthErrorCode.WORKSPACE_REQUIRED
return AuthErrorCode.TOKEN_INVALID
+40 -8
View File
@@ -1,6 +1,7 @@
"""JWT token creation and verification."""
from datetime import UTC, datetime, timedelta
from typing import Any
import jwt
from pydantic import BaseModel
@@ -10,30 +11,51 @@ from app.gateway.auth.errors import TokenError
class TokenPayload(BaseModel):
"""JWT token payload."""
"""JWT token payload.
`wid` / `role` were added in Stage 0 PR4 and are optional at the
model level so legacy 4-field tokens still parse into a value —
callers (middleware, decode_token) decide what "missing wid" means.
Post-PR4 production tokens always carry both fields.
"""
sub: str # user_id
wid: str | None = None # workspace_id (Stage 0 PR4)
role: str | None = None # owner / admin / member (Stage 0 PR4)
exp: datetime
iat: datetime | None = None
ver: int = 0 # token_version — must match User.token_version
def create_access_token(user_id: str, expires_delta: timedelta | None = None, token_version: int = 0) -> str:
def create_access_token(
user_id: str,
expires_delta: timedelta | None = None,
token_version: int = 0,
*,
workspace_id: str | None = None,
role: str | None = None,
) -> str:
"""Create a JWT access token.
Args:
user_id: The user's UUID as string
expires_delta: Optional custom expiry, defaults to 7 days
token_version: User's current token_version for invalidation
user_id: The user's UUID as string.
expires_delta: Optional custom expiry, defaults to 7 days.
token_version: User's current token_version for invalidation.
workspace_id: Optional active workspace id; encoded as the ``wid`` claim.
role: Optional workspace role; encoded as the ``role`` claim.
Returns:
Encoded JWT string
Encoded JWT string.
"""
config = get_auth_config()
expiry = expires_delta or timedelta(days=config.token_expiry_days)
now = datetime.now(UTC)
payload = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
payload: dict[str, Any] = {"sub": user_id, "exp": now + expiry, "iat": now, "ver": token_version}
if workspace_id is not None:
payload["wid"] = workspace_id
if role is not None:
payload["role"] = role
return jwt.encode(payload, config.jwt_secret, algorithm="HS256")
@@ -46,10 +68,20 @@ def decode_token(token: str) -> TokenPayload | TokenError:
config = get_auth_config()
try:
payload = jwt.decode(token, config.jwt_secret, algorithms=["HS256"])
return TokenPayload(**payload)
except jwt.ExpiredSignatureError:
return TokenError.EXPIRED
except jwt.InvalidSignatureError:
return TokenError.INVALID_SIGNATURE
except jwt.PyJWTError:
return TokenError.MALFORMED
# Reject legacy pre-PR4 tokens that lack the wid claim. Reported as
# WORKSPACE_MISSING (not MALFORMED) so middleware can surface a
# specific 401 telling the frontend to re-issue via /select-workspace.
if "wid" not in payload or payload.get("wid") is None:
return TokenError.WORKSPACE_MISSING
try:
return TokenPayload(**payload)
except Exception:
return TokenError.MALFORMED
+41
View File
@@ -31,6 +31,12 @@ class User(BaseModel):
needs_setup: bool = Field(default=False, description="True for auto-created admin until setup completes")
token_version: int = Field(default=0, description="Incremented on password change to invalidate old JWTs")
# Workspace linkage (Stage 0 PR4)
default_workspace_id: str | None = Field(
default=None,
description="The workspace the user lands in by default after login. NULL → /select-workspace.",
)
class UserResponse(BaseModel):
"""Response model for user info endpoint."""
@@ -39,3 +45,38 @@ class UserResponse(BaseModel):
email: str
system_role: Literal["admin", "user"]
needs_setup: bool = False
class UserMeWorkspace(BaseModel):
"""One workspace entry in ``GET /auth/me`` (Stage 0 PR4)."""
id: str
name: str
slug: str
role: str
class UserMeResponse(BaseModel):
"""Response model for ``GET /auth/me`` — extends UserResponse with workspaces."""
id: str
email: str
system_role: Literal["admin", "user"]
needs_setup: bool = False
default_workspace_id: str | None = None
workspaces: list[UserMeWorkspace] = []
class ActiveWorkspace(BaseModel):
"""Lightweight workspace proxy injected into the request-scoped contextvar.
Implements the structural ``CurrentWorkspace`` protocol expected by
``deerflow.runtime.workspace_context``: only ``.id`` (str) and
``.role`` (str) are required. We intentionally do *not* embed the
full ``WorkspaceRow`` here — the middleware needs to set the
contextvar on every request and an extra DB lookup just to populate
a name/slug we don't use yet would be wasted work.
"""
id: str
role: str
@@ -46,6 +46,7 @@ class SQLiteUserRepository(UserRepository):
oauth_id=row.oauth_id,
needs_setup=row.needs_setup,
token_version=row.token_version,
default_workspace_id=row.default_workspace_id,
)
@staticmethod
@@ -60,6 +61,7 @@ class SQLiteUserRepository(UserRepository):
oauth_id=user.oauth_id,
needs_setup=user.needs_setup,
token_version=user.token_version,
default_workspace_id=user.default_workspace_id,
)
# ── CRUD ──────────────────────────────────────────────────────────
@@ -106,6 +108,7 @@ class SQLiteUserRepository(UserRepository):
row.oauth_id = user.oauth_id
row.needs_setup = user.needs_setup
row.token_version = user.token_version
row.default_workspace_id = user.default_workspace_id
await session.commit()
return user
@@ -0,0 +1,83 @@
"""Workspace slug helpers for the registration / initialize flow.
Stage 0 PR4 T4.10. Two responsibilities:
1. ``auto_slug_from_email(email)`` — pure transform from an email's
local part to a base slug that matches the schema's
``^[a-z0-9](-?[a-z0-9])*$`` pattern.
2. ``next_available_slug(base, exists_check=...)`` — collision walker
that appends ``-2``, ``-3``, … until ``exists_check`` reports the
candidate is free. Kept separate from ``auto_slug_from_email`` so
the pure function can be tested without a database.
Lives in the auth package (not in ``persistence``) because the input
is the user's email — a registration-time concept that doesn't belong
in a generic ``WorkspaceRepository``.
"""
from __future__ import annotations
import re
import secrets
from collections.abc import Awaitable, Callable
# Mirror the schema's slug rules from
# ``deerflow.persistence.workspace.sql`` so callers of this module
# never need to import private constants from persistence.
_SLUG_MIN_LEN = 3
_SLUG_MAX_LEN = 32
def auto_slug_from_email(email: str) -> str:
"""Map an email to a deterministic, schema-valid base slug.
Algorithm (from workspace-schema-design §3.1):
1. Take the local part (before ``@``).
2. Replace ``+``, ``_``, ``.`` with ``-`` and lowercase.
3. Strip everything that isn't ``[a-z0-9-]``.
4. Collapse repeated ``-``; strip leading/trailing ``-``.
5. Clamp to 32 chars.
6. If the result is shorter than the schema minimum (3 chars) or
empty, fall back to ``user-{token_hex(4)}`` so we always emit
a valid slug.
The returned slug is the *base* — callers must run it through
:func:`next_available_slug` before persisting to handle collisions.
"""
local = email.split("@", 1)[0]
local = re.sub(r"[+_.]", "-", local).lower()
local = re.sub(r"[^a-z0-9-]", "", local)
local = re.sub(r"-+", "-", local).strip("-")
slug = local[:_SLUG_MAX_LEN]
if len(slug) < _SLUG_MIN_LEN:
return f"user-{secrets.token_hex(4)}"
return slug
async def next_available_slug(
base: str,
*,
exists_check: Callable[[str], Awaitable[bool]],
) -> str:
"""Return the first of ``base``, ``base-2``, ``base-3``, … that ``exists_check`` reports free.
Caller-supplied ``exists_check`` is awaited once per candidate so
we can swap in a repository's ``get_by_slug`` without coupling
this module to persistence imports.
When ``base + '-N'`` would exceed the 32-char schema limit, the
base is truncated before the suffix is appended. The walker never
returns an over-long slug.
"""
if not await exists_check(base):
return base
n = 2
while True:
suffix = f"-{n}"
max_base_len = _SLUG_MAX_LEN - len(suffix)
candidate = f"{base[:max_base_len]}{suffix}"
if not await exists_check(candidate):
return candidate
n += 1
+18 -2
View File
@@ -17,9 +17,11 @@ from starlette.responses import JSONResponse
from starlette.types import ASGIApp
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.auth.models import ActiveWorkspace
from app.gateway.authz import _ALL_PERMISSIONS, AuthContext
from app.gateway.internal_auth import INTERNAL_AUTH_HEADER_NAME, get_internal_user, is_valid_internal_auth_token
from deerflow.runtime.user_context import reset_current_user, set_current_user
from deerflow.runtime.workspace_context import reset_current_workspace, set_current_workspace
# Paths that never require authentication.
_PUBLIC_PATH_PREFIXES: tuple[str, ...] = (
@@ -119,8 +121,22 @@ class AuthMiddleware(BaseHTTPMiddleware):
# JWT-decode + DB-lookup pipeline a second time per request).
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=_ALL_PERMISSIONS)
token = set_current_user(user)
user_token = set_current_user(user)
# Inject workspace contextvar from the JWT's wid/role claims.
# decode_token has already rejected legacy no-wid tokens upstream,
# so by the time we get here payload.wid is guaranteed non-None
# for cookie-authenticated requests. Internal-auth requests skip
# the workspace contextvar (they don't have a workspace scope —
# the internal user is a system actor).
ws_token = None
payload = getattr(request.state, "auth_payload", None)
if payload is not None and payload.wid is not None:
ws_token = set_current_workspace(ActiveWorkspace(id=payload.wid, role=payload.role or "owner"))
try:
return await call_next(request)
finally:
reset_current_user(token)
if ws_token is not None:
reset_current_workspace(ws_token)
reset_current_user(user_token)
+10 -7
View File
@@ -268,24 +268,27 @@ def require_permission(
# Owner check for thread-specific resources.
#
# 2.0-rc moved thread metadata into the SQL persistence layer
# (``threads_meta`` table). We verify ownership via
# ``ThreadMetaStore.check_access``: it returns True for
# missing rows (untracked legacy thread) and for rows whose
# ``user_id`` is NULL (shared / pre-auth data), so this is
# strict-deny rather than strict-allow — only an *existing*
# row with a *different* user_id triggers 404.
# PR6: ``check_access`` now takes ``workspace_id`` as the third
# positional argument; cross-workspace always denies regardless
# of user_id match. We pull workspace_id from the contextvar
# AuthMiddleware sets per request (and fall back to "default"
# in no-auth dev mode so smoke flows keep working). Failures
# convert to **404**, not 403, so the response never leaks the
# existence of a thread that belongs to a different tenant.
if owner_check:
thread_id = kwargs.get("thread_id")
if thread_id is None:
raise ValueError("require_permission with owner_check=True requires 'thread_id' parameter")
from app.gateway.deps import get_thread_store
from deerflow.runtime.workspace_context import get_effective_workspace_id
workspace_id = get_effective_workspace_id()
thread_store = get_thread_store(request)
allowed = await thread_store.check_access(
thread_id,
str(auth.user.id),
workspace_id,
require_existing=require_existing,
)
if not allowed:
+4
View File
@@ -220,6 +220,10 @@ async def get_current_user_from_request(request: Request):
detail=AuthErrorResponse(code=AuthErrorCode.TOKEN_INVALID, message="Token revoked (password changed)").model_dump(),
)
# Stash decoded payload on request.state so AuthMiddleware can read
# wid/role for the workspace contextvar without a second decode.
request.state.auth_payload = payload
return user
+118 -8
View File
@@ -15,11 +15,60 @@ from app.gateway.auth import (
)
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import AuthErrorCode, AuthErrorResponse
from app.gateway.auth.models import UserMeResponse, UserMeWorkspace
from app.gateway.auth.workspace_slug import auto_slug_from_email, next_available_slug
from app.gateway.csrf_middleware import is_secure_request
from app.gateway.deps import get_current_user_from_request, get_local_provider
logger = logging.getLogger(__name__)
async def ensure_default_workspace(user) -> str:
"""Create the user's personal workspace + owner membership, set default_workspace_id.
Returns the new workspace id. Idempotent for users who already
have a default_workspace_id (used by both the registration flow
and the lifespan backfill in app.py).
"""
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.workspace import WorkspaceRepository
from deerflow.persistence.workspace.sql import SLUG_BLACKLIST
from deerflow.persistence.workspace_membership import WorkspaceMembershipRepository
if user.default_workspace_id:
return user.default_workspace_id
sf = get_session_factory()
ws_repo = WorkspaceRepository(sf)
m_repo = WorkspaceMembershipRepository(sf)
base_slug = auto_slug_from_email(user.email)
async def slug_exists(s: str) -> bool:
# Treat blacklisted slugs as "taken" so the walker skips them
# instead of letting WorkspaceRepository.create raise after a
# successful slug computation (the user picked a reserved name
# like "admin@example.com" → base slug "admin").
if s in SLUG_BLACKLIST:
return True
return (await ws_repo.get_by_slug(s)) is not None
unique_slug = await next_available_slug(base_slug, exists_check=slug_exists)
display_local = user.email.split("@", 1)[0]
workspace = await ws_repo.create(
name=f"{display_local}'s Workspace"[:64],
slug=unique_slug,
owner_id=str(user.id),
)
await m_repo.add(workspace_id=workspace["id"], user_id=str(user.id), role="owner")
user.default_workspace_id = workspace["id"]
await get_local_provider().update_user(user)
return workspace["id"]
router = APIRouter(prefix="/api/v1/auth", tags=["auth"])
@@ -292,7 +341,15 @@ async def login_local(
)
_record_login_success(client_ip)
token = create_access_token(str(user.id), token_version=user.token_version)
# Ensure the user has a workspace (covers pre-PR4 users still in DB
# whose default_workspace_id was never backfilled by the lifespan hook).
workspace_id = await ensure_default_workspace(user)
token = create_access_token(
str(user.id),
token_version=user.token_version,
workspace_id=workspace_id,
role="owner",
)
_set_session_cookie(response, token, request)
return LoginResponse(
@@ -316,7 +373,14 @@ async def register(request: Request, response: Response, body: RegisterRequest):
detail=AuthErrorResponse(code=AuthErrorCode.EMAIL_ALREADY_EXISTS, message="Email already registered").model_dump(),
)
token = create_access_token(str(user.id), token_version=user.token_version)
workspace_id = await ensure_default_workspace(user)
token = create_access_token(
str(user.id),
token_version=user.token_version,
workspace_id=workspace_id,
role="owner",
)
_set_session_cookie(response, token, request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
@@ -368,18 +432,57 @@ async def change_password(request: Request, response: Response, body: ChangePass
await provider.update_user(user)
# Re-issue cookie with new token_version
token = create_access_token(str(user.id), token_version=user.token_version)
# Re-issue cookie with new token_version. wid + role must be carried
# forward so the re-signed JWT still passes the AuthMiddleware
# workspace gate; ensure_default_workspace fills in for the (rare)
# case where the user predates PR4 and has not been backfilled.
workspace_id = await ensure_default_workspace(user)
token = create_access_token(
str(user.id),
token_version=user.token_version,
workspace_id=workspace_id,
role="owner",
)
_set_session_cookie(response, token, request)
return MessageResponse(message="Password changed successfully")
@router.get("/me", response_model=UserResponse)
@router.get("/me", response_model=UserMeResponse)
async def get_me(request: Request):
"""Get current authenticated user info."""
"""Get current authenticated user info, including the workspaces they belong to."""
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.workspace import WorkspaceRepository
from deerflow.persistence.workspace_membership import WorkspaceMembershipRepository
user = await get_current_user_from_request(request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role, needs_setup=user.needs_setup)
sf = get_session_factory()
workspaces: list[UserMeWorkspace] = []
if sf is not None:
ws_repo = WorkspaceRepository(sf)
m_repo = WorkspaceMembershipRepository(sf)
ws_rows = await ws_repo.list_by_user(user_id=str(user.id))
memberships = await m_repo.list_by_user(user_id=str(user.id))
role_by_ws = {m["workspace_id"]: m["role"] for m in memberships}
workspaces = [
UserMeWorkspace(
id=w["id"],
name=w["name"],
slug=w["slug"],
role=role_by_ws.get(w["id"], "member"),
)
for w in ws_rows
]
return UserMeResponse(
id=str(user.id),
email=user.email,
system_role=user.system_role,
needs_setup=user.needs_setup,
default_workspace_id=user.default_workspace_id,
workspaces=workspaces,
)
_SETUP_STATUS_COOLDOWN: dict[str, float] = {}
@@ -452,7 +555,14 @@ async def initialize_admin(request: Request, response: Response, body: Initializ
detail=AuthErrorResponse(code=AuthErrorCode.SYSTEM_ALREADY_INITIALIZED, message="System already initialized").model_dump(),
)
token = create_access_token(str(user.id), token_version=user.token_version)
workspace_id = await ensure_default_workspace(user)
token = create_access_token(
str(user.id),
token_version=user.token_version,
workspace_id=workspace_id,
role="owner",
)
_set_session_cookie(response, token, request)
return UserResponse(id=str(user.id), email=user.email, system_role=user.system_role)
@@ -11,6 +11,7 @@ from langgraph.runtime import Runtime
from deerflow.agents.thread_state import ThreadDataState
from deerflow.config.paths import Paths, get_paths
from deerflow.runtime.user_context import get_effective_user_id
from deerflow.runtime.workspace_context import get_effective_workspace_id
logger = logging.getLogger(__name__)
@@ -24,10 +25,15 @@ class ThreadDataMiddlewareState(AgentState):
class ThreadDataMiddleware(AgentMiddleware[ThreadDataMiddlewareState]):
"""Create thread data directories for each thread execution.
Creates the following directory structure:
- {base_dir}/threads/{thread_id}/user-data/workspace
- {base_dir}/threads/{thread_id}/user-data/uploads
- {base_dir}/threads/{thread_id}/user-data/outputs
PR6 routes thread storage through the workspace dimension. When a
workspace contextvar is set (production via AuthMiddleware; tests via
the autouse fixture), directories live at
``{base_dir}/workspaces/{wid}/threads/{thread_id}/user-data/{workspace,uploads,outputs}``.
In no-auth dev mode ``get_effective_workspace_id()`` returns
``"default"`` so the layout stays valid; the ``user_id`` falls through
to the same default constant. Either way thread state lives below a
workspace bucket, never directly under ``{base_dir}/threads`` (legacy)
or ``{base_dir}/users`` (PR4 layout).
Lifecycle Management:
- With lazy_init=True (default): Only compute paths, directories created on-demand
@@ -49,34 +55,18 @@ class ThreadDataMiddleware(AgentMiddleware[ThreadDataMiddlewareState]):
self._paths = Paths(base_dir) if base_dir else get_paths()
self._lazy_init = lazy_init
def _get_thread_paths(self, thread_id: str, user_id: str | None = None) -> dict[str, str]:
"""Get the paths for a thread's data directories.
Args:
thread_id: The thread ID.
user_id: Optional user ID for per-user path isolation.
Returns:
Dictionary with workspace_path, uploads_path, and outputs_path.
"""
def _get_thread_paths(self, thread_id: str, *, workspace_id: str, user_id: str) -> dict[str, str]:
return {
"workspace_path": str(self._paths.sandbox_work_dir(thread_id, user_id=user_id)),
"uploads_path": str(self._paths.sandbox_uploads_dir(thread_id, user_id=user_id)),
"outputs_path": str(self._paths.sandbox_outputs_dir(thread_id, user_id=user_id)),
"workspace_path": str(self._paths.sandbox_work_dir(thread_id, workspace_id=workspace_id)),
"uploads_path": str(self._paths.sandbox_uploads_dir(thread_id, workspace_id=workspace_id)),
"outputs_path": str(self._paths.sandbox_outputs_dir(thread_id, workspace_id=workspace_id)),
"user_id": user_id,
"workspace_id": workspace_id,
}
def _create_thread_directories(self, thread_id: str, user_id: str | None = None) -> dict[str, str]:
"""Create the thread data directories.
Args:
thread_id: The thread ID.
user_id: Optional user ID for per-user path isolation.
Returns:
Dictionary with the created directory paths.
"""
self._paths.ensure_thread_dirs(thread_id, user_id=user_id)
return self._get_thread_paths(thread_id, user_id=user_id)
def _create_thread_directories(self, thread_id: str, *, workspace_id: str, user_id: str) -> dict[str, str]:
self._paths.ensure_thread_dirs(thread_id, workspace_id=workspace_id)
return self._get_thread_paths(thread_id, workspace_id=workspace_id, user_id=user_id)
@override
def before_agent(self, state: ThreadDataMiddlewareState, runtime: Runtime) -> dict | None:
@@ -90,14 +80,15 @@ class ThreadDataMiddleware(AgentMiddleware[ThreadDataMiddlewareState]):
raise ValueError("Thread ID is required in runtime context or config.configurable")
user_id = get_effective_user_id()
workspace_id = get_effective_workspace_id()
if self._lazy_init:
# Lazy initialization: only compute paths, don't create directories
paths = self._get_thread_paths(thread_id, user_id=user_id)
paths = self._get_thread_paths(thread_id, workspace_id=workspace_id, user_id=user_id)
else:
# Eager initialization: create directories immediately
paths = self._create_thread_directories(thread_id, user_id=user_id)
logger.debug("Created thread data directories for thread %s", thread_id)
paths = self._create_thread_directories(thread_id, workspace_id=workspace_id, user_id=user_id)
logger.debug("Created thread data directories for thread %s under workspace %s", thread_id, workspace_id)
messages = list(state.get("messages", []))
last_message = messages[-1] if messages else None
+105 -79
View File
@@ -10,6 +10,7 @@ VIRTUAL_PATH_PREFIX = "/mnt/user-data"
_SAFE_THREAD_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$")
_SAFE_USER_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$")
_SAFE_WORKSPACE_ID_RE = re.compile(r"^[A-Za-z0-9_\-]+$")
def _default_local_base_dir() -> Path:
@@ -31,6 +32,13 @@ def _validate_user_id(user_id: str) -> str:
return user_id
def _validate_workspace_id(workspace_id: str) -> str:
"""Validate a workspace ID before using it in filesystem paths."""
if not _SAFE_WORKSPACE_ID_RE.match(workspace_id):
raise ValueError(f"Invalid workspace_id {workspace_id!r}: only alphanumeric characters, hyphens, and underscores are allowed.")
return workspace_id
def _join_host_path(base: str, *parts: str) -> str:
"""Join host filesystem path segments while preserving native style.
@@ -148,116 +156,129 @@ class Paths:
"""Legacy per-agent memory file: `{base_dir}/agents/{name}/memory.json`."""
return self.agent_dir(name) / "memory.json"
def user_dir(self, user_id: str) -> Path:
"""Directory for a specific user: `{base_dir}/users/{user_id}/`."""
def workspace_dir(self, workspace_id: str) -> Path:
"""Directory for a specific workspace: `{base_dir}/workspaces/{workspace_id}/`.
PR6 introduces this as the top-level isolation dimension. Per-user
state and per-thread state both live underneath their workspace so
a user with access to two workspaces never sees state bleed between
them on the filesystem.
"""
return self.base_dir / "workspaces" / _validate_workspace_id(workspace_id)
def user_dir(self, user_id: str, *, workspace_id: str | None = None) -> Path:
"""Directory for a specific user.
When ``workspace_id`` is provided (PR6+):
``{base_dir}/workspaces/{wid}/users/{user_id}/``
Otherwise (legacy layout):
``{base_dir}/users/{user_id}/``
"""
if workspace_id is not None:
return self.workspace_dir(workspace_id) / "users" / _validate_user_id(user_id)
return self.base_dir / "users" / _validate_user_id(user_id)
def user_memory_file(self, user_id: str) -> Path:
"""Per-user memory file: `{base_dir}/users/{user_id}/memory.json`."""
return self.user_dir(user_id) / "memory.json"
def user_memory_file(self, user_id: str, *, workspace_id: str | None = None) -> Path:
"""Per-user memory file under the active workspace (legacy without)."""
return self.user_dir(user_id, workspace_id=workspace_id) / "memory.json"
def user_agents_dir(self, user_id: str) -> Path:
"""Per-user root for that user's custom agents: `{base_dir}/users/{user_id}/agents/`."""
return self.user_dir(user_id) / "agents"
def user_agents_dir(self, user_id: str, *, workspace_id: str | None = None) -> Path:
"""Per-user root for custom agents under the active workspace."""
return self.user_dir(user_id, workspace_id=workspace_id) / "agents"
def user_agent_dir(self, user_id: str, agent_name: str) -> Path:
"""Per-user per-agent directory: `{base_dir}/users/{user_id}/agents/{name}/`."""
return self.user_agents_dir(user_id) / agent_name.lower()
def user_agent_dir(self, user_id: str, agent_name: str, *, workspace_id: str | None = None) -> Path:
"""Per-user per-agent directory under the active workspace."""
return self.user_agents_dir(user_id, workspace_id=workspace_id) / agent_name.lower()
def user_agent_memory_file(self, user_id: str, agent_name: str) -> Path:
"""Per-user per-agent memory: `{base_dir}/users/{user_id}/agents/{name}/memory.json`."""
return self.user_agent_dir(user_id, agent_name) / "memory.json"
def user_agent_memory_file(self, user_id: str, agent_name: str, *, workspace_id: str | None = None) -> Path:
"""Per-user per-agent memory file under the active workspace."""
return self.user_agent_dir(user_id, agent_name, workspace_id=workspace_id) / "memory.json"
def thread_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
def thread_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""
Host path for a thread's data.
When *user_id* is provided:
`{base_dir}/users/{user_id}/threads/{thread_id}/`
Otherwise (legacy layout):
`{base_dir}/threads/{thread_id}/`
Precedence — workspace beats user, both beat legacy:
This directory contains a `user-data/` subdirectory that is mounted
as `/mnt/user-data/` inside the sandbox.
* ``workspace_id`` given (PR6+):
``{base_dir}/workspaces/{wid}/threads/{thread_id}/``
* ``user_id`` only (legacy after user-isolation migration):
``{base_dir}/users/{user_id}/threads/{thread_id}/``
* neither (very legacy, pre-isolation):
``{base_dir}/threads/{thread_id}/``
The contained ``user-data/`` subdirectory is mounted as
``/mnt/user-data/`` inside the sandbox regardless of which form is
chosen — only the host-side parent differs.
Raises:
ValueError: If `thread_id` or `user_id` contains unsafe characters (path
separators or `..`) that could cause directory traversal.
ValueError: If any of the supplied ids contains unsafe characters.
"""
if workspace_id is not None:
return self.workspace_dir(workspace_id) / "threads" / _validate_thread_id(thread_id)
if user_id is not None:
return self.user_dir(user_id) / "threads" / _validate_thread_id(thread_id)
return self.base_dir / "users" / _validate_user_id(user_id) / "threads" / _validate_thread_id(thread_id)
return self.base_dir / "threads" / _validate_thread_id(thread_id)
def sandbox_work_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
def sandbox_work_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""
Host path for the agent's workspace directory.
Host: `{base_dir}/threads/{thread_id}/user-data/workspace/`
Sandbox: `/mnt/user-data/workspace/`
Host: ``{thread_dir}/user-data/workspace/``
Sandbox: ``/mnt/user-data/workspace/``
"""
return self.thread_dir(thread_id, user_id=user_id) / "user-data" / "workspace"
return self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id) / "user-data" / "workspace"
def sandbox_uploads_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
"""
Host path for user-uploaded files.
Host: `{base_dir}/threads/{thread_id}/user-data/uploads/`
Sandbox: `/mnt/user-data/uploads/`
"""
return self.thread_dir(thread_id, user_id=user_id) / "user-data" / "uploads"
def sandbox_uploads_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""Host path for user-uploaded files; sandbox: ``/mnt/user-data/uploads/``."""
return self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id) / "user-data" / "uploads"
def sandbox_outputs_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
"""
Host path for agent-generated artifacts.
Host: `{base_dir}/threads/{thread_id}/user-data/outputs/`
Sandbox: `/mnt/user-data/outputs/`
"""
return self.thread_dir(thread_id, user_id=user_id) / "user-data" / "outputs"
def sandbox_outputs_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""Host path for agent-generated artifacts; sandbox: ``/mnt/user-data/outputs/``."""
return self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id) / "user-data" / "outputs"
def acp_workspace_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
def acp_workspace_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""
Host path for the ACP workspace of a specific thread.
Host: `{base_dir}/threads/{thread_id}/acp-workspace/`
Sandbox: `/mnt/acp-workspace/`
Host path for the ACP workspace of a specific thread; sandbox: ``/mnt/acp-workspace/``.
Each thread gets its own isolated ACP workspace so that concurrent
sessions cannot read each other's ACP agent outputs.
"""
return self.thread_dir(thread_id, user_id=user_id) / "acp-workspace"
return self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id) / "acp-workspace"
def sandbox_user_data_dir(self, thread_id: str, *, user_id: str | None = None) -> Path:
"""
Host path for the user-data root.
Host: `{base_dir}/threads/{thread_id}/user-data/`
Sandbox: `/mnt/user-data/`
"""
return self.thread_dir(thread_id, user_id=user_id) / "user-data"
def sandbox_user_data_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> Path:
"""Host path for the user-data root; sandbox: ``/mnt/user-data/``."""
return self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id) / "user-data"
def host_thread_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_thread_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for a thread directory, preserving Windows path syntax."""
if workspace_id is not None:
return _join_host_path(self._host_base_dir_str(), "workspaces", _validate_workspace_id(workspace_id), "threads", _validate_thread_id(thread_id))
if user_id is not None:
return _join_host_path(self._host_base_dir_str(), "users", _validate_user_id(user_id), "threads", _validate_thread_id(thread_id))
return _join_host_path(self._host_base_dir_str(), "threads", _validate_thread_id(thread_id))
def host_sandbox_user_data_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_sandbox_user_data_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for a thread's user-data root."""
return _join_host_path(self.host_thread_dir(thread_id, user_id=user_id), "user-data")
return _join_host_path(self.host_thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id), "user-data")
def host_sandbox_work_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_sandbox_work_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for the workspace mount source."""
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, user_id=user_id), "workspace")
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, workspace_id=workspace_id, user_id=user_id), "workspace")
def host_sandbox_uploads_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_sandbox_uploads_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for the uploads mount source."""
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, user_id=user_id), "uploads")
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, workspace_id=workspace_id, user_id=user_id), "uploads")
def host_sandbox_outputs_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_sandbox_outputs_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for the outputs mount source."""
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, user_id=user_id), "outputs")
return _join_host_path(self.host_sandbox_user_data_dir(thread_id, workspace_id=workspace_id, user_id=user_id), "outputs")
def host_acp_workspace_dir(self, thread_id: str, *, user_id: str | None = None) -> str:
def host_acp_workspace_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> str:
"""Host path for the ACP workspace mount source."""
return _join_host_path(self.host_thread_dir(thread_id, user_id=user_id), "acp-workspace")
return _join_host_path(self.host_thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id), "acp-workspace")
def ensure_thread_dirs(self, thread_id: str, *, user_id: str | None = None) -> None:
def ensure_thread_dirs(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> None:
"""Create all standard sandbox directories for a thread.
Directories are created with mode 0o777 so that sandbox containers
@@ -271,24 +292,28 @@ class Paths:
ACP agent invocation.
"""
for d in [
self.sandbox_work_dir(thread_id, user_id=user_id),
self.sandbox_uploads_dir(thread_id, user_id=user_id),
self.sandbox_outputs_dir(thread_id, user_id=user_id),
self.acp_workspace_dir(thread_id, user_id=user_id),
self.sandbox_work_dir(thread_id, workspace_id=workspace_id, user_id=user_id),
self.sandbox_uploads_dir(thread_id, workspace_id=workspace_id, user_id=user_id),
self.sandbox_outputs_dir(thread_id, workspace_id=workspace_id, user_id=user_id),
self.acp_workspace_dir(thread_id, workspace_id=workspace_id, user_id=user_id),
]:
d.mkdir(parents=True, exist_ok=True)
d.chmod(0o777)
def delete_thread_dir(self, thread_id: str, *, user_id: str | None = None) -> None:
"""Delete all persisted data for a thread.
The operation is idempotent: missing thread directories are ignored.
"""
thread_dir = self.thread_dir(thread_id, user_id=user_id)
def delete_thread_dir(self, thread_id: str, *, workspace_id: str | None = None, user_id: str | None = None) -> None:
"""Delete all persisted data for a thread. Idempotent."""
thread_dir = self.thread_dir(thread_id, workspace_id=workspace_id, user_id=user_id)
if thread_dir.exists():
shutil.rmtree(thread_dir)
def resolve_virtual_path(self, thread_id: str, virtual_path: str, *, user_id: str | None = None) -> Path:
def resolve_virtual_path(
self,
thread_id: str,
virtual_path: str,
*,
workspace_id: str | None = None,
user_id: str | None = None,
) -> Path:
"""Resolve a sandbox virtual path to the actual host filesystem path.
Args:
@@ -296,7 +321,8 @@ class Paths:
virtual_path: Virtual path as seen inside the sandbox, e.g.
``/mnt/user-data/outputs/report.pdf``.
Leading slashes are stripped before matching.
user_id: Optional user ID for user-scoped path resolution.
workspace_id: Optional workspace ID for workspace-scoped resolution.
user_id: Optional user ID for legacy user-scoped resolution.
Returns:
The resolved absolute host filesystem path.
@@ -314,7 +340,7 @@ class Paths:
raise ValueError(f"Path must start with /{prefix}")
relative = stripped[len(prefix) :].lstrip("/")
base = self.sandbox_user_data_dir(thread_id, user_id=user_id).resolve()
base = self.sandbox_user_data_dir(thread_id, workspace_id=workspace_id, user_id=user_id).resolve()
actual = (base / relative).resolve()
try:
@@ -0,0 +1,18 @@
"""API key persistence — ORM model only (Stage 0 PR8).
An API key is the credential a service_account uses to call the
headless API. Each key has a public ``key_prefix`` (printed in audit
logs and used for quick lookup) and a ``key_hash`` (sha-256 of the
plaintext token, never reversed). Plaintext tokens are only ever
returned to the caller at create time.
PR8 introduces only the schema + ORM row class. Token generation,
hashing, scope parsing, rate limiting, and the API-key auth
middleware live in Stage 1 alongside the headless API surface.
"""
from __future__ import annotations
from deerflow.persistence.api_key.model import ApiKeyRow
__all__ = ["ApiKeyRow"]
@@ -0,0 +1,94 @@
"""ORM model for API keys (credentials owned by a service account)."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, ForeignKey, Index, Integer, String, text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class ApiKeyRow(Base):
__tablename__ = "api_keys"
id: Mapped[str] = mapped_column(
String(36),
primary_key=True,
comment="API key 主键,UUID 字符串(36 字符)",
)
service_account_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("service_accounts.id", ondelete="CASCADE"),
nullable=False,
comment="所属 service_accountservice_account 删除时级联清掉所有 api_key",
)
key_prefix: Mapped[str] = mapped_column(
String(16),
nullable=False,
unique=True,
comment="公开 prefix(如 'dfk_live_abc12345'),可在审计日志 / UI 中打印;全局唯一,撤销后亦不复用以避免审计混淆",
)
key_hash: Mapped[str] = mapped_column(
String(128),
nullable=False,
comment="完整 token 的 sha-256 hex64 字符;预留 128 以兼容未来更长哈希),plaintext token 仅在创建时返回给调用方",
)
name: Mapped[str] = mapped_column(
String(64),
nullable=False,
comment="key 的人类可读标签(如 'ci pipeline' / 'frontend prod'),同 service_account 内不强制唯一",
)
scopes: Mapped[str] = mapped_column(
String(1024),
nullable=False,
default="",
comment="scope 列表,逗号分隔字符串(如 'threads:read,threads:write');用 String 而非 PG text[] 以保 SQLite dev 兼容,Stage 2 切纯 PG 后可平滑迁",
)
rate_limit_rpm: Mapped[int | None] = mapped_column(
Integer,
nullable=True,
comment="每分钟请求数限制;NULL 表示走该 service_account 的默认限速(Stage 1 起生效)",
)
expires_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
comment="过期时间(UTC);NULL = 不过期",
)
last_used_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
comment="最近一次成功鉴权时间(UTC);Stage 1 鉴权中间件每次更新",
)
revoked_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
comment="撤销时间(UTC);NULL = 仍然有效。被撤销的 key 不删行(保留审计),但鉴权层据此拒绝",
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="创建时间(UTC",
)
__table_args__ = (
Index("idx_api_keys_sa", "service_account_id"),
# 部分索引:只索引活跃 keyrevoked_at IS NULL),鉴权热路径走 prefix lookup
# 撤销后的 key 不进活跃索引以减小热索引大小。SQLite + Postgres 均支持
# WHERE 子句的部分索引;双驱动维护两套等价 where 表达式。
Index(
"idx_api_keys_active",
"key_prefix",
sqlite_where=text("revoked_at IS NULL"),
postgresql_where=text("revoked_at IS NULL"),
),
{
"comment": (
"API key 表(headless API 凭证)。每行属于唯一 service_accountkey_prefix 全局唯一可在日志中打印,"
"key_hash 是完整 token 的 sha-256plaintext token 只在创建时返给调用方。撤销保留行(revoked_at 非空),"
"活跃 key 走部分索引 idx_api_keys_active 加速鉴权热路径。Stage 0 仅落 schemaStage 1 起接鉴权 + 限速。"
)
},
)
@@ -0,0 +1,16 @@
"""External user persistence — ORM model only (Stage 0 PR8).
An external user represents the end-user identity that a
service_account passes through on each call (typically via an
``X-External-User-Id`` header). The row is upserted each time a
new ``external_id`` is seen under a given service_account.
PR8 introduces only the schema + ORM row class. The upsert logic,
header parsing, and quota attribution all live in Stage 1.
"""
from __future__ import annotations
from deerflow.persistence.external_user.model import ExternalUserRow
__all__ = ["ExternalUserRow"]
@@ -0,0 +1,71 @@
"""ORM model for external users (end-user identities passed through a service account)."""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import JSON, DateTime, ForeignKey, String, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class ExternalUserRow(Base):
__tablename__ = "external_users"
id: Mapped[str] = mapped_column(
String(36),
primary_key=True,
comment="external_user 主键,UUID 字符串(36 字符)",
)
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspace(冗余存储——可经 service_account 间接得到,但直接存以加速 workspace-scope 查询);workspace 删除时级联",
)
service_account_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("service_accounts.id", ondelete="CASCADE"),
nullable=False,
comment="passthrough 的 service_accountservice_account 删除时级联",
)
external_id: Mapped[str] = mapped_column(
String(128),
nullable=False,
comment="终端调用方传入的 X-External-User-Id(最多 128 字符;推荐 UUID / opaque token,不要塞 PII",
)
display_name: Mapped[str | None] = mapped_column(
String(128),
nullable=True,
comment="可选显示名(如 'alice@customer.com');仅用于 admin UI 展示,不参与鉴权",
)
metadata_json: Mapped[dict[str, Any]] = mapped_column(
JSON,
nullable=False,
default=dict,
comment="任意 JSON 附属信息(plan tier / region / 自定义 tag);Stage 1 由 upsert 调用方写入",
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="首次见到该 external_id 的时间(UTC",
)
last_seen_at: Mapped[datetime | None] = mapped_column(
DateTime(timezone=True),
nullable=True,
comment="最近一次该 external_id 触发请求的时间(UTC);Stage 1 鉴权层每次更新",
)
__table_args__ = (
UniqueConstraint("service_account_id", "external_id", name="uq_external_users_sa_external"),
{
"comment": (
"终端用户身份表(passthrough 模式下的 end-user)。每行由 service_account 的鉴权中间件 upsert——同一 "
"(service_account_id, external_id) 组合只存一行。workspace_id 冗余存储以加速跨 SA 的 workspace-scope 聚合查询。"
"Stage 0 仅落 schemaStage 1 起接 upsert / 配额聚合。"
)
},
)
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, String, Text, UniqueConstraint
from sqlalchemy import DateTime, ForeignKey, String, Text, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -13,20 +13,22 @@ from deerflow.persistence.base import Base
class FeedbackRow(Base):
__tablename__ = "feedback"
__table_args__ = (UniqueConstraint("thread_id", "run_id", "user_id", name="uq_feedback_thread_run_user"),)
__table_args__ = (
UniqueConstraint("thread_id", "run_id", "user_id", name="uq_feedback_thread_run_user"),
{"comment": "用户对运行结果的反馈(点赞/点踩 + 文字评论),(thread, run, user) 唯一"},
)
feedback_id: Mapped[str] = mapped_column(String(64), primary_key=True)
run_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
thread_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
user_id: Mapped[str | None] = mapped_column(String(64), index=True)
message_id: Mapped[str | None] = mapped_column(String(64))
# message_id is an optional RunEventStore event identifier —
# allows feedback to target a specific message or the entire run
rating: Mapped[int] = mapped_column(nullable=False)
# +1 (thumbs-up) or -1 (thumbs-down)
comment: Mapped[str | None] = mapped_column(Text)
# Optional text feedback from the user
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
feedback_id: Mapped[str] = mapped_column(String(64), primary_key=True, comment="反馈主键")
run_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True, comment="关联的运行 IDruns.run_id")
thread_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True, comment="关联的会话 IDthreads_meta.thread_id")
user_id: Mapped[str | None] = mapped_column(String(64), index=True, comment="反馈作者;为 NULL 表示历史无主数据")
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspace。PR5 引入时 nullable 用于回填;alembic 0003 + PR6 仓储接入完成后 NOT NULL",
)
message_id: Mapped[str | None] = mapped_column(String(64), comment="可选的 RunEventStore 事件 ID;为 NULL 表示针对整次运行而非单条消息")
rating: Mapped[int] = mapped_column(nullable=False, comment="评分:+1 点赞,-1 点踩")
comment: Mapped[str | None] = mapped_column(Text, comment="可选的文字评论")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), comment="创建时间(UTC")
@@ -13,6 +13,13 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import (
_AutoSentinel as _WorkspaceAutoSentinel,
)
from deerflow.runtime.workspace_context import (
resolve_workspace_id,
)
class FeedbackRepository:
@@ -34,6 +41,7 @@ class FeedbackRepository:
thread_id: str,
rating: int,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
message_id: str | None = None,
comment: str | None = None,
) -> dict:
@@ -41,11 +49,13 @@ class FeedbackRepository:
if rating not in (1, -1):
raise ValueError(f"rating must be +1 or -1, got {rating}")
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.create")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.create")
row = FeedbackRow(
feedback_id=str(uuid.uuid4()),
run_id=run_id,
thread_id=thread_id,
user_id=resolved_user_id,
workspace_id=resolved_workspace_id,
message_id=message_id,
rating=rating,
comment=comment,
@@ -62,12 +72,16 @@ class FeedbackRepository:
feedback_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> dict | None:
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.get")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.get")
async with self._sf() as session:
row = await session.get(FeedbackRow, feedback_id)
if row is None:
return None
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return None
if resolved_user_id is not None and row.user_id != resolved_user_id:
return None
return self._row_to_dict(row)
@@ -79,9 +93,13 @@ class FeedbackRepository:
*,
limit: int = 100,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> list[dict]:
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.list_by_run")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.list_by_run")
stmt = select(FeedbackRow).where(FeedbackRow.thread_id == thread_id, FeedbackRow.run_id == run_id)
if resolved_workspace_id is not None:
stmt = stmt.where(FeedbackRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(FeedbackRow.user_id == resolved_user_id)
stmt = stmt.order_by(FeedbackRow.created_at.asc()).limit(limit)
@@ -95,9 +113,13 @@ class FeedbackRepository:
*,
limit: int = 100,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> list[dict]:
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.list_by_thread")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.list_by_thread")
stmt = select(FeedbackRow).where(FeedbackRow.thread_id == thread_id)
if resolved_workspace_id is not None:
stmt = stmt.where(FeedbackRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(FeedbackRow.user_id == resolved_user_id)
stmt = stmt.order_by(FeedbackRow.created_at.asc()).limit(limit)
@@ -110,12 +132,16 @@ class FeedbackRepository:
feedback_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> bool:
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.delete")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.delete")
async with self._sf() as session:
row = await session.get(FeedbackRow, feedback_id)
if row is None:
return False
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return False
if resolved_user_id is not None and row.user_id != resolved_user_id:
return False
await session.delete(row)
@@ -129,18 +155,22 @@ class FeedbackRepository:
thread_id: str,
rating: int,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
comment: str | None = None,
) -> dict:
"""Create or update feedback for (thread_id, run_id, user_id). rating must be +1 or -1."""
if rating not in (1, -1):
raise ValueError(f"rating must be +1 or -1, got {rating}")
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.upsert")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.upsert")
async with self._sf() as session:
stmt = select(FeedbackRow).where(
FeedbackRow.thread_id == thread_id,
FeedbackRow.run_id == run_id,
FeedbackRow.user_id == resolved_user_id,
)
if resolved_workspace_id is not None:
stmt = stmt.where(FeedbackRow.workspace_id == resolved_workspace_id)
result = await session.execute(stmt)
row = result.scalar_one_or_none()
if row is not None:
@@ -153,6 +183,7 @@ class FeedbackRepository:
run_id=run_id,
thread_id=thread_id,
user_id=resolved_user_id,
workspace_id=resolved_workspace_id,
rating=rating,
comment=comment,
created_at=datetime.now(UTC),
@@ -168,15 +199,19 @@ class FeedbackRepository:
thread_id: str,
run_id: str,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> bool:
"""Delete the current user's feedback for a run. Returns True if a record was deleted."""
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.delete_by_run")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.delete_by_run")
async with self._sf() as session:
stmt = select(FeedbackRow).where(
FeedbackRow.thread_id == thread_id,
FeedbackRow.run_id == run_id,
FeedbackRow.user_id == resolved_user_id,
)
if resolved_workspace_id is not None:
stmt = stmt.where(FeedbackRow.workspace_id == resolved_workspace_id)
result = await session.execute(stmt)
row = result.scalar_one_or_none()
if row is None:
@@ -190,10 +225,14 @@ class FeedbackRepository:
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> dict[str, dict]:
"""Return feedback grouped by run_id for a thread: {run_id: feedback_dict}."""
resolved_user_id = resolve_user_id(user_id, method_name="FeedbackRepository.list_by_thread_grouped")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="FeedbackRepository.list_by_thread_grouped")
stmt = select(FeedbackRow).where(FeedbackRow.thread_id == thread_id)
if resolved_workspace_id is not None:
stmt = stmt.where(FeedbackRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(FeedbackRow.user_id == resolved_user_id)
async with self._sf() as session:
@@ -0,0 +1,44 @@
"""users.default_workspace_id column + FK to workspaces
Revision ID: 0001_users_default_workspace
Revises: None
Create Date: 2026-05-12
First Alembic revision for the DeerFlow application schema. Adds
`users.default_workspace_id` so a newly-registered user can be sent
back to their default workspace on next login without consulting the
memberships table.
Existing deployments created `users` via `metadata.create_all()` without
this column; running `alembic upgrade head` on those DBs will simply add
the column (ON DELETE SET NULL FK), no data backfill needed.
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
# Alembic identifiers.
revision: str = "0001_users_default_workspace"
down_revision: str | None = None
branch_labels: str | None = None
depends_on: str | None = None
def upgrade() -> None:
with op.batch_alter_table("users") as batch:
batch.add_column(sa.Column("default_workspace_id", sa.String(36), nullable=True))
batch.create_foreign_key(
"fk_users_default_workspace",
"workspaces",
["default_workspace_id"],
["id"],
ondelete="SET NULL",
)
def downgrade() -> None:
with op.batch_alter_table("users") as batch:
batch.drop_constraint("fk_users_default_workspace", type_="foreignkey")
batch.drop_column("default_workspace_id")
@@ -0,0 +1,70 @@
"""Business tables: nullable workspace_id + FK to workspaces
Revision ID: 0002_business_tables_workspace
Revises: 0001_users_default_workspace
Create Date: 2026-05-13
Stage 0 PR5 step 1/2 (the second step lives in revision 0003).
This revision adds a *nullable* ``workspace_id`` column to the four
business tables that need tenancy scoping:
* ``threads_meta``
* ``runs``
* ``feedback``
* ``run_events``
The column is nullable here on purpose — running ``upgrade`` on a
database with existing rows leaves those rows with ``workspace_id = NULL``
until ``scripts/backfill_workspace_id.py`` populates them. Once the
backfill finishes, revision 0003 flips the column to ``NOT NULL`` and
adds the threads_meta ``(workspace_id, thread_id)`` UNIQUE index.
A composite index ``idx_threads_meta_workspace_user_updated`` is added
on ``threads_meta`` to support the common "list a workspace's threads
for a user, newest first" access pattern that PR6 routers will rely on.
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision: str = "0002_business_tables_workspace"
down_revision: str | None = "0001_users_default_workspace"
branch_labels: str | None = None
depends_on: str | None = None
# Tables that receive the new column. The order matters only for human
# readability in migration logs — there are no inter-table data deps
# during ALTER, and the FK references workspaces (introduced in PR3) which
# is already present at this point.
_BUSINESS_TABLES = ("threads_meta", "runs", "feedback", "run_events")
def upgrade() -> None:
for table in _BUSINESS_TABLES:
with op.batch_alter_table(table) as batch:
batch.add_column(sa.Column("workspace_id", sa.String(36), nullable=True))
batch.create_foreign_key(
f"fk_{table}_workspace_id",
"workspaces",
["workspace_id"],
["id"],
ondelete="CASCADE",
)
op.create_index(
"idx_threads_meta_workspace_user_updated",
"threads_meta",
["workspace_id", "user_id", "updated_at"],
)
def downgrade() -> None:
op.drop_index("idx_threads_meta_workspace_user_updated", table_name="threads_meta")
for table in reversed(_BUSINESS_TABLES):
with op.batch_alter_table(table) as batch:
batch.drop_constraint(f"fk_{table}_workspace_id", type_="foreignkey")
batch.drop_column("workspace_id")
@@ -0,0 +1,62 @@
"""Business tables: workspace_id NOT NULL + UNIQUE(workspace_id, thread_id)
Revision ID: 0003_business_tables_workspace_not_null
Revises: 0002_business_tables_workspace
Create Date: 2026-05-13
Stage 0 PR5 step 2/2. Flips ``workspace_id`` on the four business
tables to ``NOT NULL`` and adds the threads_meta ``(workspace_id,
thread_id)`` UNIQUE index promised in workspace-schema-design §4.
**Refuses to upgrade** if any of the four tables still has rows with
``workspace_id IS NULL`` — the operator must run
``scripts/backfill_workspace_id.py`` first. Doing the NOT NULL ALTER
with stragglers in place would either fail (Postgres) or silently
corrupt SQLite tables via ``batch_alter_table`` rebuilds.
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision: str = "0003_business_tables_workspace_not_null"
down_revision: str | None = "0002_business_tables_workspace"
branch_labels: str | None = None
depends_on: str | None = None
_BUSINESS_TABLES = ("threads_meta", "runs", "feedback", "run_events")
class _BackfillRequiredError(RuntimeError):
"""Raised when null workspace_id rows remain at the start of upgrade."""
def upgrade() -> None:
conn = op.get_bind()
for table in _BUSINESS_TABLES:
# Quoted identifier is fine here — table names are constants in this
# module, no operator input reaches the SQL.
count = conn.execute(sa.text(f"SELECT count(*) FROM {table} WHERE workspace_id IS NULL")).scalar() or 0
if count > 0:
raise _BackfillRequiredError(
f"Cannot ALTER {table}.workspace_id to NOT NULL: {count} row(s) still have workspace_id=NULL. Run `python scripts/backfill_workspace_id.py` first.",
)
for table in _BUSINESS_TABLES:
with op.batch_alter_table(table) as batch:
batch.alter_column("workspace_id", existing_type=sa.String(36), nullable=False)
op.create_index(
"idx_threads_meta_workspace_thread",
"threads_meta",
["workspace_id", "thread_id"],
unique=True,
)
def downgrade() -> None:
op.drop_index("idx_threads_meta_workspace_thread", table_name="threads_meta")
for table in reversed(_BUSINESS_TABLES):
with op.batch_alter_table(table) as batch:
batch.alter_column("workspace_id", existing_type=sa.String(36), nullable=True)
@@ -8,16 +8,37 @@ The actual ORM classes have moved to entity-specific subpackages:
- ``deerflow.persistence.run``
- ``deerflow.persistence.feedback``
- ``deerflow.persistence.user``
- ``deerflow.persistence.workspace`` (Stage 0 PR3)
- ``deerflow.persistence.workspace_membership`` (Stage 0 PR3)
- ``deerflow.persistence.service_account`` (Stage 0 PR8)
- ``deerflow.persistence.api_key`` (Stage 0 PR8)
- ``deerflow.persistence.external_user`` (Stage 0 PR8)
``RunEventRow`` remains in ``deerflow.persistence.models.run_event`` because
its storage implementation lives in ``deerflow.runtime.events.store.db`` and
there is no matching entity directory.
"""
from deerflow.persistence.api_key.model import ApiKeyRow
from deerflow.persistence.external_user.model import ExternalUserRow
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.service_account.model import ServiceAccountRow
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
__all__ = ["FeedbackRow", "RunEventRow", "RunRow", "ThreadMetaRow", "UserRow"]
__all__ = [
"ApiKeyRow",
"ExternalUserRow",
"FeedbackRow",
"RunEventRow",
"RunRow",
"ServiceAccountRow",
"ThreadMetaRow",
"UserRow",
"WorkspaceMembershipRow",
"WorkspaceRow",
]
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import JSON, DateTime, Index, String, Text, UniqueConstraint
from sqlalchemy import JSON, DateTime, ForeignKey, Index, String, Text, UniqueConstraint
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -13,23 +13,31 @@ from deerflow.persistence.base import Base
class RunEventRow(Base):
__tablename__ = "run_events"
id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
thread_id: Mapped[str] = mapped_column(String(64), nullable=False)
run_id: Mapped[str] = mapped_column(String(64), nullable=False)
# Owner of the conversation this event belongs to. Nullable for data
# created before auth was introduced; populated by auth middleware on
# new writes and by the boot-time orphan migration on existing rows.
user_id: Mapped[str | None] = mapped_column(String(64), nullable=True, index=True)
event_type: Mapped[str] = mapped_column(String(32), nullable=False)
category: Mapped[str] = mapped_column(String(16), nullable=False)
# "message" | "trace" | "lifecycle"
content: Mapped[str] = mapped_column(Text, default="")
event_metadata: Mapped[dict] = mapped_column(JSON, default=dict)
seq: Mapped[int] = mapped_column(nullable=False)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
id: Mapped[int] = mapped_column(primary_key=True, autoincrement=True, comment="自增主键")
thread_id: Mapped[str] = mapped_column(String(64), nullable=False, comment="所属会话 IDthreads_meta.thread_id")
run_id: Mapped[str] = mapped_column(String(64), nullable=False, comment="所属运行 IDruns.run_id")
user_id: Mapped[str | None] = mapped_column(
String(64),
nullable=True,
index=True,
comment="会话所有者;为 NULL 表示鉴权引入之前的历史数据,新写入由 auth 中间件填充,启动期 orphan 迁移会回填存量",
)
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspace。PR5 引入时 nullable 用于回填;alembic 0003 + PR6 仓储接入完成后 NOT NULL",
)
event_type: Mapped[str] = mapped_column(String(32), nullable=False, comment="事件子类型(具体含义由 category 决定,如 ai_message_chunk、tool_call、run_started")
category: Mapped[str] = mapped_column(String(16), nullable=False, comment='事件大类:"message" 消息 / "trace" 追踪 / "lifecycle" 生命周期')
content: Mapped[str] = mapped_column(Text, default="", comment="事件文本内容(消息体、错误、状态字符串等)")
event_metadata: Mapped[dict] = mapped_column(JSON, default=dict, comment="事件结构化元数据(JSON),随 event_type 而异")
seq: Mapped[int] = mapped_column(nullable=False, comment="在 thread 内的全局递增序号;与 thread_id 组合唯一")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), comment="创建时间(UTC")
__table_args__ = (
UniqueConstraint("thread_id", "seq", name="uq_events_thread_seq"),
Index("ix_events_thread_cat_seq", "thread_id", "category", "seq"),
Index("ix_events_run", "thread_id", "run_id", "seq"),
{"comment": "运行事件流(消息/追踪/生命周期事件按 seq 顺序追加,是消息回放与审计的真源)"},
)
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import JSON, DateTime, Index, String, Text
from sqlalchemy import JSON, DateTime, ForeignKey, Index, String, Text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -13,37 +13,49 @@ from deerflow.persistence.base import Base
class RunRow(Base):
__tablename__ = "runs"
run_id: Mapped[str] = mapped_column(String(64), primary_key=True)
thread_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True)
assistant_id: Mapped[str | None] = mapped_column(String(128))
user_id: Mapped[str | None] = mapped_column(String(64), index=True)
status: Mapped[str] = mapped_column(String(20), default="pending")
# "pending" | "running" | "success" | "error" | "timeout" | "interrupted"
run_id: Mapped[str] = mapped_column(String(64), primary_key=True, comment="运行主键")
thread_id: Mapped[str] = mapped_column(String(64), nullable=False, index=True, comment="所属会话 IDthreads_meta.thread_id")
assistant_id: Mapped[str | None] = mapped_column(String(128), comment="使用的 Assistant ID(自定义智能体名);为 NULL 表示默认 lead agent")
user_id: Mapped[str | None] = mapped_column(String(64), index=True, comment="发起本次运行的用户 ID")
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspace。PR5 引入时 nullable 用于回填;alembic 0003 + PR6 仓储接入完成后 NOT NULL",
)
status: Mapped[str] = mapped_column(
String(20),
default="pending",
comment='运行状态:"pending" / "running" / "success" / "error" / "timeout" / "interrupted"',
)
model_name: Mapped[str | None] = mapped_column(String(128), comment="本次运行的主模型名(来自 config.yaml.models[*].name")
multitask_strategy: Mapped[str] = mapped_column(
String(20),
default="reject",
comment='并发策略:同一 thread 已有运行时怎么处理("reject" / "interrupt" / "rollback" / "enqueue"',
)
metadata_json: Mapped[dict] = mapped_column(JSON, default=dict, comment="运行级元数据(JSON),如 channel/source 等")
kwargs_json: Mapped[dict] = mapped_column(JSON, default=dict, comment="提交运行时的额外参数(JSON),如 thinking_enabled、tool 配置等")
error: Mapped[str | None] = mapped_column(Text, comment="运行失败时的错误文本;成功时为 NULL")
model_name: Mapped[str | None] = mapped_column(String(128))
multitask_strategy: Mapped[str] = mapped_column(String(20), default="reject")
metadata_json: Mapped[dict] = mapped_column(JSON, default=dict)
kwargs_json: Mapped[dict] = mapped_column(JSON, default=dict)
error: Mapped[str | None] = mapped_column(Text)
message_count: Mapped[int] = mapped_column(default=0, comment="本次运行产生的消息总数(便利字段,避免列表页查 RunEventStore")
first_human_message: Mapped[str | None] = mapped_column(Text, comment="首条用户消息文本预览(用于列表展示)")
last_ai_message: Mapped[str | None] = mapped_column(Text, comment="末条 AI 消息文本预览(用于列表展示)")
# Convenience fields (for listing pages without querying RunEventStore)
message_count: Mapped[int] = mapped_column(default=0)
first_human_message: Mapped[str | None] = mapped_column(Text)
last_ai_message: Mapped[str | None] = mapped_column(Text)
total_input_tokens: Mapped[int] = mapped_column(default=0, comment="累计输入 token 数(运行结束时由 RunJournal 落盘)")
total_output_tokens: Mapped[int] = mapped_column(default=0, comment="累计输出 token 数")
total_tokens: Mapped[int] = mapped_column(default=0, comment="累计 token 总数 = input + output")
llm_call_count: Mapped[int] = mapped_column(default=0, comment="累计 LLM 调用次数")
lead_agent_tokens: Mapped[int] = mapped_column(default=0, comment="主 agent 自身消耗的 token 数")
subagent_tokens: Mapped[int] = mapped_column(default=0, comment="子 agenttask 工具委派)消耗的 token 数")
middleware_tokens: Mapped[int] = mapped_column(default=0, comment="中间件(如 summarization、title)消耗的 token 数")
# Token usage (accumulated in-memory by RunJournal, written on run completion)
total_input_tokens: Mapped[int] = mapped_column(default=0)
total_output_tokens: Mapped[int] = mapped_column(default=0)
total_tokens: Mapped[int] = mapped_column(default=0)
llm_call_count: Mapped[int] = mapped_column(default=0)
lead_agent_tokens: Mapped[int] = mapped_column(default=0)
subagent_tokens: Mapped[int] = mapped_column(default=0)
middleware_tokens: Mapped[int] = mapped_column(default=0)
follow_up_to_run_id: Mapped[str | None] = mapped_column(String(64), comment="续接的上一次运行 ID(用于'重新生成'/'继续'等链式调用)")
# Follow-up association
follow_up_to_run_id: Mapped[str | None] = mapped_column(String(64))
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), comment="创建时间(UTC")
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), comment="最近更新时间(UTC,写入时自动更新)")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
__table_args__ = (Index("ix_runs_thread_status", "thread_id", "status"),)
__table_args__ = (
Index("ix_runs_thread_status", "thread_id", "status"),
{"comment": "运行(一次完整 agent 执行)的元数据 + 累计 token 指标"},
)
@@ -17,6 +17,13 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.run.model import RunRow
from deerflow.runtime.runs.store.base import RunStore
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import (
_AutoSentinel as _WorkspaceAutoSentinel,
)
from deerflow.runtime.workspace_context import (
resolve_workspace_id,
)
class RunRepository(RunStore):
@@ -70,6 +77,7 @@ class RunRepository(RunStore):
thread_id,
assistant_id=None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
status="pending",
multitask_strategy="reject",
metadata=None,
@@ -79,12 +87,14 @@ class RunRepository(RunStore):
follow_up_to_run_id=None,
):
resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.put")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="RunRepository.put")
now = datetime.now(UTC)
row = RunRow(
run_id=run_id,
thread_id=thread_id,
assistant_id=assistant_id,
user_id=resolved_user_id,
workspace_id=resolved_workspace_id,
status=status,
multitask_strategy=multitask_strategy,
metadata_json=self._safe_json(metadata) or {},
@@ -103,12 +113,16 @@ class RunRepository(RunStore):
run_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.get")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="RunRepository.get")
async with self._sf() as session:
row = await session.get(RunRow, run_id)
if row is None:
return None
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return None
if resolved_user_id is not None and row.user_id != resolved_user_id:
return None
return self._row_to_dict(row)
@@ -118,10 +132,14 @@ class RunRepository(RunStore):
thread_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
limit=100,
):
resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.list_by_thread")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="RunRepository.list_by_thread")
stmt = select(RunRow).where(RunRow.thread_id == thread_id)
if resolved_workspace_id is not None:
stmt = stmt.where(RunRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(RunRow.user_id == resolved_user_id)
stmt = stmt.order_by(RunRow.created_at.desc()).limit(limit)
@@ -142,12 +160,16 @@ class RunRepository(RunStore):
run_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="RunRepository.delete")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="RunRepository.delete")
async with self._sf() as session:
row = await session.get(RunRow, run_id)
if row is None:
return
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return
if resolved_user_id is not None and row.user_id != resolved_user_id:
return
await session.delete(row)
@@ -0,0 +1,17 @@
"""Service account persistence — ORM model only (Stage 0 PR8).
A service account is a non-human principal that lives inside a workspace
and authenticates via API keys rather than email + password. Each
service account belongs to exactly one workspace and is created by a
human user (``created_by``).
PR8 introduces only the schema + ORM row class. Repository, API-key
authentication middleware, and the ``@require_permission`` scope
upgrade live in Stage 1 alongside the headless API surface.
"""
from __future__ import annotations
from deerflow.persistence.service_account.model import ServiceAccountRow
__all__ = ["ServiceAccountRow"]
@@ -0,0 +1,83 @@
"""ORM model for service accounts (non-human principals inside a workspace)."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, ForeignKey, Index, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class ServiceAccountRow(Base):
__tablename__ = "service_accounts"
id: Mapped[str] = mapped_column(
String(36),
primary_key=True,
comment="服务账号主键,UUID 字符串(36 字符),与 users.id 类型对齐",
)
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspaceworkspace 删除时级联清掉所有 service_account(连同其 api_keys / external_users",
)
name: Mapped[str] = mapped_column(
String(64),
nullable=False,
comment="服务账号显示名(同 workspace 内不强制唯一;Stage 1 可由 admin UI 重复使用同名 + 不同 key)",
)
role: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="member",
comment='服务账号在 workspace 内的角色字符串:Stage 0 仅支持 "member"Stage 2 RBAC 打开 "admin"/"viewer"。用 String(16) 而非 enum 以便未来扩枚举值不动 schema',
)
identity_mode: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="collapsed",
comment=(
'身份模式三态:"collapsed"(所有调用 collapse 到该 service_account;不记录 external_user/'
' "external_passthrough"(每次调用必带 X-External-User-Id,写入 external_users 表)/'
' "both"(带就写、不带就 collapse)。Stage 1 API key 鉴权层据此分流'
),
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="active",
comment='状态:"active"(正常)/ "suspended"admin 暂停)/ "deleted"(软删;保留审计)',
)
created_by: Mapped[str] = mapped_column(
String(36),
ForeignKey("users.id", ondelete="RESTRICT"),
nullable=False,
comment="创建者 user_id;删除该 user 时 RESTRICT 阻拦(必须先转移或删除该 user 名下所有 service_account",
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="创建时间(UTC",
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
comment="最近更新时间(UTC,写入时自动更新)",
)
__table_args__ = (
Index("idx_service_accounts_workspace", "workspace_id", "status"),
{
"comment": (
"服务账号表(headless API 的非人身份)。每个 service_account 属于唯一 workspace"
"通过 api_keys 表的 API key 鉴权调用 Gatewayidentity_mode 控制是否在 external_users 表"
"记录终端用户身份。Stage 0 仅落 schemaStage 1 起接 API key 鉴权 + 路由 scope 升级。"
)
},
)
@@ -4,12 +4,18 @@ Implementations:
- ThreadMetaRepository: SQL-backed (sqlite / postgres via SQLAlchemy)
- MemoryThreadMetaStore: wraps LangGraph BaseStore (memory mode)
All mutating and querying methods accept a ``user_id`` parameter with
three-state semantics (see :mod:`deerflow.runtime.user_context`):
All mutating and querying methods accept both a ``user_id`` parameter
(member-scoped owner check) and a ``workspace_id`` parameter (tenant
scope). Both follow three-state semantics:
- ``AUTO`` (default): resolve from the request-scoped contextvar.
- Explicit ``str``: use the provided value verbatim.
- Explicit ``None``: bypass owner filtering (migration/CLI only).
- Explicit ``None``: bypass that filter (migration / CLI only).
The workspace scope is the **outer** boundary: a row in workspace A is
unreachable from any user_id under workspace B. ``check_access`` returns
False on cross-workspace mismatch so the route layer can convert it into
a 404 instead of leaking thread existence across tenants.
"""
from __future__ import annotations
@@ -17,6 +23,8 @@ from __future__ import annotations
import abc
from deerflow.runtime.user_context import AUTO, _AutoSentinel
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import _AutoSentinel as _WorkspaceAutoSentinel
class ThreadMetaStore(abc.ABC):
@@ -27,13 +35,20 @@ class ThreadMetaStore(abc.ABC):
*,
assistant_id: str | None = None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
display_name: str | None = None,
metadata: dict | None = None,
) -> dict:
pass
@abc.abstractmethod
async def get(self, thread_id: str, *, user_id: str | None | _AutoSentinel = AUTO) -> dict | None:
async def get(
self,
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> dict | None:
pass
@abc.abstractmethod
@@ -45,32 +60,72 @@ class ThreadMetaStore(abc.ABC):
limit: int = 100,
offset: int = 0,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> list[dict]:
pass
@abc.abstractmethod
async def update_display_name(self, thread_id: str, display_name: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
async def update_display_name(
self,
thread_id: str,
display_name: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
pass
@abc.abstractmethod
async def update_status(self, thread_id: str, status: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
async def update_status(
self,
thread_id: str,
status: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
pass
@abc.abstractmethod
async def update_metadata(self, thread_id: str, metadata: dict, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
async def update_metadata(
self,
thread_id: str,
metadata: dict,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
"""Merge ``metadata`` into the thread's metadata field.
Existing keys are overwritten by the new values; keys absent from
``metadata`` are preserved. No-op if the thread does not exist
or the owner check fails.
or the user/workspace check fails.
"""
pass
@abc.abstractmethod
async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
"""Check if ``user_id`` has access to ``thread_id``."""
async def check_access(
self,
thread_id: str,
user_id: str,
workspace_id: str,
*,
require_existing: bool = False,
) -> bool:
"""Check whether ``user_id`` (in ``workspace_id``) can access ``thread_id``.
Cross-workspace access returns ``False`` unconditionally so the
decorator layer can convert it into a 404 — never leak the
existence of a thread that belongs to a different tenant.
"""
pass
@abc.abstractmethod
async def delete(self, thread_id: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
async def delete(
self,
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
pass
@@ -13,6 +13,13 @@ from langgraph.store.base import BaseStore
from deerflow.persistence.thread_meta.base import ThreadMetaStore
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import (
_AutoSentinel as _WorkspaceAutoSentinel,
)
from deerflow.runtime.workspace_context import (
resolve_workspace_id,
)
from deerflow.utils.time import coerce_iso, now_iso
THREADS_NS: tuple[str, ...] = ("threads",)
@@ -26,15 +33,19 @@ class MemoryThreadMetaStore(ThreadMetaStore):
self,
thread_id: str,
user_id: str | None | _AutoSentinel,
workspace_id: str | None | _WorkspaceAutoSentinel,
method_name: str,
) -> dict | None:
"""Fetch a record and verify ownership. Returns a mutable copy, or None."""
resolved = resolve_user_id(user_id, method_name=method_name)
"""Fetch a record and verify workspace + ownership. Returns a mutable copy, or None."""
resolved_user = resolve_user_id(user_id, method_name=method_name)
resolved_workspace = resolve_workspace_id(workspace_id, method_name=method_name)
item = await self._store.aget(THREADS_NS, thread_id)
if item is None:
return None
record = dict(item.value)
if resolved is not None and record.get("user_id") != resolved:
if resolved_workspace is not None and record.get("workspace_id") != resolved_workspace:
return None
if resolved_user is not None and record.get("user_id") != resolved_user:
return None
return record
@@ -44,15 +55,18 @@ class MemoryThreadMetaStore(ThreadMetaStore):
*,
assistant_id: str | None = None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
display_name: str | None = None,
metadata: dict | None = None,
) -> dict:
resolved_user_id = resolve_user_id(user_id, method_name="MemoryThreadMetaStore.create")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="MemoryThreadMetaStore.create")
now = now_iso()
record: dict[str, Any] = {
"thread_id": thread_id,
"assistant_id": assistant_id,
"user_id": resolved_user_id,
"workspace_id": resolved_workspace_id,
"display_name": display_name,
"status": "idle",
"metadata": metadata or {},
@@ -63,8 +77,14 @@ class MemoryThreadMetaStore(ThreadMetaStore):
await self._store.aput(THREADS_NS, thread_id, record)
return record
async def get(self, thread_id: str, *, user_id: str | None | _AutoSentinel = AUTO) -> dict | None:
return await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.get")
async def get(
self,
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> dict | None:
return await self._get_owned_record(thread_id, user_id, workspace_id, "MemoryThreadMetaStore.get")
async def search(
self,
@@ -74,13 +94,17 @@ class MemoryThreadMetaStore(ThreadMetaStore):
limit: int = 100,
offset: int = 0,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> list[dict]:
resolved_user_id = resolve_user_id(user_id, method_name="MemoryThreadMetaStore.search")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="MemoryThreadMetaStore.search")
filter_dict: dict[str, Any] = {}
if metadata:
filter_dict.update(metadata)
if status:
filter_dict["status"] = status
if resolved_workspace_id is not None:
filter_dict["workspace_id"] = resolved_workspace_id
if resolved_user_id is not None:
filter_dict["user_id"] = resolved_user_id
@@ -92,33 +116,64 @@ class MemoryThreadMetaStore(ThreadMetaStore):
)
return [self._item_to_dict(item) for item in items]
async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
async def check_access(
self,
thread_id: str,
user_id: str,
workspace_id: str,
*,
require_existing: bool = False,
) -> bool:
item = await self._store.aget(THREADS_NS, thread_id)
if item is None:
return not require_existing
record_workspace_id = item.value.get("workspace_id")
if record_workspace_id is not None and record_workspace_id != workspace_id:
return False
record_user_id = item.value.get("user_id")
if record_user_id is None:
return True
return record_user_id == user_id
async def update_display_name(self, thread_id: str, display_name: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
record = await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.update_display_name")
async def update_display_name(
self,
thread_id: str,
display_name: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
record = await self._get_owned_record(thread_id, user_id, workspace_id, "MemoryThreadMetaStore.update_display_name")
if record is None:
return
record["display_name"] = display_name
record["updated_at"] = now_iso()
await self._store.aput(THREADS_NS, thread_id, record)
async def update_status(self, thread_id: str, status: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
record = await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.update_status")
async def update_status(
self,
thread_id: str,
status: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
record = await self._get_owned_record(thread_id, user_id, workspace_id, "MemoryThreadMetaStore.update_status")
if record is None:
return
record["status"] = status
record["updated_at"] = now_iso()
await self._store.aput(THREADS_NS, thread_id, record)
async def update_metadata(self, thread_id: str, metadata: dict, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
record = await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.update_metadata")
async def update_metadata(
self,
thread_id: str,
metadata: dict,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
record = await self._get_owned_record(thread_id, user_id, workspace_id, "MemoryThreadMetaStore.update_metadata")
if record is None:
return
merged = dict(record.get("metadata") or {})
@@ -127,8 +182,14 @@ class MemoryThreadMetaStore(ThreadMetaStore):
record["updated_at"] = now_iso()
await self._store.aput(THREADS_NS, thread_id, record)
async def delete(self, thread_id: str, *, user_id: str | None | _AutoSentinel = AUTO) -> None:
record = await self._get_owned_record(thread_id, user_id, "MemoryThreadMetaStore.delete")
async def delete(
self,
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
record = await self._get_owned_record(thread_id, user_id, workspace_id, "MemoryThreadMetaStore.delete")
if record is None:
return
await self._store.adelete(THREADS_NS, thread_id)
@@ -4,7 +4,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import JSON, DateTime, String
from sqlalchemy import JSON, DateTime, ForeignKey, Index, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -13,11 +13,25 @@ from deerflow.persistence.base import Base
class ThreadMetaRow(Base):
__tablename__ = "threads_meta"
thread_id: Mapped[str] = mapped_column(String(64), primary_key=True)
assistant_id: Mapped[str | None] = mapped_column(String(128), index=True)
user_id: Mapped[str | None] = mapped_column(String(64), index=True)
display_name: Mapped[str | None] = mapped_column(String(256))
status: Mapped[str] = mapped_column(String(20), default="idle")
metadata_json: Mapped[dict] = mapped_column(JSON, default=dict)
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC))
thread_id: Mapped[str] = mapped_column(String(64), primary_key=True, comment="会话主键(LangGraph thread_id")
assistant_id: Mapped[str | None] = mapped_column(String(128), index=True, comment="关联的 Assistant ID(自定义智能体名);为 NULL 表示默认 lead agent")
user_id: Mapped[str | None] = mapped_column(String(64), index=True, comment="会话所有者;为 NULL 表示历史无主数据")
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
nullable=False,
comment="所属 workspace。PR5 引入时 nullable 用于回填;alembic 0003 + PR6 仓储接入完成后 NOT NULL",
)
display_name: Mapped[str | None] = mapped_column(String(256), comment="会话显示名(自动生成的标题或用户手改)")
status: Mapped[str] = mapped_column(String(20), default="idle", comment='会话状态:"idle" 空闲 / "busy" 正在产出')
metadata_json: Mapped[dict] = mapped_column(JSON, default=dict, comment="任意扩展元数据(JSON")
created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), comment="创建时间(UTC")
updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC), onupdate=lambda: datetime.now(UTC), comment="最近更新时间(UTC,写入时自动更新)")
__table_args__ = (
# Workspace-scoped "list threads of this user, newest first" index;
# added by alembic 0002. Mirrored on the ORM side so create_all()
# produces the same shape on fresh dev databases.
Index("idx_threads_meta_workspace_user_updated", "workspace_id", "user_id", "updated_at"),
{"comment": "会话元数据(每个 LangGraph thread 的概要信息)"},
)
@@ -11,6 +11,13 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.thread_meta.base import ThreadMetaStore
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import (
_AutoSentinel as _WorkspaceAutoSentinel,
)
from deerflow.runtime.workspace_context import (
resolve_workspace_id,
)
class ThreadMetaRepository(ThreadMetaStore):
@@ -33,17 +40,21 @@ class ThreadMetaRepository(ThreadMetaStore):
*,
assistant_id: str | None = None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
display_name: str | None = None,
metadata: dict | None = None,
) -> dict:
# Auto-resolve user_id from contextvar when AUTO; explicit None
# creates an orphan row (used by migration scripts).
# Auto-resolve both user_id and workspace_id from contextvars when
# AUTO; explicit None creates an orphan row (used by migration
# scripts that intentionally bypass scope).
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.create")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.create")
now = datetime.now(UTC)
row = ThreadMetaRow(
thread_id=thread_id,
assistant_id=assistant_id,
user_id=resolved_user_id,
workspace_id=resolved_workspace_id,
display_name=display_name,
metadata_json=metadata or {},
created_at=now,
@@ -60,43 +71,52 @@ class ThreadMetaRepository(ThreadMetaStore):
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> dict | None:
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.get")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.get")
stmt = select(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id)
if resolved_workspace_id is not None:
stmt = stmt.where(ThreadMetaRow.workspace_id == resolved_workspace_id)
async with self._sf() as session:
row = await session.get(ThreadMetaRow, thread_id)
row = (await session.execute(stmt)).scalar_one_or_none()
if row is None:
return None
# Enforce owner filter unless explicitly bypassed (user_id=None).
# Owner filter still applies inside the workspace scope.
if resolved_user_id is not None and row.user_id != resolved_user_id:
return None
return self._row_to_dict(row)
async def check_access(self, thread_id: str, user_id: str, *, require_existing: bool = False) -> bool:
"""Check if ``user_id`` has access to ``thread_id``.
async def check_access(
self,
thread_id: str,
user_id: str,
workspace_id: str,
*,
require_existing: bool = False,
) -> bool:
"""Check if ``user_id`` in ``workspace_id`` has access to ``thread_id``.
Two modes — one row, two distinct semantics depending on what
the caller is about to do:
Three filters layered, from outside in:
- ``require_existing=False`` (default, permissive):
Returns True for: row missing (untracked legacy thread),
``row.user_id`` is None (shared / pre-auth data),
or ``row.user_id == user_id``. Use for **read-style**
decorators where treating an untracked thread as accessible
preserves backward-compat.
- ``require_existing=True`` (strict):
Returns True **only** when the row exists AND
(``row.user_id == user_id`` OR ``row.user_id is None``).
Use for **destructive / mutating** decorators (DELETE, PATCH,
state-update) so a thread that has *already been deleted*
cannot be re-targeted by any caller — closing the
delete-idempotence cross-user gap where the row vanishing
made every other user appear to "own" it.
- Cross-workspace is **always** denied (returns False), even when
the row exists and ``user_id`` matches. The decorator layer
converts a False into a 404 so cross-tenant access never leaks
the existence of a thread.
- Missing row honours ``require_existing``: False by default
(permissive — untracked legacy threads still readable), True
for destructive routes (DELETE / PATCH) so a re-targeted ghost
row cannot be claimed.
- Within the workspace, ``row.user_id IS NULL`` keeps the legacy
"shared / pre-auth" semantics — readable by anyone in the
workspace. ``row.user_id == user_id`` is the normal case.
"""
async with self._sf() as session:
row = await session.get(ThreadMetaRow, thread_id)
if row is None:
return not require_existing
if row.workspace_id is not None and row.workspace_id != workspace_id:
return False
if row.user_id is None:
return True
return row.user_id == user_id
@@ -109,14 +129,19 @@ class ThreadMetaRepository(ThreadMetaStore):
limit: int = 100,
offset: int = 0,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> list[dict]:
"""Search threads with optional metadata and status filters.
Owner filter is enforced by default: caller must be in a user
context. Pass ``user_id=None`` to bypass (migration/CLI).
Both workspace and owner filters are enforced by default. Pass
``workspace_id=None`` and / or ``user_id=None`` to bypass for
migration / CLI paths.
"""
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.search")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.search")
stmt = select(ThreadMetaRow).order_by(ThreadMetaRow.updated_at.desc())
if resolved_workspace_id is not None:
stmt = stmt.where(ThreadMetaRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(ThreadMetaRow.user_id == resolved_user_id)
if status:
@@ -138,12 +163,22 @@ class ThreadMetaRepository(ThreadMetaStore):
result = await session.execute(stmt)
return [self._row_to_dict(r) for r in result.scalars()]
async def _check_ownership(self, session: AsyncSession, thread_id: str, resolved_user_id: str | None) -> bool:
"""Return True if the row exists and is owned (or filter bypassed)."""
if resolved_user_id is None:
return True # explicit bypass
async def _check_ownership(
self,
session: AsyncSession,
thread_id: str,
resolved_user_id: str | None,
resolved_workspace_id: str | None,
) -> bool:
"""Return True if the row exists, is in scope, and is owned (or filter bypassed)."""
row = await session.get(ThreadMetaRow, thread_id)
return row is not None and row.user_id == resolved_user_id
if row is None:
return False
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return False
if resolved_user_id is not None and row.user_id != resolved_user_id:
return False
return True
async def update_display_name(
self,
@@ -151,11 +186,13 @@ class ThreadMetaRepository(ThreadMetaStore):
display_name: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
"""Update the display_name (title) for a thread."""
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_display_name")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.update_display_name")
async with self._sf() as session:
if not await self._check_ownership(session, thread_id, resolved_user_id):
if not await self._check_ownership(session, thread_id, resolved_user_id, resolved_workspace_id):
return
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(display_name=display_name, updated_at=datetime.now(UTC)))
await session.commit()
@@ -166,10 +203,12 @@ class ThreadMetaRepository(ThreadMetaStore):
status: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_status")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.update_status")
async with self._sf() as session:
if not await self._check_ownership(session, thread_id, resolved_user_id):
if not await self._check_ownership(session, thread_id, resolved_user_id, resolved_workspace_id):
return
await session.execute(update(ThreadMetaRow).where(ThreadMetaRow.thread_id == thread_id).values(status=status, updated_at=datetime.now(UTC)))
await session.commit()
@@ -180,18 +219,22 @@ class ThreadMetaRepository(ThreadMetaStore):
metadata: dict,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
"""Merge ``metadata`` into ``metadata_json``.
Read-modify-write inside a single session/transaction so concurrent
callers see consistent state. No-op if the row does not exist or
the user_id check fails.
the workspace / user check fails.
"""
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.update_metadata")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.update_metadata")
async with self._sf() as session:
row = await session.get(ThreadMetaRow, thread_id)
if row is None:
return
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return
if resolved_user_id is not None and row.user_id != resolved_user_id:
return
merged = dict(row.metadata_json or {})
@@ -205,12 +248,16 @@ class ThreadMetaRepository(ThreadMetaStore):
thread_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
) -> None:
resolved_user_id = resolve_user_id(user_id, method_name="ThreadMetaRepository.delete")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="ThreadMetaRepository.delete")
async with self._sf() as session:
row = await session.get(ThreadMetaRow, thread_id)
if row is None:
return
if resolved_workspace_id is not None and row.workspace_id != resolved_workspace_id:
return
if resolved_user_id is not None and row.user_id != resolved_user_id:
return
await session.delete(row)
@@ -13,7 +13,7 @@ from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import Boolean, DateTime, Index, String, text
from sqlalchemy import Boolean, DateTime, ForeignKey, Index, String, text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
@@ -22,31 +22,26 @@ from deerflow.persistence.base import Base
class UserRow(Base):
__tablename__ = "users"
# UUIDs are stored as 36-char strings for cross-backend portability.
id: Mapped[str] = mapped_column(String(36), primary_key=True)
email: Mapped[str] = mapped_column(String(320), unique=True, nullable=False, index=True)
password_hash: Mapped[str | None] = mapped_column(String(128), nullable=True)
# "admin" | "user" — kept as plain string to avoid ALTER TABLE pain
# when new roles are introduced.
system_role: Mapped[str] = mapped_column(String(16), nullable=False, default="user")
id: Mapped[str] = mapped_column(String(36), primary_key=True, comment="用户主键,UUID 字符串(36 字符),跨数据库可移植")
email: Mapped[str] = mapped_column(String(320), unique=True, nullable=False, index=True, comment="登录邮箱,全局唯一")
password_hash: Mapped[str | None] = mapped_column(String(128), nullable=True, comment="本地账户的密码哈希;OAuth-only 用户为 NULL")
system_role: Mapped[str] = mapped_column(String(16), nullable=False, default="user", comment='系统角色:"admin""user";用字符串以便未来扩展角色而不必 ALTER TABLE')
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="账户创建时间(UTC",
)
oauth_provider: Mapped[str | None] = mapped_column(String(32), nullable=True, comment="OAuth 提供商名(如 google/github);本地账户为 NULL")
oauth_id: Mapped[str | None] = mapped_column(String(128), nullable=True, comment="OAuth 提供商内的用户 ID;与 oauth_provider 组合需唯一")
needs_setup: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False, comment="是否需要完成首次设置(admin 自动创建后改密码/邮箱)")
token_version: Mapped[int] = mapped_column(nullable=False, default=0, comment="JWT 令牌版本号;自增即吊销该用户所有旧令牌")
default_workspace_id: Mapped[str | None] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="SET NULL"),
nullable=True,
comment="登录后默认进入的 workspaceNULL 时强制走 pickeruser 多 workspace 场景)",
)
# OAuth linkage (optional). A partial unique index enforces one
# account per (provider, oauth_id) pair, leaving NULL/NULL rows
# unconstrained so plain password accounts can coexist.
oauth_provider: Mapped[str | None] = mapped_column(String(32), nullable=True)
oauth_id: Mapped[str | None] = mapped_column(String(128), nullable=True)
# Auth lifecycle flags
needs_setup: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)
token_version: Mapped[int] = mapped_column(nullable=False, default=0)
__table_args__ = (
Index(
@@ -56,4 +51,5 @@ class UserRow(Base):
unique=True,
sqlite_where=text("oauth_provider IS NOT NULL AND oauth_id IS NOT NULL"),
),
{"comment": "用户账户表(本地密码登录 + OAuth 联合登录)"},
)
@@ -0,0 +1,17 @@
"""Workspace persistence — ORM model + repository.
A workspace is the multi-tenant scope unit. Every user has at least
one (auto-created on registration; their personal workspace where
they are sole owner). Team plans get multi-member workspaces.
Stage 0 PR3 introduces the schema + repository; PR4 wires it into
the registration / login flow; PR5+ ALTER existing business tables
to FK back to ``workspaces.id``.
"""
from __future__ import annotations
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace.sql import WorkspaceRepository, WorkspaceValidationError
__all__ = ["WorkspaceRepository", "WorkspaceRow", "WorkspaceValidationError"]
@@ -0,0 +1,58 @@
"""ORM model for workspaces (multi-tenant scope)."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, ForeignKey, String
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class WorkspaceRow(Base):
__tablename__ = "workspaces"
id: Mapped[str] = mapped_column(
String(36),
primary_key=True,
comment="工作空间主键,UUID 字符串(36 字符),与 users.id 类型对齐",
)
name: Mapped[str] = mapped_column(
String(64),
nullable=False,
comment="工作空间显示名(用户注册时默认 <email 前缀>'s Workspace",
)
slug: Mapped[str] = mapped_column(
String(32),
nullable=False,
unique=True,
comment="URL 标识(^[a-z0-9](-?[a-z0-9])*$3-32 字符);全局唯一,DB 存小写",
)
status: Mapped[str] = mapped_column(
String(16),
nullable=False,
default="active",
comment='状态:"active"(正常)/ "suspended"(平台 admin 暂停)/ "deleted"(软删)',
)
owner_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("users.id", ondelete="RESTRICT"),
nullable=False,
comment="所有者 user_id;与 workspace_memberships 中 role='owner' 行严格一致(事务保证);删除 owner 时 RESTRICT 阻拦(必须先转让所有权)",
)
created_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="创建时间(UTC",
)
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
comment="最近更新时间(UTC,写入时自动更新)",
)
__table_args__ = ({"comment": ("工作空间表(多租户隔离粒度单位)。每个用户注册时自动建一个 1 人 workspace,owner 即注册者。团队订阅时 workspace 可有多个 member。")},)
@@ -0,0 +1,230 @@
"""SQLAlchemy-backed workspace repository.
CRUD + slug lookup for ``workspaces``. Membership-aware methods
(``get``, ``list_by_user``) JOIN against ``workspace_memberships`` so
callers cannot read workspaces they don't belong to.
Three-state ``user_id`` parameter (same convention as
:class:`ThreadMetaRepository`):
- ``AUTO`` → read from contextvar; raise if unset
- explicit ``str`` → override contextvar (admin / tests)
- explicit ``None`` → no filter (migration / CLI only)
"""
from __future__ import annotations
import re
import uuid
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
from deerflow.runtime.user_context import AUTO, _AutoSentinel, resolve_user_id
# slug 字符集 / 长度(与 workspace-schema-design §2.1 锁定一致)。
_SLUG_PATTERN = re.compile(r"^[a-z0-9](-?[a-z0-9])*$")
_SLUG_MIN_LEN = 3
_SLUG_MAX_LEN = 32
# slug 黑名单(应用层校验,不写 DB constraint)。包含 ADR-007 §4 保留 slug
# + 路径 + Next.js 保留 + 业务保留词。
SLUG_BLACKLIST = frozenset(
{
"admin",
"api",
"auth",
"login",
"signup",
"accept-invite",
"pricing",
"docs",
"status",
"platform",
"system",
"health",
"static",
"public",
"favicon.ico",
"robots.txt",
"sitemap.xml",
"_next",
".well-known",
"settings",
"billing",
"onboarding",
"select-workspace",
}
)
# 允许的 status 集合。
_VALID_STATUSES = frozenset({"active", "suspended", "deleted"})
class WorkspaceValidationError(ValueError):
"""Raised when workspace input fails application-layer validation
(slug format / blacklist / status enum)."""
def _validate_slug(slug: str) -> None:
"""Raise :class:`WorkspaceValidationError` if slug is invalid."""
if not isinstance(slug, str):
raise WorkspaceValidationError(f"slug must be a string, got {type(slug).__name__}")
if not (_SLUG_MIN_LEN <= len(slug) <= _SLUG_MAX_LEN):
raise WorkspaceValidationError(f"slug length must be between {_SLUG_MIN_LEN} and {_SLUG_MAX_LEN}, got {len(slug)}")
if not _SLUG_PATTERN.fullmatch(slug):
raise WorkspaceValidationError(f"slug {slug!r} does not match required pattern ^[a-z0-9](-?[a-z0-9])*$")
if slug in SLUG_BLACKLIST:
raise WorkspaceValidationError(f"slug {slug!r} is reserved")
def _validate_status(status: str) -> None:
if status not in _VALID_STATUSES:
raise WorkspaceValidationError(f"status {status!r} is not in allowed set {_VALID_STATUSES!r}")
class WorkspaceRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
@staticmethod
def _row_to_dict(row: WorkspaceRow) -> dict[str, Any]:
return {
"id": row.id,
"name": row.name,
"slug": row.slug,
"status": row.status,
"owner_id": row.owner_id,
"created_at": row.created_at.isoformat() if row.created_at else None,
"updated_at": row.updated_at.isoformat() if row.updated_at else None,
}
async def create(
self,
*,
name: str,
slug: str,
owner_id: str,
workspace_id: str | None = None,
status: str = "active",
) -> dict[str, Any]:
"""Create a new workspace.
``workspace_id`` is optional — UUID v4 is generated if omitted.
Caller is responsible for creating the matching ``owner`` row
in ``workspace_memberships`` (this is typically done in the same
transaction by the registration flow; we deliberately don't bundle
it here to keep the repository single-responsibility).
Raises :class:`WorkspaceValidationError` for invalid slug / status.
Raises :class:`sqlalchemy.exc.IntegrityError` for slug collision
or invalid owner_id FK.
"""
_validate_slug(slug)
_validate_status(status)
wid = workspace_id or str(uuid.uuid4())
now = datetime.now(UTC)
row = WorkspaceRow(
id=wid,
name=name,
slug=slug,
status=status,
owner_id=owner_id,
created_at=now,
updated_at=now,
)
async with self._sf() as session:
session.add(row)
await session.commit()
await session.refresh(row)
return self._row_to_dict(row)
async def get(
self,
workspace_id: str,
*,
user_id: str | None | _AutoSentinel = AUTO,
) -> dict[str, Any] | None:
"""Return workspace row IFF caller is a member.
``user_id=None`` bypasses the membership filter (migration / CLI).
Returns ``None`` if the workspace does not exist or the caller is
not a member.
"""
resolved_user_id = resolve_user_id(user_id, method_name="WorkspaceRepository.get")
async with self._sf() as session:
row = await session.get(WorkspaceRow, workspace_id)
if row is None:
return None
if resolved_user_id is None:
# Explicit bypass (migration / admin path).
return self._row_to_dict(row)
# Membership check via separate SELECT (cheap; index covers it).
membership = await session.execute(
select(WorkspaceMembershipRow).where(
WorkspaceMembershipRow.workspace_id == workspace_id,
WorkspaceMembershipRow.user_id == resolved_user_id,
)
)
if membership.scalar_one_or_none() is None:
return None
return self._row_to_dict(row)
async def get_by_slug(self, slug: str) -> dict[str, Any] | None:
"""Public slug lookup — does NOT check membership.
Used by path-based routing (``/{slug}/...``) where we need to
resolve slug → workspace_id BEFORE we know if caller belongs.
Membership check happens downstream in the route handler.
"""
async with self._sf() as session:
result = await session.execute(select(WorkspaceRow).where(WorkspaceRow.slug == slug))
row = result.scalar_one_or_none()
return self._row_to_dict(row) if row else None
async def list_by_user(
self,
*,
user_id: str | None | _AutoSentinel = AUTO,
) -> list[dict[str, Any]]:
"""Return all workspaces caller is a member of, ordered by joined_at desc.
``user_id=None`` lists ALL workspaces (migration / admin path).
"""
resolved_user_id = resolve_user_id(user_id, method_name="WorkspaceRepository.list_by_user")
async with self._sf() as session:
stmt = select(WorkspaceRow).order_by(WorkspaceRow.created_at.desc())
if resolved_user_id is not None:
stmt = stmt.join(
WorkspaceMembershipRow,
WorkspaceMembershipRow.workspace_id == WorkspaceRow.id,
).where(WorkspaceMembershipRow.user_id == resolved_user_id)
result = await session.execute(stmt)
return [self._row_to_dict(r) for r in result.scalars()]
async def update_status(self, workspace_id: str, status: str) -> None:
"""Platform-admin operation: change workspace status (active/suspended/deleted).
No membership check — this is for platform-level operations. Audit
logging belongs at the route layer.
"""
_validate_status(status)
async with self._sf() as session:
await session.execute(update(WorkspaceRow).where(WorkspaceRow.id == workspace_id).values(status=status, updated_at=datetime.now(UTC)))
await session.commit()
async def delete(self, workspace_id: str) -> None:
"""Hard-delete a workspace. CASCADE drops all memberships.
Intentionally no membership check — caller (platform admin route)
must enforce authorization. Stage 0 doesn't expose this to end
users; Stage 2+ adds it behind owner_only permission.
"""
async with self._sf() as session:
row = await session.get(WorkspaceRow, workspace_id)
if row is not None:
await session.delete(row)
await session.commit()
@@ -0,0 +1,24 @@
"""Workspace membership persistence — ORM model + repository.
Tracks who is a member of which workspace and what their role is
within that workspace. Composite primary key ``(workspace_id, user_id)``;
``role`` follows the three-state ``owner`` / ``admin`` / ``member`` model
(Stage 0 only writes ``'owner'``; Stage 2 RBAC rollout opens admin/member).
A workspace MUST have exactly one ``owner`` — enforced by a partial
unique index. Owner transfer is a two-row transactional swap.
"""
from __future__ import annotations
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
from deerflow.persistence.workspace_membership.sql import (
MembershipValidationError,
WorkspaceMembershipRepository,
)
__all__ = [
"MembershipValidationError",
"WorkspaceMembershipRepository",
"WorkspaceMembershipRow",
]
@@ -0,0 +1,62 @@
"""ORM model for workspace memberships (which user is in which workspace, with what role)."""
from __future__ import annotations
from datetime import UTC, datetime
from sqlalchemy import DateTime, ForeignKey, Index, String, text
from sqlalchemy.orm import Mapped, mapped_column
from deerflow.persistence.base import Base
class WorkspaceMembershipRow(Base):
__tablename__ = "workspace_memberships"
workspace_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("workspaces.id", ondelete="CASCADE"),
primary_key=True,
comment="所属 workspaceworkspace 删除时级联清掉成员记录",
)
user_id: Mapped[str] = mapped_column(
String(36),
ForeignKey("users.id", ondelete="CASCADE"),
primary_key=True,
comment="成员 user_id;用户删除时级联清掉成员记录",
)
role: Mapped[str] = mapped_column(
String(16),
nullable=False,
comment='角色字符串:Stage 0 仅写 "owner"Stage 2 RBAC 打开 "admin"/"member"。用 String(16) 而非 enum 以便未来加 "viewer"/"auditor" 不动 schema',
)
invited_by: Mapped[str | None] = mapped_column(
String(36),
ForeignKey("users.id", ondelete="SET NULL"),
nullable=True,
comment="邀请人 user_idStage 2 invitation 流程才写);邀请人被删时此字段清空(不影响成员记录本身)",
)
joined_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True),
nullable=False,
default=lambda: datetime.now(UTC),
comment="加入 workspace 的时间(UTC",
)
__table_args__ = (
# 倒查索引:列出 user 所属的所有 workspace/auth/me 用)。
# 复合主键 (workspace_id, user_id) 的前导列是 workspace_id
# 所以"按 user_id 查"需要单独的索引。
Index("idx_workspace_memberships_user", "user_id", "workspace_id"),
# 一个 workspace 严格 1 个 owner —— partial unique on role='owner'。
# SQLite + Postgres 都支持 WHERE 子句的 partial unique;双驱动
# 维护两套等价 where 表达式。
Index(
"idx_one_owner_per_workspace",
"workspace_id",
unique=True,
sqlite_where=text("role = 'owner'"),
postgresql_where=text("role = 'owner'"),
),
{"comment": ("工作空间成员表(multi-tenant RBAC)。复合 PK (workspace_id, user_id);每个 workspace 必有恰好 1 个 ownerpartial unique 约束保证)。Stage 0 仅写 ownerStage 2 RBAC 打开 admin/member。")},
)
@@ -0,0 +1,149 @@
"""SQLAlchemy-backed workspace membership repository.
Manages the ``workspace_memberships`` join table. Stage 0 only writes
``role='owner'`` (single-user workspaces); the repository accepts the
full Stage 2 RBAC role enum so the schema is forward-compatible.
Owner transfer is intentionally NOT modelled here as a single method —
it requires a two-row transactional swap with careful retry semantics,
and belongs in the auth router (PR4+) where it can be wrapped in a
permission check.
"""
from __future__ import annotations
from datetime import UTC, datetime
from typing import Any
from sqlalchemy import delete, select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
# 允许的 role 字面值。Stage 0 实际仅写 owneradmin/member 留给 Stage 2 RBAC。
_VALID_ROLES = frozenset({"owner", "admin", "member"})
class MembershipValidationError(ValueError):
"""Raised when role is not in the allowed enum."""
def _validate_role(role: str) -> None:
if role not in _VALID_ROLES:
raise MembershipValidationError(f"role {role!r} is not in allowed set {_VALID_ROLES!r}")
class WorkspaceMembershipRepository:
def __init__(self, session_factory: async_sessionmaker[AsyncSession]) -> None:
self._sf = session_factory
@staticmethod
def _row_to_dict(row: WorkspaceMembershipRow) -> dict[str, Any]:
return {
"workspace_id": row.workspace_id,
"user_id": row.user_id,
"role": row.role,
"invited_by": row.invited_by,
"joined_at": row.joined_at.isoformat() if row.joined_at else None,
}
async def add(
self,
*,
workspace_id: str,
user_id: str,
role: str,
invited_by: str | None = None,
) -> dict[str, Any]:
"""Insert a new membership row.
Raises:
- :class:`MembershipValidationError` for unknown role values
- :class:`sqlalchemy.exc.IntegrityError` for:
- duplicate (workspace_id, user_id) — composite PK collision
- second ``owner`` in the same workspace — partial unique index
- invalid workspace_id / user_id / invited_by FK
"""
_validate_role(role)
row = WorkspaceMembershipRow(
workspace_id=workspace_id,
user_id=user_id,
role=role,
invited_by=invited_by,
joined_at=datetime.now(UTC),
)
async with self._sf() as session:
session.add(row)
await session.commit()
await session.refresh(row)
return self._row_to_dict(row)
async def remove(self, *, workspace_id: str, user_id: str) -> bool:
"""Delete a membership row; returns True if a row was deleted.
Owner removal is allowed at the repository layer — auth router
(PR4) layers the "cannot remove last owner" rule on top.
"""
async with self._sf() as session:
result = await session.execute(
delete(WorkspaceMembershipRow).where(
WorkspaceMembershipRow.workspace_id == workspace_id,
WorkspaceMembershipRow.user_id == user_id,
)
)
await session.commit()
return (result.rowcount or 0) > 0
async def list_by_user(self, *, user_id: str) -> list[dict[str, Any]]:
"""Return all memberships for ``user_id``, ordered by joined_at desc.
No contextvar resolution here — caller is responsible for passing
the correct user_id. Used by ``/auth/me`` to list workspaces a
user belongs to.
"""
async with self._sf() as session:
result = await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.user_id == user_id).order_by(WorkspaceMembershipRow.joined_at.desc()))
return [self._row_to_dict(r) for r in result.scalars()]
async def list_by_workspace(self, *, workspace_id: str) -> list[dict[str, Any]]:
"""Return all members of a workspace, ordered by joined_at asc."""
async with self._sf() as session:
result = await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.workspace_id == workspace_id).order_by(WorkspaceMembershipRow.joined_at.asc()))
return [self._row_to_dict(r) for r in result.scalars()]
async def get_role(self, *, workspace_id: str, user_id: str) -> str | None:
"""Return the role string, or None if user is not a member."""
async with self._sf() as session:
result = await session.execute(
select(WorkspaceMembershipRow.role).where(
WorkspaceMembershipRow.workspace_id == workspace_id,
WorkspaceMembershipRow.user_id == user_id,
)
)
return result.scalar_one_or_none()
async def change_role(
self,
*,
workspace_id: str,
user_id: str,
new_role: str,
) -> bool:
"""Update a member's role; returns True iff a row was updated.
Validates ``new_role`` against the allowed enum. Owner-transfer
flow needs to swap two rows atomically — do that with a manual
transaction in the caller; this method is for non-owner changes.
"""
_validate_role(new_role)
async with self._sf() as session:
result = await session.execute(
update(WorkspaceMembershipRow)
.where(
WorkspaceMembershipRow.workspace_id == workspace_id,
WorkspaceMembershipRow.user_id == user_id,
)
.values(role=new_role)
)
await session.commit()
return (result.rowcount or 0) > 0
@@ -20,6 +20,7 @@ from __future__ import annotations
import asyncio
import contextlib
import logging
import re
from collections.abc import AsyncIterator
from langgraph.types import Checkpointer
@@ -114,7 +115,14 @@ async def _async_checkpointer_from_database(db_config) -> AsyncIterator[Checkpoi
if not db_config.postgres_url:
raise ValueError("database.postgres_url is required for the postgres backend")
async with AsyncPostgresSaver.from_conn_string(db_config.postgres_url) as saver:
# LangGraph's AsyncPostgresSaver wraps psycopg directly and expects a
# libpq-style conninfo (`postgresql://...`). DeerFlow's own SQLAlchemy
# engine uses the same `postgres_url` but needs the `+asyncpg` dialect
# prefix. Strip the dialect prefix here so the same env var/config
# value satisfies both paths.
lg_conn_str = re.sub(r"^postgresql\+\w+://", "postgresql://", db_config.postgres_url)
async with AsyncPostgresSaver.from_conn_string(lg_conn_str) as saver:
await saver.setup()
yield saver
return
@@ -17,6 +17,14 @@ from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.runtime.events.store.base import RunEventStore
from deerflow.runtime.user_context import AUTO, _AutoSentinel, get_current_user, resolve_user_id
from deerflow.runtime.workspace_context import AUTO as WORKSPACE_AUTO
from deerflow.runtime.workspace_context import (
_AutoSentinel as _WorkspaceAutoSentinel,
)
from deerflow.runtime.workspace_context import (
get_current_workspace,
resolve_workspace_id,
)
logger = logging.getLogger(__name__)
@@ -86,6 +94,19 @@ class DbRunEventStore(RunEventStore):
user = get_current_user()
return str(user.id) if user is not None else None
@staticmethod
def _workspace_id_from_context() -> str | None:
"""Soft read of workspace_id from contextvar for write paths.
Mirrors :meth:`_user_id_from_context`. Returns ``None`` (no stamp)
when no workspace is in context — typical for background worker
writes that fire outside an HTTP request. The DB column is
nullable through PR5 and becomes NOT NULL only after the alembic
0003 migration runs (verified by the backfill path).
"""
workspace = get_current_workspace()
return str(workspace.id) if workspace is not None else None
async def put(self, *, thread_id, run_id, event_type, category, content="", metadata=None, created_at=None): # noqa: D401
"""Write a single event — low-frequency path only.
@@ -98,6 +119,7 @@ class DbRunEventStore(RunEventStore):
content, metadata = self._truncate_trace(category, content, metadata)
db_content, metadata = self._content_to_db(content, metadata)
user_id = self._user_id_from_context()
workspace_id = self._workspace_id_from_context()
async with self._sf() as session:
async with session.begin():
# Use FOR UPDATE to serialize seq assignment within a thread.
@@ -109,6 +131,7 @@ class DbRunEventStore(RunEventStore):
thread_id=thread_id,
run_id=run_id,
user_id=user_id,
workspace_id=workspace_id,
event_type=event_type,
category=category,
content=db_content,
@@ -123,6 +146,7 @@ class DbRunEventStore(RunEventStore):
if not events:
return []
user_id = self._user_id_from_context()
workspace_id = self._workspace_id_from_context()
async with self._sf() as session:
async with session.begin():
# Get max seq for the thread (assume all events in batch belong to same thread).
@@ -143,6 +167,7 @@ class DbRunEventStore(RunEventStore):
thread_id=e["thread_id"],
run_id=e["run_id"],
user_id=e.get("user_id", user_id),
workspace_id=e.get("workspace_id", workspace_id),
event_type=e["event_type"],
category=category,
content=db_content,
@@ -162,9 +187,13 @@ class DbRunEventStore(RunEventStore):
before_seq=None,
after_seq=None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.list_messages")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.list_messages")
stmt = select(RunEventRow).where(RunEventRow.thread_id == thread_id, RunEventRow.category == "message")
if resolved_workspace_id is not None:
stmt = stmt.where(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(RunEventRow.user_id == resolved_user_id)
if before_seq is not None:
@@ -194,9 +223,13 @@ class DbRunEventStore(RunEventStore):
event_types=None,
limit=500,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.list_events")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.list_events")
stmt = select(RunEventRow).where(RunEventRow.thread_id == thread_id, RunEventRow.run_id == run_id)
if resolved_workspace_id is not None:
stmt = stmt.where(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(RunEventRow.user_id == resolved_user_id)
if event_types:
@@ -215,13 +248,17 @@ class DbRunEventStore(RunEventStore):
before_seq=None,
after_seq=None,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.list_messages_by_run")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.list_messages_by_run")
stmt = select(RunEventRow).where(
RunEventRow.thread_id == thread_id,
RunEventRow.run_id == run_id,
RunEventRow.category == "message",
)
if resolved_workspace_id is not None:
stmt = stmt.where(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(RunEventRow.user_id == resolved_user_id)
if before_seq is not None:
@@ -246,9 +283,13 @@ class DbRunEventStore(RunEventStore):
thread_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.count_messages")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.count_messages")
stmt = select(func.count()).select_from(RunEventRow).where(RunEventRow.thread_id == thread_id, RunEventRow.category == "message")
if resolved_workspace_id is not None:
stmt = stmt.where(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
stmt = stmt.where(RunEventRow.user_id == resolved_user_id)
async with self._sf() as session:
@@ -259,10 +300,14 @@ class DbRunEventStore(RunEventStore):
thread_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.delete_by_thread")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.delete_by_thread")
async with self._sf() as session:
count_conditions = [RunEventRow.thread_id == thread_id]
if resolved_workspace_id is not None:
count_conditions.append(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
count_conditions.append(RunEventRow.user_id == resolved_user_id)
count_stmt = select(func.count()).select_from(RunEventRow).where(*count_conditions)
@@ -278,10 +323,14 @@ class DbRunEventStore(RunEventStore):
run_id,
*,
user_id: str | None | _AutoSentinel = AUTO,
workspace_id: str | None | _WorkspaceAutoSentinel = WORKSPACE_AUTO,
):
resolved_user_id = resolve_user_id(user_id, method_name="DbRunEventStore.delete_by_run")
resolved_workspace_id = resolve_workspace_id(workspace_id, method_name="DbRunEventStore.delete_by_run")
async with self._sf() as session:
count_conditions = [RunEventRow.thread_id == thread_id, RunEventRow.run_id == run_id]
if resolved_workspace_id is not None:
count_conditions.append(RunEventRow.workspace_id == resolved_workspace_id)
if resolved_user_id is not None:
count_conditions.append(RunEventRow.user_id == resolved_user_id)
count_stmt = select(func.count()).select_from(RunEventRow).where(*count_conditions)
@@ -0,0 +1,182 @@
"""Request-scoped workspace context for multi-tenant authorization.
Sibling of :mod:`deerflow.runtime.user_context`. Holds a
:class:`~contextvars.ContextVar` that the gateway's auth middleware sets
after JWT verification (PR4 will wire this up). Repository methods read
the contextvar via a sentinel default parameter, letting routers stay
free of ``workspace_id`` boilerplate.
Three-state semantics for the repository ``workspace_id`` parameter:
- ``AUTO`` (sentinel, default): read from contextvar; raise
:class:`RuntimeError` if unset.
- Explicit ``str``: use the provided value, overriding contextvar.
- Explicit ``None``: no WHERE clause — used only by migration scripts
and admin CLIs that intentionally bypass workspace isolation.
Concept boundary
----------------
A workspace is the multi-tenant scope: a single-user free account is
its own 1-person workspace; a team subscription is a multi-member
workspace. The user_id contextvar narrows further to "which member of
the workspace", letting some operations be member-scoped while others
(skill install, billing, etc.) are workspace-scoped.
Dependency direction
--------------------
``persistence`` (lower layer) reads from this module; ``gateway.auth``
(higher layer) writes to it. ``CurrentWorkspace`` is defined here as a
:class:`typing.Protocol` so that ``persistence`` never needs to import
the concrete ``Workspace`` row class from ``deerflow.persistence.workspace``.
Any object with ``.id: str`` and ``.role: str`` attributes structurally
satisfies the protocol.
Asyncio semantics
-----------------
Identical to ``user_context``: ``ContextVar`` is task-local under asyncio.
``asyncio.create_task`` inherits the parent task's workspace context;
threading.Timer does **not** (callers spawning timers must capture
``get_effective_workspace_id()`` at enqueue time, the same way
:mod:`deerflow.agents.memory.queue` captures ``user_id``).
"""
from __future__ import annotations
from contextvars import ContextVar, Token
from typing import Final, Protocol, runtime_checkable
@runtime_checkable
class CurrentWorkspace(Protocol):
"""Structural type for the current active workspace.
Any object with ``.id: str`` and ``.role: str`` attributes satisfies
this protocol. Concrete implementations live in
``app.gateway.auth.models`` (PR4 will add them).
``role`` is the *caller's* role within this workspace
(``'owner'`` / ``'admin'`` / ``'member'``), not the workspace's
own metadata. Stage 0 sees ``'owner'`` only — Stage 2 RBAC rollout
opens up the other values.
"""
id: str
role: str
_current_workspace: Final[ContextVar[CurrentWorkspace | None]] = ContextVar("deerflow_current_workspace", default=None)
def set_current_workspace(workspace: CurrentWorkspace) -> Token[CurrentWorkspace | None]:
"""Set the current workspace for this async task.
Returns a reset token that should be passed to
:func:`reset_current_workspace` in a ``finally`` block to restore
the previous context.
"""
return _current_workspace.set(workspace)
def reset_current_workspace(token: Token[CurrentWorkspace | None]) -> None:
"""Restore the context to the state captured by ``token``."""
_current_workspace.reset(token)
def get_current_workspace() -> CurrentWorkspace | None:
"""Return the current workspace, or ``None`` if unset.
Safe to call in any context. Used by code paths that can proceed
without a workspace (migration scripts, public endpoints).
"""
return _current_workspace.get()
def require_current_workspace() -> CurrentWorkspace:
"""Return the current workspace, or raise :class:`RuntimeError`.
Used by repository code that must not be called outside a
request-authenticated context. The error message is phrased so
that a caller debugging a stack trace can locate the offending
code path.
"""
workspace = _current_workspace.get()
if workspace is None:
raise RuntimeError("repository accessed without workspace context")
return workspace
# ---------------------------------------------------------------------------
# Effective workspace_id helpers (filesystem isolation)
# ---------------------------------------------------------------------------
DEFAULT_WORKSPACE_ID: Final[str] = "default"
def get_effective_workspace_id() -> str:
"""Return the current workspace id as a string, or DEFAULT_WORKSPACE_ID if unset.
Unlike :func:`require_current_workspace` this never raises — it is
designed for filesystem-path resolution where a valid workspace
bucket is always needed (PR6 will switch
``Paths.thread_dir(workspace_id=...)`` to read from here).
"""
workspace = _current_workspace.get()
if workspace is None:
return DEFAULT_WORKSPACE_ID
return str(workspace.id)
# ---------------------------------------------------------------------------
# Sentinel-based workspace_id resolution
# ---------------------------------------------------------------------------
#
# Repository methods accept a ``workspace_id`` keyword-only argument that
# defaults to ``AUTO``. The three possible values drive distinct
# behaviours; see the docstring on :func:`resolve_workspace_id`.
class _AutoSentinel:
"""Singleton marker meaning 'resolve workspace_id from contextvar'."""
_instance: _AutoSentinel | None = None
def __new__(cls) -> _AutoSentinel:
if cls._instance is None:
cls._instance = super().__new__(cls)
return cls._instance
def __repr__(self) -> str:
return "<AUTO>"
AUTO: Final[_AutoSentinel] = _AutoSentinel()
def resolve_workspace_id(
value: str | None | _AutoSentinel,
*,
method_name: str = "repository method",
) -> str | None:
"""Resolve the workspace_id parameter passed to a repository method.
Three-state semantics:
- :data:`AUTO` (default): read from contextvar; raise
:class:`RuntimeError` if no workspace is in context. This is the
common case for request-scoped calls.
- Explicit ``str``: use the provided id verbatim, overriding any
contextvar value. Useful for tests and admin-override flows.
- Explicit ``None``: no filter — the repository should skip the
workspace_id WHERE clause entirely. Reserved for migration scripts
and CLI tools that intentionally bypass workspace isolation.
"""
if isinstance(value, _AutoSentinel):
workspace = _current_workspace.get()
if workspace is None:
raise RuntimeError(
f"{method_name} called with workspace_id=AUTO but no workspace context is set; pass an explicit workspace_id, set the contextvar via auth middleware, or opt out with workspace_id=None for migration/CLI paths."
)
# Coerce to ``str`` at the boundary; persistence stores
# ``workspace_id`` as ``String(36)`` (UUID v4 text).
return str(workspace.id)
return value
+4
View File
@@ -46,6 +46,10 @@ postgres = [
"psycopg[binary]>=3.3.3",
"psycopg-pool>=3.3.0",
]
postgres-test = [
"deerflow-harness[postgres]",
"testcontainers[postgres]>=4.0",
]
pymupdf = ["pymupdf4llm>=0.0.17"]
[build-system]
+4 -1
View File
@@ -25,6 +25,7 @@ dependencies = [
[project.optional-dependencies]
postgres = ["deerflow-harness[postgres]"]
postgres-test = ["deerflow-harness[postgres-test]"]
[dependency-groups]
dev = [
@@ -36,7 +37,9 @@ dev = [
[tool.pytest.ini_options]
markers = [
"no_auto_user: disable the conftest autouse contextvar fixture for this test",
"no_auto_user: disable the conftest autouse user contextvar fixture for this test",
"no_auto_workspace: disable the conftest autouse workspace contextvar fixture for this test",
"postgres: requires a Postgres testcontainer (Docker daemon); skipped if unavailable",
]
[tool.uv]
+306
View File
@@ -0,0 +1,306 @@
"""Backfill ``workspace_id`` on PR5 business tables.
Three-step backfill (each idempotent — re-running picks up where a crash
left off because every step's WHERE clause filters already-processed rows):
1. For each user without ``default_workspace_id``: create a personal
workspace + ``owner`` membership + write back the user's
``default_workspace_id``.
2. ``UPDATE`` each of ``threads_meta`` / ``runs`` / ``feedback`` /
``run_events`` setting ``workspace_id`` from the row's owner's
``users.default_workspace_id``. Only touches rows where
``workspace_id IS NULL`` and ``user_id IS NOT NULL``.
3. Any rows still with ``workspace_id IS NULL`` (truly orphan — they had
``user_id = NULL`` to begin with) are assigned the *legacy* workspace
UUID ``00000000-0000-0000-0000-000000000000``. The script creates
that workspace on demand, owned by the platform admin.
Usage::
PYTHONPATH=. python scripts/backfill_workspace_id.py [--dry-run]
T5.4 only ships the skeleton — the three step bodies are filled in by
T5.5 / T5.6 / T5.7 along with their per-step tests.
"""
from __future__ import annotations
import argparse
import asyncio
import logging
from typing import Any
from sqlalchemy import func, select, update
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
from app.gateway.auth.workspace_slug import auto_slug_from_email, next_available_slug
from deerflow.persistence.base import Base
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace import WorkspaceRepository
from deerflow.persistence.workspace.sql import SLUG_BLACKLIST
from deerflow.persistence.workspace_membership import WorkspaceMembershipRepository
logger = logging.getLogger(__name__)
# The legacy workspace anchor. Stage 0 LOCK'd UUID — chosen as the standard
# nil UUID so SQL log scans can spot it instantly.
LEGACY_WORKSPACE_ID = "00000000-0000-0000-0000-000000000000"
LEGACY_WORKSPACE_SLUG = "legacy"
LEGACY_WORKSPACE_NAME = "Legacy Workspace"
# The four business tables that gained ``workspace_id`` in alembic 0002.
_BUSINESS_TABLES: tuple[str, ...] = ("threads_meta", "runs", "feedback", "run_events")
# Map table-name to ORM class so we can build a portable correlated UPDATE
# using SQLAlchemy expression language (SQLite < 3.33 lacks UPDATE-FROM
# but supports correlated subqueries on every version we ship).
_TABLE_MODELS: dict[str, type[Base]] = {
"threads_meta": ThreadMetaRow,
"runs": RunRow,
"feedback": FeedbackRow,
"run_events": RunEventRow,
}
async def _step1_create_workspaces_for_users(
session_factory: async_sessionmaker[AsyncSession],
*,
dry_run: bool,
) -> int:
"""Create one workspace + owner membership for each user missing default_workspace_id.
Returns the count of users a workspace was created for. Idempotent —
users that already have ``default_workspace_id`` are skipped so a
crashed run can resume safely.
"""
async with session_factory() as session:
result = await session.execute(select(UserRow.id, UserRow.email).where(UserRow.default_workspace_id.is_(None)))
candidates = [(row.id, row.email) for row in result]
if not candidates:
return 0
ws_repo = WorkspaceRepository(session_factory)
m_repo = WorkspaceMembershipRepository(session_factory)
created = 0
for user_id, email in candidates:
base_slug = auto_slug_from_email(email)
async def slug_exists(s: str) -> bool:
if s in SLUG_BLACKLIST:
return True
return (await ws_repo.get_by_slug(s)) is not None
unique_slug = await next_available_slug(base_slug, exists_check=slug_exists)
if dry_run:
logger.info("WOULD create workspace for user=%s email=%s slug=%s", user_id, email, unique_slug)
created += 1
continue
display_local = email.split("@", 1)[0]
workspace = await ws_repo.create(
name=f"{display_local}'s Workspace"[:64],
slug=unique_slug,
owner_id=user_id,
)
await m_repo.add(workspace_id=workspace["id"], user_id=user_id, role="owner")
async with session_factory() as session:
await session.execute(update(UserRow).where(UserRow.id == user_id).values(default_workspace_id=workspace["id"]))
await session.commit()
created += 1
logger.info("Created workspace %s (slug=%s) for user=%s", workspace["id"], unique_slug, user_id)
return created
async def _step2_update_table_from_users(
session_factory: async_sessionmaker[AsyncSession],
table: str,
*,
dry_run: bool,
) -> int:
"""UPDATE *table* setting workspace_id from owner's users.default_workspace_id.
Uses a correlated subquery (portable across SQLite + Postgres). Filters
``workspace_id IS NULL AND user_id IS NOT NULL`` so already-set rows
and truly orphan rows are skipped (Step 3 handles the latter).
Returns the number of rows updated (or that *would* be updated under
``dry_run``).
"""
model = _TABLE_MODELS[table]
workspace_col = model.workspace_id
user_col = model.user_id
# Subquery: pull the user's default_workspace_id for each row.
correlated_default = select(UserRow.default_workspace_id).where(UserRow.id == user_col).scalar_subquery()
if dry_run:
# Count rows whose owner has a default_workspace_id assigned — only
# those would get touched by the actual UPDATE.
count_stmt = select(func.count()).select_from(model).join(UserRow, UserRow.id == user_col).where(workspace_col.is_(None), user_col.is_not(None), UserRow.default_workspace_id.is_not(None))
async with session_factory() as session:
count = (await session.execute(count_stmt)).scalar_one() or 0
logger.info("WOULD update %d rows in %s from users.default_workspace_id", count, table)
return int(count)
stmt = update(model).where(workspace_col.is_(None), user_col.is_not(None)).values(workspace_id=correlated_default)
async with session_factory() as session:
result = await session.execute(stmt)
await session.commit()
rowcount = result.rowcount or 0
logger.info("Updated %d rows in %s from users.default_workspace_id", rowcount, table)
return int(rowcount)
async def _ensure_legacy_workspace(
session_factory: async_sessionmaker[AsyncSession],
*,
dry_run: bool,
) -> bool:
"""Create the ``legacy_workspace`` anchor row idempotently.
The anchor is needed before Step 3 can point orphan rows at it. We
pick the platform admin (``system_role='admin'``) as owner; if no
admin exists yet we fall back to the oldest user. If the database
has no users at all we refuse to continue — running this script on
an unbootstrapped DB would create a workspace with no owner and the
FK to ``users`` would fail anyway.
Returns True if the workspace was just created (or would be, under
``dry_run``). False if it already existed.
"""
from deerflow.persistence.workspace.model import WorkspaceRow
async with session_factory() as session:
existing = await session.get(WorkspaceRow, LEGACY_WORKSPACE_ID)
if existing is not None:
return False
async with session_factory() as session:
admin_id = (await session.execute(select(UserRow.id).where(UserRow.system_role == "admin").order_by(UserRow.created_at).limit(1))).scalar_one_or_none()
if admin_id is None:
admin_id = (await session.execute(select(UserRow.id).order_by(UserRow.created_at).limit(1))).scalar_one_or_none()
if admin_id is None:
raise RuntimeError(
"Cannot create legacy_workspace: no users exist. Bootstrap an admin via /auth/initialize before running backfill.",
)
if dry_run:
logger.info("WOULD create legacy_workspace (id=%s) owned by user=%s", LEGACY_WORKSPACE_ID, admin_id)
return True
ws_repo = WorkspaceRepository(session_factory)
await ws_repo.create(
workspace_id=LEGACY_WORKSPACE_ID,
name=LEGACY_WORKSPACE_NAME,
slug=LEGACY_WORKSPACE_SLUG,
owner_id=admin_id,
)
m_repo = WorkspaceMembershipRepository(session_factory)
await m_repo.add(workspace_id=LEGACY_WORKSPACE_ID, user_id=admin_id, role="owner")
logger.info("Created legacy_workspace (id=%s) owned by user=%s", LEGACY_WORKSPACE_ID, admin_id)
return True
async def _step3_assign_legacy_workspace(
session_factory: async_sessionmaker[AsyncSession],
table: str,
*,
dry_run: bool,
) -> int:
"""Assign LEGACY_WORKSPACE_ID to *table* rows still missing workspace_id.
Callers should ensure :func:`_ensure_legacy_workspace` has run first;
the orchestrator does this between Step 2 and Step 3. Orphan rows are
rows whose ``user_id`` was already NULL (or pointed at a deleted user)
so Step 2's correlated subquery left them untouched.
"""
model = _TABLE_MODELS[table]
workspace_col = model.workspace_id
if dry_run:
count_stmt = select(func.count()).select_from(model).where(workspace_col.is_(None))
async with session_factory() as session:
count = (await session.execute(count_stmt)).scalar_one() or 0
logger.info("WOULD assign %d orphan row(s) in %s to legacy_workspace", count, table)
return int(count)
stmt = update(model).where(workspace_col.is_(None)).values(workspace_id=LEGACY_WORKSPACE_ID)
async with session_factory() as session:
result = await session.execute(stmt)
await session.commit()
rowcount = result.rowcount or 0
logger.info("Assigned %d orphan row(s) in %s to legacy_workspace", rowcount, table)
return int(rowcount)
async def backfill(
session_factory: async_sessionmaker[AsyncSession],
*,
dry_run: bool = False,
) -> dict[str, Any]:
"""Run all three backfill steps; return a per-step row-count report.
Order matters: Step 1 must populate ``users.default_workspace_id``
before Step 2 can correlate business rows back through ``users``.
"""
report: dict[str, Any] = {"dry_run": dry_run}
report["users_workspaces_created"] = await _step1_create_workspaces_for_users(session_factory, dry_run=dry_run)
for table in _BUSINESS_TABLES:
report[f"{table}_from_users"] = await _step2_update_table_from_users(session_factory, table, dry_run=dry_run)
report["legacy_workspace_created"] = await _ensure_legacy_workspace(session_factory, dry_run=dry_run)
for table in _BUSINESS_TABLES:
report[f"{table}_legacy"] = await _step3_assign_legacy_workspace(session_factory, table, dry_run=dry_run)
return report
def _build_session_factory_from_config() -> async_sessionmaker[AsyncSession]:
"""Build an async session factory from the active config.yaml.
Avoids importing on module load so unit tests can stub
``session_factory`` directly without booting the full config pipeline.
"""
from deerflow.config import get_app_config
from deerflow.persistence.engine import get_session_factory, init_engine_from_config
asyncio.run(init_engine_from_config(get_app_config().database))
sf = get_session_factory()
if sf is None:
raise RuntimeError(
"database.backend=memory: nothing to backfill. Switch config.yaml to sqlite/postgres first.",
)
return sf
def main() -> None:
parser = argparse.ArgumentParser(description="Backfill workspace_id on Stage 0 business tables (idempotent).")
parser.add_argument(
"--dry-run",
action="store_true",
help="Print the rows each step would touch without writing.",
)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
sf = _build_session_factory_from_config()
report = asyncio.run(backfill(sf, dry_run=args.dry_run))
logger.info("Backfill report (dry_run=%s):", args.dry_run)
for key, value in report.items():
if key == "dry_run":
continue
logger.info(" %s: %s", key, value)
if __name__ == "__main__":
main()
@@ -0,0 +1,212 @@
"""One-time migration: lift per-user thread / memory / agent dirs into per-workspace layout.
PR4 introduced ``{base_dir}/users/{user_id}/...`` for per-user isolation. PR6
adds the multi-tenant top dimension: ``{base_dir}/workspaces/{wid}/...``,
with per-user state nested under each workspace
(``{base_dir}/workspaces/{wid}/users/{uid}/memory.json`` etc.). This script
walks the old PR4 layout, looks up each user's ``default_workspace_id``
from the ``users`` table, and rewrites the path.
Usage:
PYTHONPATH=. python scripts/migrate_paths_to_workspace.py [--dry-run] [--default-workspace WID]
The script is idempotent — re-running it after a successful migration is a no-op.
A ``--dry-run`` invocation must not write anything; it only logs what would happen.
Mapping rules:
- ``{base_dir}/users/{uid}/threads/{tid}/`` -> ``{base_dir}/workspaces/{wid}/threads/{tid}/``
- ``{base_dir}/users/{uid}/memory.json`` -> ``{base_dir}/workspaces/{wid}/users/{uid}/memory.json``
- ``{base_dir}/users/{uid}/agents/{name}/`` -> ``{base_dir}/workspaces/{wid}/users/{uid}/agents/{name}/``
``{wid}`` is read from ``users.default_workspace_id``. Users without one
fall through to ``--default-workspace`` (defaults to the special bucket
``"legacy_workspace"`` so they can be triaged manually). Pre-existing
destinations are preserved; legacy copies are moved to
``{base_dir}/migration-conflicts/<...>`` for human review.
"""
from __future__ import annotations
import argparse
import logging
import shutil
import sqlite3
from pathlib import Path
from deerflow.config.paths import Paths, get_paths
logger = logging.getLogger(__name__)
LEGACY_WORKSPACE_FALLBACK = "legacy_workspace"
def _load_user_workspaces(paths: Paths) -> dict[str, str | None]:
"""Read ``user_id -> default_workspace_id`` from the local sqlite DB.
Returns an empty dict when the database does not exist (fresh install
/ Postgres-only deployments). The Postgres path is out of scope for
this script — the operator should run it with ``--default-workspace``
set to the target workspace and skip the DB lookup.
"""
db_path = paths.base_dir / "deer-flow.db"
if not db_path.exists():
logger.info("No sqlite database at %s — every user will use the fallback workspace.", db_path)
return {}
conn = sqlite3.connect(str(db_path))
try:
try:
cursor = conn.execute("SELECT id, default_workspace_id FROM users")
except sqlite3.OperationalError as e:
logger.warning("Failed to query users.default_workspace_id: %s", e)
return {}
return {row[0]: row[1] for row in cursor.fetchall()}
finally:
conn.close()
def _resolve_workspace(user_id: str, user_workspaces: dict[str, str | None], fallback: str) -> str:
wid = user_workspaces.get(user_id)
return wid or fallback
def _move(src: Path, dest: Path, conflict_root: Path, *, dry_run: bool, label: str) -> str:
"""Move ``src`` to ``dest``; on conflict, divert legacy under ``conflict_root``.
Returns a short string describing what happened (for the report).
"""
if dest.exists():
conflict_dest = conflict_root / label / src.name
if not dry_run:
conflict_dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(src), str(conflict_dest))
logger.warning("Conflict for %s/%s: legacy copy diverted to %s", label, src.name, conflict_dest)
return f"conflict -> {conflict_dest}"
if not dry_run:
dest.parent.mkdir(parents=True, exist_ok=True)
shutil.move(str(src), str(dest))
return f"moved -> {dest}"
def migrate_user_tree(
paths: Paths,
user_id: str,
workspace_id: str,
*,
dry_run: bool,
) -> list[dict]:
"""Lift one user's tree under a workspace. Returns per-asset report rows."""
report: list[dict] = []
legacy_user_dir = paths.base_dir / "users" / user_id
if not legacy_user_dir.exists():
return report
conflict_root = paths.base_dir / "migration-conflicts" / "workspace-migration"
# 1. threads/{tid}/ -> workspaces/{wid}/threads/{tid}/
legacy_threads = legacy_user_dir / "threads"
if legacy_threads.exists():
for thread_dir in sorted(legacy_threads.iterdir()):
if not thread_dir.is_dir():
continue
dest = paths.thread_dir(thread_dir.name, workspace_id=workspace_id)
action = _move(thread_dir, dest, conflict_root, dry_run=dry_run, label=f"threads/{workspace_id}")
report.append({"asset": "thread", "user_id": user_id, "workspace_id": workspace_id, "name": thread_dir.name, "action": action})
if not dry_run and legacy_threads.exists() and not any(legacy_threads.iterdir()):
legacy_threads.rmdir()
# 2. memory.json -> workspaces/{wid}/users/{uid}/memory.json
legacy_mem = legacy_user_dir / "memory.json"
if legacy_mem.exists():
dest = paths.user_memory_file(user_id, workspace_id=workspace_id)
action = _move(legacy_mem, dest, conflict_root, dry_run=dry_run, label=f"users/{user_id}/memory")
report.append({"asset": "memory", "user_id": user_id, "workspace_id": workspace_id, "name": "memory.json", "action": action})
# 3. agents/{name}/ -> workspaces/{wid}/users/{uid}/agents/{name}/
legacy_agents = legacy_user_dir / "agents"
if legacy_agents.exists():
for agent_dir in sorted(legacy_agents.iterdir()):
if not agent_dir.is_dir():
continue
dest = paths.user_agent_dir(user_id, agent_dir.name, workspace_id=workspace_id)
action = _move(agent_dir, dest, conflict_root, dry_run=dry_run, label=f"users/{user_id}/agents")
report.append({"asset": "agent", "user_id": user_id, "workspace_id": workspace_id, "name": agent_dir.name, "action": action})
if not dry_run and legacy_agents.exists() and not any(legacy_agents.iterdir()):
legacy_agents.rmdir()
# 4. If the user dir is now empty, remove it so lifespan warnings clear.
if not dry_run and legacy_user_dir.exists() and not any(legacy_user_dir.iterdir()):
legacy_user_dir.rmdir()
return report
def migrate(
paths: Paths,
*,
user_workspaces: dict[str, str | None],
fallback_workspace: str,
dry_run: bool,
) -> list[dict]:
"""Top-level entry point: iterate every user under ``{base_dir}/users``."""
legacy_users = paths.base_dir / "users"
if not legacy_users.exists():
logger.info("No legacy ``users/`` directory under %s — nothing to migrate.", paths.base_dir)
return []
report: list[dict] = []
for user_dir in sorted(legacy_users.iterdir()):
if not user_dir.is_dir():
continue
user_id = user_dir.name
workspace_id = _resolve_workspace(user_id, user_workspaces, fallback_workspace)
logger.info("Migrating user %s -> workspace %s", user_id, workspace_id)
report.extend(migrate_user_tree(paths, user_id, workspace_id, dry_run=dry_run))
if not dry_run and legacy_users.exists() and not any(legacy_users.iterdir()):
legacy_users.rmdir()
return report
def main() -> None:
parser = argparse.ArgumentParser(description="Lift legacy per-user paths into per-workspace layout (PR6).")
parser.add_argument("--dry-run", action="store_true", help="Log actions without making changes.")
parser.add_argument(
"--default-workspace",
default=LEGACY_WORKSPACE_FALLBACK,
metavar="WID",
help=(f"Workspace id to use for users without a ``default_workspace_id`` in the DB. Defaults to ``{LEGACY_WORKSPACE_FALLBACK}`` (matches the orphan-row bucket used by PR5 backfill)."),
)
args = parser.parse_args()
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
paths = get_paths()
logger.info("Base directory: %s", paths.base_dir)
logger.info("Dry run: %s", args.dry_run)
logger.info("Fallback workspace: %s", args.default_workspace)
user_workspaces = _load_user_workspaces(paths)
logger.info("Loaded %d user->workspace mappings from DB", len(user_workspaces))
report = migrate(
paths,
user_workspaces=user_workspaces,
fallback_workspace=args.default_workspace,
dry_run=args.dry_run,
)
if not report:
logger.info("Nothing to migrate.")
return
logger.info("Migration report (%d entries):", len(report))
for entry in report:
logger.info(" asset=%s user=%s workspace=%s name=%s action=%s", entry["asset"], entry["user_id"], entry["workspace_id"], entry["name"], entry["action"])
if __name__ == "__main__":
main()
+420
View File
@@ -0,0 +1,420 @@
#!/usr/bin/env bash
# verify_stage0.sh — systematic verification of the Stage 0 multi-tenant rollout.
#
# Layers (each can be run individually; defaults run all that are applicable):
# static — full pytest suite + lint + boundary scan (no external deps)
# paths — local filesystem layout: legacy users/ should be empty after migration
# rds — schema + index + alembic state on the remote Postgres (needs DATABASE_URL)
# runtime — gateway health probes (needs `make dev` running)
# e2e — register 2 users via curl, each creates a thread, cross-access must 404
#
# Usage:
# ./scripts/verify_stage0.sh # run everything that has prerequisites
# ./scripts/verify_stage0.sh static rds # only those two
# DATABASE_URL=postgres://... ./scripts/verify_stage0.sh
# GATEWAY_URL=http://localhost:8001 ./scripts/verify_stage0.sh runtime e2e
#
# Exit code: 0 if every executed assertion passes, 1 if any fail.
set -uo pipefail
# ── colours ──────────────────────────────────────────────────────────────
if [ -t 1 ]; then
C_RED=$'\033[0;31m'; C_GREEN=$'\033[0;32m'; C_YELLOW=$'\033[0;33m'
C_BLUE=$'\033[0;34m'; C_BOLD=$'\033[1m'; C_DIM=$'\033[2m'; C_RESET=$'\033[0m'
else
C_RED=''; C_GREEN=''; C_YELLOW=''; C_BLUE=''; C_BOLD=''; C_DIM=''; C_RESET=''
fi
# ── counters ─────────────────────────────────────────────────────────────
PASS_COUNT=0
FAIL_COUNT=0
WARN_COUNT=0
FAILED_STEPS=()
SKIPPED_PHASES=()
ok() { printf ' %s✓%s %s\n' "$C_GREEN" "$C_RESET" "$1"; PASS_COUNT=$((PASS_COUNT+1)); }
fail() { printf ' %s✗%s %s\n' "$C_RED" "$C_RESET" "$1"; FAIL_COUNT=$((FAIL_COUNT+1)); FAILED_STEPS+=("$1"); }
warn() { printf ' %s!%s %s\n' "$C_YELLOW" "$C_RESET" "$1"; WARN_COUNT=$((WARN_COUNT+1)); }
info() { printf ' %s·%s %s\n' "$C_DIM" "$C_RESET" "$1"; }
phase() { printf '\n%s== %s ==%s\n' "$C_BLUE$C_BOLD" "$1" "$C_RESET"; }
# Run-cmd helpers — capture both streams for grep but return original exit code.
run() {
local label="$1"; shift
local out
out=$("$@" 2>&1)
local rc=$?
if [ "$rc" -eq 0 ]; then
ok "$label"
printf '%s' "$out"
return 0
fi
fail "$label (exit $rc)"
printf '%s%s%s\n' "$C_DIM" "$out" "$C_RESET" >&2
return "$rc"
}
# ── env ──────────────────────────────────────────────────────────────────
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
BACKEND_DIR="$REPO_ROOT/backend"
GATEWAY_URL="${GATEWAY_URL:-http://localhost:8001}"
DEER_FLOW_HOME="${DEER_FLOW_HOME:-$REPO_ROOT/.deer-flow}"
# ── phase: static ────────────────────────────────────────────────────────
phase_static() {
phase "STATIC — unit + boundary + full pytest"
cd "$BACKEND_DIR"
# boundary scans first — fast and high-signal
info "running boundary scans (harness + workspace)"
if PYTHONPATH=. uv run pytest -q \
tests/test_harness_boundary.py \
tests/test_workspace_boundary.py \
tests/test_workspace_boundary_self.py >/tmp/verify_boundary.log 2>&1; then
local boundary_passed
boundary_passed=$(grep -Eo '[0-9]+ passed' /tmp/verify_boundary.log | head -1)
ok "boundary scans green ($boundary_passed)"
else
fail "boundary scans red — see /tmp/verify_boundary.log"
fi
info "running full pytest (this takes ~90-130s)"
local pytest_log=/tmp/verify_full_pytest.log
PYTHONPATH=. uv run pytest -q >"$pytest_log" 2>&1
local rc=$?
local last_line
last_line=$(tail -1 "$pytest_log")
info "result: $last_line"
# Parse "<X> passed, <Y> failed, <Z> skipped"
local passed_n failed_n
passed_n=$(printf '%s' "$last_line" | grep -Eo '[0-9]+ passed' | grep -Eo '[0-9]+' | head -1 || echo 0)
failed_n=$(printf '%s' "$last_line" | grep -Eo '[0-9]+ failed' | grep -Eo '[0-9]+' | head -1 || echo 0)
if [ "${passed_n:-0}" -ge 3250 ]; then
ok "pytest pass count $passed_n ≥ 3250 (PR8 baseline)"
else
fail "pytest pass count $passed_n < 3250 — regression suspected"
fi
if [ "${failed_n:-0}" -le 18 ]; then
ok "pytest fail count $failed_n ≤ 18 (Stage 0 known-flake ceiling)"
[ "${failed_n:-0}" -gt 0 ] && warn "non-zero failures expected to be in the 18 caplog flake set; cross-check tail of /tmp/verify_full_pytest.log"
else
fail "pytest fail count $failed_n > 18 — new failures introduced beyond known caplog flake set"
fi
info "running ruff lint"
if make lint >/tmp/verify_lint.log 2>&1; then
ok "ruff lint clean"
else
fail "ruff lint dirty — see /tmp/verify_lint.log"
fi
}
# ── phase: paths ─────────────────────────────────────────────────────────
phase_paths() {
phase "PATHS — legacy users/ migration state"
if [ ! -d "$DEER_FLOW_HOME" ]; then
warn "DEER_FLOW_HOME ($DEER_FLOW_HOME) does not exist — fresh install, nothing to migrate"
return
fi
info "DEER_FLOW_HOME = $DEER_FLOW_HOME"
local legacy_dir="$DEER_FLOW_HOME/users"
if [ ! -d "$legacy_dir" ]; then
ok "no legacy users/ directory present (PR6 migration not needed or already done)"
else
local count
count=$(find "$legacy_dir" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l | tr -d ' ')
if [ "$count" -eq 0 ]; then
ok "legacy users/ is empty (migration complete or no prior data)"
else
warn "legacy users/ still has $count user dir(s) — run \`make migrate-paths\` (after \`make migrate-paths DRY_RUN=1\` to preview)"
fi
fi
local workspace_dir="$DEER_FLOW_HOME/workspaces"
if [ -d "$workspace_dir" ]; then
local wcount
wcount=$(find "$workspace_dir" -mindepth 1 -maxdepth 1 -type d 2>/dev/null | wc -l | tr -d ' ')
ok "workspaces/ layout in place ($wcount workspace dir(s))"
else
warn "workspaces/ dir does not exist yet — will be created when the first thread is opened"
fi
# Stage 0 PR6 dry-run check: just exercise the script flag, do not perform writes.
info "exercising migrate-paths dry-run (no writes)"
cd "$REPO_ROOT"
if make migrate-paths DRY_RUN=1 >/tmp/verify_migrate_dry.log 2>&1; then
ok "make migrate-paths DRY_RUN=1 ran without error"
else
fail "make migrate-paths DRY_RUN=1 failed — see /tmp/verify_migrate_dry.log"
fi
}
# ── phase: rds ───────────────────────────────────────────────────────────
phase_rds() {
phase "RDS — schema + alembic + indexes"
if [ -z "${DATABASE_URL:-}" ]; then
warn "DATABASE_URL not set — skipping RDS phase"
SKIPPED_PHASES+=("rds (DATABASE_URL unset)")
return
fi
if ! command -v psql >/dev/null 2>&1; then
fail "psql not installed — cannot run RDS phase"
return
fi
info "DATABASE_URL host: $(printf '%s' "$DATABASE_URL" | sed -E 's#.*@([^/]+)/.*#\1#')"
# PG version sanity
local pg_version
pg_version=$(psql "$DATABASE_URL" -tA -c "SELECT version()" 2>/dev/null | head -1)
if [ -z "$pg_version" ]; then
fail "cannot connect to RDS — check DATABASE_URL"
return
fi
ok "PG reachable: $pg_version"
# Alembic head
info "checking alembic head against expected 0003"
cd "$BACKEND_DIR"
if PYTHONPATH=. uv run alembic current 2>/tmp/verify_alembic.err | tee /tmp/verify_alembic.log | grep -q "^0003"; then
ok "alembic current is at 0003 (workspace_id NOT NULL + UNIQUE(wid, tid))"
else
fail "alembic current is not at 0003 — see /tmp/verify_alembic.log"
fi
cd "$REPO_ROOT"
# PR8 tables present
info "checking PR8 tables exist"
local pr8_count
pr8_count=$(psql "$DATABASE_URL" -tA -c "
SELECT count(*) FROM information_schema.tables
WHERE table_schema = current_schema()
AND table_name IN ('service_accounts','api_keys','external_users');
" 2>/dev/null | tr -d ' ')
if [ "${pr8_count:-0}" -eq 3 ]; then
ok "service_accounts + api_keys + external_users all present"
else
fail "PR8 tables missing — expected 3, got ${pr8_count:-0}"
fi
# Partial index on api_keys: WHERE revoked_at IS NULL
info "checking idx_api_keys_active partial-index predicate"
local idx_def
idx_def=$(psql "$DATABASE_URL" -tA -c "
SELECT indexdef FROM pg_indexes
WHERE schemaname = current_schema()
AND indexname = 'idx_api_keys_active';
" 2>/dev/null | head -1)
if printf '%s' "$idx_def" | grep -qi 'WHERE.*revoked_at IS NULL'; then
ok "idx_api_keys_active has 'WHERE revoked_at IS NULL' predicate"
else
fail "idx_api_keys_active missing or wrong predicate — got: ${idx_def:-<none>}"
fi
# 4 business tables workspace_id NOT NULL
info "checking workspace_id NOT NULL on 4 business tables"
local nullable_rows
nullable_rows=$(psql "$DATABASE_URL" -tA -c "
SELECT table_name FROM information_schema.columns
WHERE table_schema = current_schema()
AND column_name = 'workspace_id'
AND table_name IN ('thread_meta','runs','feedback','run_events')
AND is_nullable = 'YES';
" 2>/dev/null)
if [ -z "$nullable_rows" ]; then
ok "thread_meta + runs + feedback + run_events all have workspace_id NOT NULL"
else
fail "workspace_id is nullable in: $(printf '%s' "$nullable_rows" | tr '\n' ' ')"
fi
# Unique (workspace_id, thread_id) on thread_meta
info "checking UNIQUE(workspace_id, thread_id) on thread_meta"
local uq_count
uq_count=$(psql "$DATABASE_URL" -tA -c "
SELECT count(*) FROM pg_indexes
WHERE schemaname = current_schema()
AND tablename = 'thread_meta'
AND indexdef ILIKE '%UNIQUE%workspace_id%thread_id%';
" 2>/dev/null | tr -d ' ')
if [ "${uq_count:-0}" -ge 1 ]; then
ok "UNIQUE(workspace_id, thread_id) constraint/index present"
else
fail "no UNIQUE(workspace_id, thread_id) on thread_meta"
fi
}
# ── phase: runtime ───────────────────────────────────────────────────────
phase_runtime() {
phase "RUNTIME — gateway health"
if ! command -v curl >/dev/null 2>&1; then
fail "curl missing — cannot run runtime phase"
return
fi
info "probing $GATEWAY_URL/health"
local health
health=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$GATEWAY_URL/health" 2>/dev/null || true)
if [ "$health" = "200" ]; then
ok "gateway /health returns 200"
else
fail "gateway /health returned '$health' — is \`make dev\` running?"
SKIPPED_PHASES+=("e2e (gateway not reachable)")
SKIP_E2E=1
fi
}
# ── phase: e2e ───────────────────────────────────────────────────────────
phase_e2e() {
phase "E2E — register 2 users + cross-workspace 404"
if [ "${SKIP_E2E:-0}" = "1" ]; then
warn "skipped because gateway probe failed"
return
fi
if ! command -v curl >/dev/null 2>&1; then
fail "curl missing"
return
fi
local tag
tag=$(date +%s)
local alice="alice-${tag}@verify.local"
local bob="bob-${tag}@verify.local"
local pw="VerifyStage0_${tag}"
local jar_a=/tmp/verify_alice_${tag}.cookies
local jar_b=/tmp/verify_bob_${tag}.cookies
rm -f "$jar_a" "$jar_b"
register_user() {
local jar="$1"; local email="$2"
curl -sS -c "$jar" -o /tmp/verify_register_$$.json -w '%{http_code}' \
-H 'Content-Type: application/json' \
-d "{\"email\":\"$email\",\"password\":\"$pw\"}" \
"$GATEWAY_URL/api/auth/register"
}
info "registering Alice + Bob"
local code_a code_b
code_a=$(register_user "$jar_a" "$alice")
code_b=$(register_user "$jar_b" "$bob")
if [ "$code_a" = "201" ] && [ "$code_b" = "201" ]; then
ok "both users registered (201 / 201)"
else
fail "registration failed: alice=$code_a, bob=$code_b"
return
fi
# Pull CSRF tokens from cookie jars (csrf_token cookie value, 4th-from-last field).
csrf_from_jar() {
awk '$6 == "csrf_token" { print $7 }' "$1" | tail -1
}
local csrf_a csrf_b
csrf_a=$(csrf_from_jar "$jar_a")
csrf_b=$(csrf_from_jar "$jar_b")
if [ -n "$csrf_a" ] && [ -n "$csrf_b" ]; then
ok "CSRF token captured for both sessions"
else
fail "CSRF cookie missing (alice='${csrf_a:0:8}...', bob='${csrf_b:0:8}...')"
return
fi
create_thread() {
local jar="$1"; local csrf="$2"; local tid="$3"
curl -sS -b "$jar" -o /tmp/verify_thread_$$.json -w '%{http_code}' \
-H 'Content-Type: application/json' \
-H "X-CSRF-Token: $csrf" \
-d "{\"thread_id\":\"$tid\"}" \
"$GATEWAY_URL/api/threads"
}
local tid_a="verify-alice-${tag}"
local tid_b="verify-bob-${tag}"
info "Alice creates thread $tid_a; Bob creates thread $tid_b"
local cta ctb
cta=$(create_thread "$jar_a" "$csrf_a" "$tid_a")
ctb=$(create_thread "$jar_b" "$csrf_b" "$tid_b")
if [ "$cta" = "200" ] && [ "$ctb" = "200" ]; then
ok "both threads created (200 / 200)"
else
fail "thread creation failed: alice=$cta, bob=$ctb"
return
fi
# Cross access: Alice tries to GET Bob's thread → must be 404 (per PR6 contract:
# cross-workspace returns 404, not 403, to avoid leaking existence).
info "Alice → Bob's thread (GET)"
local cross_get
cross_get=$(curl -sS -b "$jar_a" -o /dev/null -w '%{http_code}' "$GATEWAY_URL/api/threads/$tid_b")
if [ "$cross_get" = "404" ]; then
ok "cross-workspace GET returns 404 (no existence leak)"
else
fail "cross-workspace GET returned '$cross_get', expected 404 — PR6 isolation broken"
fi
info "Alice → Bob's thread (DELETE)"
local cross_del
cross_del=$(curl -sS -b "$jar_a" -X DELETE -H "X-CSRF-Token: $csrf_a" \
-o /dev/null -w '%{http_code}' "$GATEWAY_URL/api/threads/$tid_b")
if [ "$cross_del" = "404" ]; then
ok "cross-workspace DELETE returns 404"
else
fail "cross-workspace DELETE returned '$cross_del', expected 404"
fi
# Same-workspace GET — sanity check Alice can still reach her own thread.
info "Alice → Alice's thread (sanity)"
local same_get
same_get=$(curl -sS -b "$jar_a" -o /dev/null -w '%{http_code}' "$GATEWAY_URL/api/threads/$tid_a")
if [ "$same_get" = "200" ]; then
ok "same-workspace GET returns 200 (isolation is not over-blocking)"
else
fail "same-workspace GET returned '$same_get', expected 200"
fi
info "cookie jars left in /tmp for debugging: $jar_a $jar_b"
}
# ── main ─────────────────────────────────────────────────────────────────
main() {
local args=("$@")
if [ ${#args[@]} -eq 0 ]; then
args=(static paths rds runtime e2e)
fi
printf '%s%sStage 0 verification — %s%s\n' "$C_BOLD" "$C_BLUE" "$(date)" "$C_RESET"
printf 'Repo root: %s\n' "$REPO_ROOT"
printf 'Phases: %s\n' "${args[*]}"
for phase_name in "${args[@]}"; do
case "$phase_name" in
static) phase_static ;;
paths) phase_paths ;;
rds) phase_rds ;;
runtime) phase_runtime ;;
e2e) phase_e2e ;;
all)
phase_static; phase_paths; phase_rds; phase_runtime; phase_e2e
;;
*)
warn "unknown phase: $phase_name"
;;
esac
done
printf '\n%s%s──── summary ────%s\n' "$C_BOLD" "$C_BLUE" "$C_RESET"
printf ' %s%d passed%s %s%d failed%s %s%d warn%s\n' \
"$C_GREEN" "$PASS_COUNT" "$C_RESET" \
"$C_RED" "$FAIL_COUNT" "$C_RESET" \
"$C_YELLOW" "$WARN_COUNT" "$C_RESET"
if [ ${#SKIPPED_PHASES[@]} -gt 0 ]; then
printf ' skipped: %s\n' "${SKIPPED_PHASES[*]}"
fi
if [ ${#FAILED_STEPS[@]} -gt 0 ]; then
printf '\n%sfailing steps:%s\n' "$C_RED" "$C_RESET"
for step in "${FAILED_STEPS[@]}"; do
printf ' ✗ %s\n' "$step"
done
fi
[ "$FAIL_COUNT" -eq 0 ]
}
main "$@"
+50 -4
View File
@@ -36,8 +36,16 @@ from fastapi import FastAPI, Request, Response
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.types import ASGIApp
from app.gateway.auth.models import User
from app.gateway.auth.models import ActiveWorkspace, User
from app.gateway.authz import AuthContext, Permissions
from deerflow.runtime.user_context import (
reset_current_user,
set_current_user,
)
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
# Default permission set granted to the stub user. Mirrors `_ALL_PERMISSIONS`
# in authz.py — kept inline so the tests don't import a private symbol.
@@ -67,22 +75,55 @@ class _StubAuthMiddleware(BaseHTTPMiddleware):
Mirrors what production ``AuthMiddleware`` does after the JWT decode
+ DB lookup short-circuit, so ``@require_permission`` finds an
authenticated context and skips its own re-authentication path.
Optionally stamps the workspace contextvar too — needed by the PR6
decorator path that calls ``get_effective_workspace_id()`` before
delegating to ``check_access``.
"""
def __init__(self, app: ASGIApp, user_factory: Callable[[], User]) -> None:
def __init__(
self,
app: ASGIApp,
user_factory: Callable[[], User],
workspace_factory: Callable[[], ActiveWorkspace | None] | None = None,
override_user_contextvar: bool = False,
) -> None:
super().__init__(app)
self._user_factory = user_factory
self._workspace_factory = workspace_factory
# Tests that only need ``request.state.auth`` (the @require_permission
# path) keep the autouse user contextvar — flipping it to a per-call
# UUID would break legacy tests whose routes resolve paths via
# ``get_effective_user_id()``. Cross-user / cross-workspace tests opt
# in by setting this flag so the contextvar matches the request user.
self._override_user_contextvar = override_user_contextvar
async def dispatch(self, request: Request, call_next: Callable) -> Response:
user = self._user_factory()
request.state.user = user
request.state.auth = AuthContext(user=user, permissions=list(_STUB_PERMISSIONS))
return await call_next(request)
user_token = set_current_user(user) if self._override_user_contextvar else None
ws_token = None
if self._workspace_factory is not None:
workspace = self._workspace_factory()
if workspace is not None:
request.state.workspace = workspace
ws_token = set_current_workspace(workspace)
try:
return await call_next(request)
finally:
if ws_token is not None:
reset_current_workspace(ws_token)
if user_token is not None:
reset_current_user(user_token)
def make_authed_test_app(
*,
user_factory: Callable[[], User] | None = None,
workspace_factory: Callable[[], ActiveWorkspace | None] | None = None,
override_user_contextvar: bool = False,
owner_check_passes: bool = True,
) -> FastAPI:
"""Build a FastAPI test app with stub auth + permissive thread_store.
@@ -103,7 +144,12 @@ def make_authed_test_app(
"""
factory = user_factory or _make_stub_user
app = FastAPI()
app.add_middleware(_StubAuthMiddleware, user_factory=factory)
app.add_middleware(
_StubAuthMiddleware,
user_factory=factory,
workspace_factory=workspace_factory,
override_user_contextvar=override_user_contextvar,
)
repo = MagicMock()
repo.check_access = AsyncMock(return_value=owner_check_passes)
+28
View File
@@ -0,0 +1,28 @@
# PR7 — boundary scan allowlist for direct LangGraph checkpoint/saver imports.
#
# Paths are relative to backend/. A path here means: "this file is permitted to
# import langgraph.checkpoint.* at runtime". Adding to this list requires a
# matching justification in the PR that introduces the new importer.
#
# Consumed by tests/test_workspace_boundary.py.
#
# Imports inside `if TYPE_CHECKING:` blocks are exempt automatically — they do
# not pull the symbol into runtime — so type-only references (e.g. annotations
# on `BaseCheckpointSaver` parameters) do NOT need to be listed here.
langgraph_checkpoint_importers = [
# Gateway thread plumbing: constructs an empty checkpoint when initialising
# a new thread's state via the LangGraph runtime.
"app/gateway/routers/threads.py",
# Harness checkpointer factories: the single authorised place to construct
# InMemorySaver / SqliteSaver / PostgresSaver implementations. Everywhere
# else must obtain a checkpointer via `app.gateway.deps.get_checkpointer`
# or `deerflow.runtime.checkpointer` helpers.
"packages/harness/deerflow/runtime/checkpointer/async_provider.py",
"packages/harness/deerflow/runtime/checkpointer/provider.py",
# Background run worker: uses `empty_checkpoint` to seed state for runs
# resumed from a missing/expired checkpoint id.
"packages/harness/deerflow/runtime/runs/worker.py",
]
+127
View File
@@ -15,6 +15,13 @@ import pytest
# Make 'app' and 'deerflow' importable from any working directory
sys.path.insert(0, str(Path(__file__).parent.parent))
sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "scripts"))
# Make 'fixtures.*' importable as plugin modules from this conftest.
sys.path.insert(0, str(Path(__file__).parent))
# Register fixture plugin modules so tests can request fixtures by name
# without ad-hoc imports. ``fixtures.postgres`` provides
# ``postgres_container`` (session-scoped) and ``postgres_url`` (per-test).
pytest_plugins = ["fixtures.postgres"]
# Break the circular import chain that exists in production code:
# deerflow.subagents.__init__
@@ -38,6 +45,95 @@ _executor_mock.get_background_task_result = MagicMock()
sys.modules["deerflow.subagents.executor"] = _executor_mock
# ---------------------------------------------------------------------------
# Auto-seed test workspace + user when Base.metadata.create_all() runs
# ---------------------------------------------------------------------------
#
# PR6 makes every business-row INSERT carry ``workspace_id`` (resolved
# from the autouse workspace contextvar = "test-workspace-autouse"). The
# Stage 0 schema has a NOT NULL FK from those rows to ``workspaces`` and
# from ``workspaces.owner_id`` to ``users``. Without the seed below,
# every legacy repo test would fail with a FOREIGN KEY error the moment
# it tries to insert a thread.
#
# We register an ``after_create`` hook on ``Base.metadata`` so that
# whenever ``init_engine`` finishes ``create_all()`` (the auto-create
# path used by tests and dev), the two anchor rows are present. Alembic
# migration tests don't trigger create_all so they are unaffected and
# keep exercising real FK constraints in isolation.
def _register_test_seed_listener() -> None:
"""Attach an after_create hook that seeds the autouse user + workspace."""
try:
from sqlalchemy import event, update
from sqlalchemy.dialects.postgresql import insert as pg_insert
from sqlalchemy.dialects.sqlite import insert as sqlite_insert
from deerflow.persistence.base import Base
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
except ImportError:
return
from datetime import UTC, datetime
def _seed(_target, connection, **kw): # noqa: ARG001
tables = {t.name for t in kw.get("tables", []) or []}
if "users" not in tables or "workspaces" not in tables:
return
dialect = connection.dialect.name
now = datetime.now(UTC)
# Seed both rows with a consistent ``default_workspace_id`` so the
# PR5 backfill script (which scans ``users.default_workspace_id IS
# NULL``) does not pick up the test fixtures as candidates.
# Insert user first with NULL default_workspace_id (chicken-and-egg
# with workspaces.owner_id FK), then workspace, then UPDATE the user
# row to point at the workspace so the PR5 backfill script does not
# pick up the autouse user as a candidate.
user_values = {
"id": "test-user-autouse",
"email": "test-user-autouse@local",
"password_hash": None,
"system_role": "user",
"created_at": now,
"oauth_provider": None,
"oauth_id": None,
"needs_setup": False,
"token_version": 0,
"default_workspace_id": None,
}
workspace_values = {
"id": "test-workspace-autouse",
"name": "Autouse Test Workspace",
"slug": "autouse-test",
"status": "active",
"owner_id": "test-user-autouse",
"created_at": now,
"updated_at": now,
}
if dialect == "sqlite":
user_stmt = sqlite_insert(UserRow.__table__).values(**user_values).on_conflict_do_nothing(index_elements=["id"])
ws_stmt = sqlite_insert(WorkspaceRow.__table__).values(**workspace_values).on_conflict_do_nothing(index_elements=["id"])
elif dialect == "postgresql":
user_stmt = pg_insert(UserRow.__table__).values(**user_values).on_conflict_do_nothing(index_elements=["id"])
ws_stmt = pg_insert(WorkspaceRow.__table__).values(**workspace_values).on_conflict_do_nothing(index_elements=["id"])
else:
return
connection.execute(user_stmt)
connection.execute(ws_stmt)
connection.execute(update(UserRow.__table__).where(UserRow.__table__.c.id == "test-user-autouse").where(UserRow.__table__.c.default_workspace_id.is_(None)).values(default_workspace_id="test-workspace-autouse"))
event.listen(Base.metadata, "after_create", _seed)
_register_test_seed_listener()
@pytest.fixture()
def provisioner_module():
"""Load docker/provisioner/app.py as an importable test module.
@@ -110,3 +206,34 @@ def _auto_user_context(request):
yield
finally:
reset_current_user(token)
@pytest.fixture(autouse=True)
def _auto_workspace_context(request):
"""Inject a default ``test-workspace-autouse`` into the workspace contextvar.
Mirror of :func:`_auto_user_context`. PR6 adds ``workspace_id=AUTO``
sentinels to every repository method; without an autouse workspace
fixture every legacy persistence test would raise RuntimeError.
Opt-out via ``@pytest.mark.no_auto_workspace``.
"""
if request.node.get_closest_marker("no_auto_workspace"):
yield
return
try:
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
except ImportError:
yield
return
workspace = SimpleNamespace(id="test-workspace-autouse", role="owner")
token = set_current_workspace(workspace)
try:
yield
finally:
reset_current_workspace(token)
View File
+82
View File
@@ -0,0 +1,82 @@
"""Postgres testcontainer fixtures for Stage 0 PR1.
Provides per-test ephemeral database isolation atop a single session-scoped
container. Tests marked ``@pytest.mark.postgres`` request the ``postgres_url``
fixture, which yields an asyncpg connection URL pointing at a freshly-created
database. The database is force-dropped after the test (any leaked connections
get pg_terminate_backend'd first).
Why per-database rather than per-schema:
asyncpg (the SQLAlchemy async driver we use) doesn't honor URL-embedded
search_path the way psycopg does. Per-database isolation is one extra
CREATE/DROP per test (~50ms), but lets test app code use its full schema
unchanged.
"""
from __future__ import annotations
import secrets
from collections.abc import Iterator
import pytest
@pytest.fixture(scope="session")
def postgres_container():
"""Session-scoped Postgres 16 container shared across all postgres tests.
Started once per pytest session. Subsequent tests piggy-back on the same
container; each gets its own database via the ``postgres_url`` fixture.
Skipped (and the test marked skip) if Docker is unavailable on the host —
testcontainers raises ``DockerException`` when it can't reach the daemon.
"""
try:
from testcontainers.postgres import PostgresContainer
except ImportError as exc: # pragma: no cover - install boundary
pytest.skip(f"testcontainers[postgres] not installed: {exc}")
try:
# Image tag aligned with the production Aliyun RDS (PostgreSQL 17.9
# confirmed by `make doctor` 2026-05-11). Bump together with RDS upgrades.
with PostgresContainer("postgres:17-alpine") as pg:
yield pg
except Exception as exc: # pragma: no cover - environment-dependent
# DockerException, ConnectionError, etc. — surface a skip rather than
# an error so devs without Docker can still run the rest of the suite.
pytest.skip(f"could not start Postgres container ({exc})")
@pytest.fixture
def postgres_url(postgres_container) -> Iterator[str]:
"""Per-test ephemeral database URL (asyncpg dialect).
Each invocation creates a unique database on the shared container and
yields its URL. Teardown force-drops the database, terminating any
backend connections the test forgot to close.
"""
import psycopg
from psycopg import sql
db_name = f"test_{secrets.token_hex(8)}"
raw = postgres_container.get_connection_url() # postgresql+psycopg2://...
# Strip the SQLAlchemy dialect prefix so plain psycopg can connect.
base = raw.replace("postgresql+psycopg2://", "postgresql://")
parent_url = base.rsplit("/", 1)[0] + "/postgres"
# CREATE DATABASE must run outside a transaction; psycopg autocommit=True.
with psycopg.connect(parent_url, autocommit=True) as conn:
conn.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(db_name)))
asyncpg_url = raw.replace("postgresql+psycopg2://", "postgresql+asyncpg://").rsplit("/", 1)[0] + f"/{db_name}"
try:
yield asyncpg_url
finally:
with psycopg.connect(parent_url, autocommit=True) as conn:
# Kick any leaked connections so DROP DATABASE doesn't block.
conn.execute(
"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE datname = %s AND pid <> pg_backend_pid()",
(db_name,),
)
conn.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(db_name)))
@@ -0,0 +1,357 @@
"""Alembic 0002 / 0003 round-trips on both Postgres and SQLite.
PR5 has two migrations:
* ``0002_business_tables_workspace`` — adds *nullable* ``workspace_id``
+ FK to ``workspaces`` on ``threads_meta`` / ``runs`` / ``feedback`` /
``run_events``, plus a composite index on threads_meta for the common
"list a workspace's threads for a user, newest first" query.
* ``0003_business_tables_workspace_not_null`` — flips the column to
``NOT NULL`` (refusing to upgrade if NULL rows remain) and adds the
UNIQUE (workspace_id, thread_id) index on ``threads_meta``.
Pattern mirrors ``test_alembic_default_workspace_id.py``: synchronous
test bodies (alembic's command layer is sync, ``env.py`` calls
``asyncio.run`` internally — running under pytest-anyio would explode
that nested event loop). The pre-migration schema is bootstrapped with
just the columns those migrations touch, so we don't depend on the
production ORM staying frozen.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.exc import IntegrityError
_ALEMBIC_INI = Path(__file__).resolve().parents[1] / "packages" / "harness" / "deerflow" / "persistence" / "migrations" / "alembic.ini"
# Tables the PR5 migrations touch.
_BUSINESS_TABLES = ("threads_meta", "runs", "feedback", "run_events")
def _make_alembic_config(url: str) -> Config:
cfg = Config(str(_ALEMBIC_INI))
cfg.set_main_option("sqlalchemy.url", url)
return cfg
def _bootstrap_pre_pr5_schema(sync_url: str) -> None:
"""Create the post-PR4 schema PR5 migrations expect.
Includes ``workspaces`` (FK target), ``users`` (already has
``default_workspace_id`` from 0001 — but we skip 0001 here and bootstrap
the columns directly so the migration's behaviour can be tested in
isolation), and the four business tables.
"""
engine = create_engine(sync_url)
with engine.begin() as conn:
conn.execute(
text(
"""
CREATE TABLE workspaces (
id VARCHAR(36) PRIMARY KEY,
name VARCHAR(64) NOT NULL
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE users (
id VARCHAR(36) PRIMARY KEY,
email VARCHAR(255) NOT NULL,
default_workspace_id VARCHAR(36)
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE threads_meta (
thread_id VARCHAR(64) PRIMARY KEY,
user_id VARCHAR(64),
updated_at TIMESTAMP
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE runs (
run_id VARCHAR(64) PRIMARY KEY,
thread_id VARCHAR(64) NOT NULL,
user_id VARCHAR(64)
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE feedback (
feedback_id VARCHAR(64) PRIMARY KEY,
thread_id VARCHAR(64) NOT NULL,
run_id VARCHAR(64) NOT NULL,
user_id VARCHAR(64),
rating INTEGER NOT NULL
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE run_events (
id INTEGER PRIMARY KEY,
thread_id VARCHAR(64) NOT NULL,
run_id VARCHAR(64) NOT NULL,
user_id VARCHAR(64),
event_type VARCHAR(32) NOT NULL,
category VARCHAR(16) NOT NULL,
content TEXT,
seq INTEGER NOT NULL
)
"""
)
)
# Pretend 0001 has already run so 0002 is the next revision applied.
conn.execute(
text(
"""
CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL PRIMARY KEY)
"""
)
)
conn.execute(text("INSERT INTO alembic_version (version_num) VALUES ('0001_users_default_workspace')"))
engine.dispose()
def _assert_workspace_id_present(sync_url: str, *, nullable: bool) -> None:
engine = create_engine(sync_url)
insp = inspect(engine)
for table in _BUSINESS_TABLES:
cols = {c["name"]: c for c in insp.get_columns(table)}
assert "workspace_id" in cols, f"{table}: workspace_id missing; got {list(cols)}"
col = cols["workspace_id"]
assert col["nullable"] is nullable, f"{table}.workspace_id nullable expected {nullable}, got {col['nullable']}"
fks = insp.get_foreign_keys(table)
ws_fks = [fk for fk in fks if fk.get("referred_table") == "workspaces" and fk.get("constrained_columns") == ["workspace_id"]]
assert ws_fks, f"{table}: FK to workspaces missing; got {fks}"
idxs = {i["name"] for i in insp.get_indexes("threads_meta")}
assert "idx_threads_meta_workspace_user_updated" in idxs, f"threads_meta composite index missing; got {idxs}"
engine.dispose()
def _assert_workspace_id_absent(sync_url: str) -> None:
engine = create_engine(sync_url)
insp = inspect(engine)
for table in _BUSINESS_TABLES:
cols = {c["name"] for c in insp.get_columns(table)}
assert "workspace_id" not in cols, f"{table}: workspace_id still present; got {cols}"
idxs = {i["name"] for i in insp.get_indexes("threads_meta")}
assert "idx_threads_meta_workspace_user_updated" not in idxs, f"threads_meta composite index still present; got {idxs}"
engine.dispose()
# ---------- SQLite tests ----------------------------------------------------
def test_sqlite_upgrade_0002_adds_workspace_id_column() -> None:
"""0002 adds nullable workspace_id + FK + composite index on SQLite."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_assert_workspace_id_present(sync_url, nullable=True)
def test_sqlite_downgrade_0002_removes_workspace_id_column() -> None:
"""0002 downgrade drops the column + FK + composite index cleanly."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_assert_workspace_id_present(sync_url, nullable=True)
command.downgrade(cfg, "-1")
_assert_workspace_id_absent(sync_url)
# ---------- Postgres tests --------------------------------------------------
@pytest.mark.postgres
def test_postgres_upgrade_0002_adds_workspace_id_column(postgres_url: str) -> None:
"""Postgres: 0002 adds nullable workspace_id + FK + composite index."""
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_assert_workspace_id_present(sync_url, nullable=True)
@pytest.mark.postgres
def test_postgres_downgrade_0002_removes_workspace_id_column(postgres_url: str) -> None:
"""Postgres: 0002 downgrade drops column + FK + index cleanly."""
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_assert_workspace_id_present(sync_url, nullable=True)
command.downgrade(cfg, "-1")
_assert_workspace_id_absent(sync_url)
# ---------- 0003 helpers ----------------------------------------------------
def _populate_business_rows_for_0003(sync_url: str, workspace_id: str = "w1", thread_id: str = "t1", run_id: str = "r1") -> None:
"""Insert one row per business table with workspace_id populated.
Used as the pre-condition for the 0003 happy-path test: every row
has a non-NULL workspace_id, so the pre-flight count is zero and
the NOT NULL ALTER succeeds.
"""
engine = create_engine(sync_url)
with engine.begin() as conn:
conn.execute(text("INSERT INTO workspaces (id, name) VALUES (:id, :name)"), {"id": workspace_id, "name": "W"})
conn.execute(text("INSERT INTO threads_meta (thread_id, workspace_id) VALUES (:t, :w)"), {"t": thread_id, "w": workspace_id})
conn.execute(text("INSERT INTO runs (run_id, thread_id, workspace_id) VALUES (:r, :t, :w)"), {"r": run_id, "t": thread_id, "w": workspace_id})
conn.execute(text("INSERT INTO feedback (feedback_id, thread_id, run_id, rating, workspace_id) VALUES ('f1', :t, :r, 1, :w)"), {"t": thread_id, "r": run_id, "w": workspace_id})
conn.execute(text("INSERT INTO run_events (thread_id, run_id, event_type, category, seq, workspace_id) VALUES (:t, :r, 'x', 'lifecycle', 1, :w)"), {"t": thread_id, "r": run_id, "w": workspace_id})
engine.dispose()
def _assert_workspace_id_not_null_and_unique_index(sync_url: str) -> None:
engine = create_engine(sync_url)
insp = inspect(engine)
for table in _BUSINESS_TABLES:
cols = {c["name"]: c for c in insp.get_columns(table)}
assert cols["workspace_id"]["nullable"] is False, f"{table}.workspace_id should be NOT NULL after 0003; got {cols['workspace_id']}"
idxs = {i["name"]: i for i in insp.get_indexes("threads_meta")}
assert "idx_threads_meta_workspace_thread" in idxs, f"UNIQUE index missing; got {idxs}"
# SQLAlchemy reflection returns ``unique`` as int(1) on SQLite and
# bool(True) on Postgres — assert truthiness so both backends pass.
assert idxs["idx_threads_meta_workspace_thread"]["unique"], f"index should be UNIQUE; got {idxs['idx_threads_meta_workspace_thread']}"
engine.dispose()
# ---------- 0003 SQLite tests -----------------------------------------------
def test_sqlite_upgrade_0003_succeeds_with_populated_workspace_id() -> None:
"""0003 NOT NULL ALTER + UNIQUE index lands when no NULL rows remain."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_populate_business_rows_for_0003(sync_url)
command.upgrade(cfg, "0003_business_tables_workspace_not_null")
_assert_workspace_id_not_null_and_unique_index(sync_url)
def test_sqlite_upgrade_0003_requires_no_null_workspace_id() -> None:
"""0003 refuses to upgrade if any business row still has workspace_id=NULL."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0002_business_tables_workspace")
# Leave one threads_meta row with workspace_id NULL.
engine = create_engine(sync_url)
with engine.begin() as conn:
conn.execute(text("INSERT INTO threads_meta (thread_id) VALUES ('t-orphan')"))
engine.dispose()
with pytest.raises(RuntimeError, match="Cannot ALTER"):
command.upgrade(cfg, "0003_business_tables_workspace_not_null")
def test_sqlite_threads_meta_unique_workspace_thread() -> None:
"""After 0003, duplicate (workspace_id, thread_id) raises IntegrityError on insert."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_populate_business_rows_for_0003(sync_url)
command.upgrade(cfg, "0003_business_tables_workspace_not_null")
engine = create_engine(sync_url)
# threads_meta.thread_id is the table's PRIMARY KEY in our bootstrap
# schema, so a second row with the same thread_id would always fail.
# Use a *different* thread_id with the same (workspace_id, thread_id)
# pair would imply changing thread_id — that's not possible. Instead
# we drop the PK constraint via a fresh table that omits it, so the
# UNIQUE index is the only barrier.
with engine.begin() as conn:
conn.execute(text("CREATE TABLE threads_meta_test_unique (id INTEGER PRIMARY KEY, thread_id VARCHAR(64), workspace_id VARCHAR(36))"))
conn.execute(text("CREATE UNIQUE INDEX idx_test_unique ON threads_meta_test_unique (workspace_id, thread_id)"))
conn.execute(text("INSERT INTO threads_meta_test_unique (thread_id, workspace_id) VALUES ('t', 'w')"))
with pytest.raises(IntegrityError):
with engine.begin() as conn:
conn.execute(text("INSERT INTO threads_meta_test_unique (thread_id, workspace_id) VALUES ('t', 'w')"))
engine.dispose()
# ---------- 0003 Postgres tests ---------------------------------------------
@pytest.mark.postgres
def test_postgres_upgrade_0003_succeeds_with_populated_workspace_id(postgres_url: str) -> None:
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "0002_business_tables_workspace")
_populate_business_rows_for_0003(sync_url)
command.upgrade(cfg, "0003_business_tables_workspace_not_null")
_assert_workspace_id_not_null_and_unique_index(sync_url)
@pytest.mark.postgres
def test_postgres_upgrade_0003_requires_no_null_workspace_id(postgres_url: str) -> None:
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr5_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "0002_business_tables_workspace")
engine = create_engine(sync_url)
with engine.begin() as conn:
conn.execute(text("INSERT INTO threads_meta (thread_id) VALUES ('t-orphan')"))
engine.dispose()
with pytest.raises(RuntimeError, match="Cannot ALTER"):
command.upgrade(cfg, "0003_business_tables_workspace_not_null")
@@ -0,0 +1,148 @@
"""Alembic revision 0001 round-trips on both Postgres and SQLite.
Verifies the first DeerFlow migration adds `users.default_workspace_id`
(with FK to `workspaces`) on upgrade and removes it on downgrade. Both
backends are exercised because the migration relies on
`op.batch_alter_table` for SQLite ALTER compatibility — we want to know
if either dialect regresses.
These tests are synchronous: alembic's command layer is sync, and our
`env.py` calls `asyncio.run(...)` internally. Running under
pytest-anyio would put us inside an event loop and crash that
`asyncio.run` call, so we keep the test bodies plain `def`.
"""
from __future__ import annotations
import tempfile
from pathlib import Path
import pytest
from alembic import command
from alembic.config import Config
from sqlalchemy import create_engine, inspect, text
_ALEMBIC_INI = Path(__file__).resolve().parents[1] / "packages" / "harness" / "deerflow" / "persistence" / "migrations" / "alembic.ini"
def _make_alembic_config(url: str) -> Config:
cfg = Config(str(_ALEMBIC_INI))
cfg.set_main_option("sqlalchemy.url", url)
return cfg
def _bootstrap_pre_pr4_schema(sync_url: str) -> None:
"""Create the minimal pre-PR4 schema the migration needs to ALTER.
Only `users` (without `default_workspace_id`) and `workspaces` (just `id`)
are required so the FK target resolves. The rest of the production
schema is irrelevant to this migration.
"""
engine = create_engine(sync_url)
with engine.begin() as conn:
conn.execute(
text(
"""
CREATE TABLE workspaces (
id VARCHAR(36) PRIMARY KEY,
name VARCHAR(64) NOT NULL
)
"""
)
)
conn.execute(
text(
"""
CREATE TABLE users (
id VARCHAR(36) PRIMARY KEY,
email VARCHAR(255) NOT NULL
)
"""
)
)
engine.dispose()
def _assert_column_present(sync_url: str) -> None:
engine = create_engine(sync_url)
insp = inspect(engine)
cols = {c["name"] for c in insp.get_columns("users")}
assert "default_workspace_id" in cols, f"column missing; got {cols}"
fks = insp.get_foreign_keys("users")
fk_to_ws = [fk for fk in fks if fk.get("referred_table") == "workspaces"]
assert fk_to_ws, f"FK to workspaces missing; got {fks}"
assert fk_to_ws[0]["constrained_columns"] == ["default_workspace_id"]
assert fk_to_ws[0]["referred_columns"] == ["id"]
engine.dispose()
def _assert_column_absent(sync_url: str) -> None:
engine = create_engine(sync_url)
insp = inspect(engine)
cols = {c["name"] for c in insp.get_columns("users")}
assert "default_workspace_id" not in cols, f"column still present; got {cols}"
engine.dispose()
# ---------- SQLite tests ----------------------------------------------------
def test_sqlite_upgrade_adds_default_workspace_id_with_fk() -> None:
"""SQLite: upgrade 0001 adds column + FK; batch_alter_table works."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr4_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0001_users_default_workspace")
_assert_column_present(sync_url)
def test_sqlite_downgrade_removes_default_workspace_id() -> None:
"""SQLite: downgrade 0001 removes the column it added."""
with tempfile.TemporaryDirectory() as tmp:
db_path = Path(tmp) / "test.db"
sync_url = f"sqlite:///{db_path}"
async_url = f"sqlite+aiosqlite:///{db_path}"
_bootstrap_pre_pr4_schema(sync_url)
cfg = _make_alembic_config(async_url)
command.upgrade(cfg, "0001_users_default_workspace")
_assert_column_present(sync_url)
command.downgrade(cfg, "-1")
_assert_column_absent(sync_url)
# ---------- Postgres tests --------------------------------------------------
@pytest.mark.postgres
def test_postgres_upgrade_adds_default_workspace_id_with_fk(postgres_url: str) -> None:
"""Postgres: upgrade 0001 adds column + FK pointing at workspaces(id)."""
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr4_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "head")
_assert_column_present(sync_url)
@pytest.mark.postgres
def test_postgres_downgrade_removes_default_workspace_id(postgres_url: str) -> None:
"""Postgres: downgrade 0001 cleanly drops the FK and column."""
sync_url = postgres_url.replace("+asyncpg", "+psycopg")
_bootstrap_pre_pr4_schema(sync_url)
cfg = _make_alembic_config(postgres_url)
command.upgrade(cfg, "head")
_assert_column_present(sync_url)
command.downgrade(cfg, "-1")
_assert_column_absent(sync_url)
+156
View File
@@ -0,0 +1,156 @@
"""Schema tests for ``ApiKeyRow`` (Stage 0 PR8).
PR8 is schema-only — no repository class, no API. Tests exercise raw
ORM behaviour: column-level UNIQUE on key_prefix, the dual-dialect
partial index DDL (sqlite_where + postgresql_where), and CASCADE on
service_account delete.
"""
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from sqlalchemy import delete
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.api_key import ApiKeyRow
from deerflow.persistence.service_account import ServiceAccountRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def _setup(tmp_path):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
return get_session_factory()
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_parents(sf) -> None:
now = datetime.now(UTC)
async with sf() as session:
session.add(UserRow(id="u-alice", email="alice@example.com"))
await session.commit()
async with sf() as session:
session.add(WorkspaceRow(id="w-1", name="Alice WS", slug="alice", owner_id="u-alice"))
await session.commit()
async with sf() as session:
session.add(
ServiceAccountRow(
id="sa-1",
workspace_id="w-1",
name="bot",
role="member",
identity_mode="collapsed",
status="active",
created_by="u-alice",
created_at=now,
updated_at=now,
)
)
await session.commit()
def _make_key(*, key_id: str, prefix: str, revoked_at: datetime | None = None) -> ApiKeyRow:
now = datetime.now(UTC)
return ApiKeyRow(
id=key_id,
service_account_id="sa-1",
key_prefix=prefix,
key_hash="0" * 64,
name="default",
scopes="",
rate_limit_rpm=None,
expires_at=None,
last_used_at=None,
revoked_at=revoked_at,
created_at=now,
)
# ---------------------------------------------------------------------------
# T8.4-1 — column-level UNIQUE on key_prefix
# ---------------------------------------------------------------------------
async def test_unique_key_prefix_enforced(tmp_path):
"""Two api_key rows sharing the same key_prefix raise IntegrityError."""
sf = await _setup(tmp_path)
try:
await _seed_parents(sf)
async with sf() as session:
session.add(_make_key(key_id="ak-1", prefix="dfk_live_abc12345"))
await session.commit()
with pytest.raises(IntegrityError):
async with sf() as session:
session.add(_make_key(key_id="ak-2", prefix="dfk_live_abc12345"))
await session.commit()
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# T8.4-2 — partial index DDL covers both SQLite and Postgres
# ---------------------------------------------------------------------------
def test_active_index_declares_both_dialect_where_clauses():
"""idx_api_keys_active must compile to a partial index on both drivers.
The plan locks ``sqlite_where`` *and* ``postgresql_where`` so the same
Index emits a partial index regardless of backend (Stage 0 dev still
runs SQLite locally; production is Postgres). Both ``dialect_options``
entries must be present.
"""
active_idx = next((idx for idx in ApiKeyRow.__table__.indexes if idx.name == "idx_api_keys_active"), None)
assert active_idx is not None, "idx_api_keys_active not declared"
sqlite_where = active_idx.dialect_options.get("sqlite", {}).get("where")
postgres_where = active_idx.dialect_options.get("postgresql", {}).get("where")
assert sqlite_where is not None, "sqlite_where missing on idx_api_keys_active"
assert postgres_where is not None, "postgresql_where missing on idx_api_keys_active"
assert "revoked_at IS NULL" in str(sqlite_where)
assert "revoked_at IS NULL" in str(postgres_where)
# ---------------------------------------------------------------------------
# T8.4-3 — CASCADE on service_account delete
# ---------------------------------------------------------------------------
async def test_cascade_on_service_account_delete(tmp_path):
"""Deleting the parent service_account removes all child api_keys."""
sf = await _setup(tmp_path)
try:
await _seed_parents(sf)
async with sf() as session:
session.add(_make_key(key_id="ak-cascade-1", prefix="dfk_live_cascade1"))
session.add(_make_key(key_id="ak-cascade-2", prefix="dfk_live_cascade2"))
await session.commit()
async with sf() as session:
await session.execute(delete(ServiceAccountRow).where(ServiceAccountRow.id == "sa-1"))
await session.commit()
async with sf() as session:
row1 = await session.get(ApiKeyRow, "ak-cascade-1")
row2 = await session.get(ApiKeyRow, "ak-cascade-2")
assert row1 is None
assert row2 is None
finally:
await _cleanup()
+5 -5
View File
@@ -101,7 +101,7 @@ def test_create_and_decode_token():
import os
os.environ["AUTH_JWT_SECRET"] = "test-secret-key-for-jwt-testing-minimum-32-chars"
token = create_access_token(user_id)
token = create_access_token(user_id, workspace_id="ws-test", role="owner")
assert isinstance(token, str)
payload = decode_token(token)
@@ -132,7 +132,7 @@ def test_decode_token_invalid():
def test_create_token_custom_expiry():
"""Custom expiry is respected."""
user_id = str(uuid4())
token = create_access_token(user_id, expires_delta=timedelta(hours=1))
token = create_access_token(user_id, expires_delta=timedelta(hours=1), workspace_id="ws-test", role="owner")
payload = decode_token(token)
assert payload is not None
assert payload.sub == user_id
@@ -420,7 +420,7 @@ def test_jwt_encodes_ver():
from app.gateway.auth.errors import TokenError
os.environ["AUTH_JWT_SECRET"] = "test-secret-key-for-jwt-testing-minimum-32-chars"
token = create_access_token(str(uuid4()), token_version=3)
token = create_access_token(str(uuid4()), token_version=3, workspace_id="ws-test", role="owner")
payload = decode_token(token)
assert not isinstance(payload, TokenError)
assert payload.ver == 3
@@ -433,7 +433,7 @@ def test_jwt_default_ver_zero():
from app.gateway.auth.errors import TokenError
os.environ["AUTH_JWT_SECRET"] = "test-secret-key-for-jwt-testing-minimum-32-chars"
token = create_access_token(str(uuid4()))
token = create_access_token(str(uuid4()), workspace_id="ws-test", role="owner")
payload = decode_token(token)
assert not isinstance(payload, TokenError)
assert payload.ver == 0
@@ -447,7 +447,7 @@ def test_token_version_mismatch_rejects():
os.environ["AUTH_JWT_SECRET"] = "test-secret-key-for-jwt-testing-minimum-32-chars"
user_id = str(uuid4())
token = create_access_token(user_id, token_version=0)
token = create_access_token(user_id, token_version=0, workspace_id="ws-test", role="owner")
mock_user = User(id=user_id, email="test@example.com", password_hash="hash", token_version=1)
+1 -1
View File
@@ -69,7 +69,7 @@ def test_decode_token_returns_token_error_on_malformed():
def test_decode_token_returns_payload_on_valid():
_setup_config()
token = create_access_token("user-123")
token = create_access_token("user-123", workspace_id="ws-test", role="owner")
result = decode_token(token)
assert not isinstance(result, TokenError)
assert result.sub == "user-123"
+52
View File
@@ -0,0 +1,52 @@
"""JWT carries wid + role claims (Stage 0 PR4).
Tokens issued after PR4 must include `wid` (workspace_id) and `role`
(owner/admin/member) so the AuthMiddleware can resolve the active
workspace without a DB hit. Legacy token compatibility lives in
:mod:`test_legacy_token_compat`.
"""
from __future__ import annotations
from uuid import uuid4
import jwt
import pytest
from app.gateway.auth import create_access_token, decode_token
from app.gateway.auth.config import get_auth_config
@pytest.fixture(autouse=True)
def _stable_jwt_secret(monkeypatch):
"""Pin a deterministic JWT secret across tests in this module."""
monkeypatch.setenv("AUTH_JWT_SECRET", "test-secret-key-for-jwt-testing-minimum-32-chars")
yield
def test_jwt_includes_wid_and_role() -> None:
"""create_access_token records wid + role in the encoded payload."""
user_id = str(uuid4())
workspace_id = str(uuid4())
token = create_access_token(user_id, workspace_id=workspace_id, role="owner")
raw = jwt.decode(token, get_auth_config().jwt_secret, algorithms=["HS256"])
assert raw["sub"] == user_id
assert raw["wid"] == workspace_id
assert raw["role"] == "owner"
def test_decode_round_trip_keeps_wid_and_role() -> None:
"""decode_token returns a TokenPayload exposing wid + role attributes."""
user_id = str(uuid4())
workspace_id = str(uuid4())
token = create_access_token(user_id, workspace_id=workspace_id, role="member")
payload = decode_token(token)
# decode_token returns TokenError on failure — must be the success branch here.
assert hasattr(payload, "wid"), f"got {payload!r}"
assert payload.sub == user_id
assert payload.wid == workspace_id
assert payload.role == "member"
@@ -0,0 +1,90 @@
"""``GET /auth/me`` returns the user's workspace memberships.
Stage 0 PR4 T4.11. After PR4 the frontend needs to discover which
workspaces the current user belongs to (eventually for a picker UI).
Each membership entry surfaces ``id``, ``name``, ``slug``, ``role``
so the picker can render the list without a second roundtrip.
"""
from __future__ import annotations
import asyncio
import os
import pytest
from fastapi.testclient import TestClient
os.environ.setdefault("AUTH_JWT_SECRET", "test-secret-key-auth-me-workspaces-32+")
from app.gateway.auth.config import AuthConfig, set_auth_config
_TEST_SECRET = "test-secret-key-auth-me-workspaces-32+"
@pytest.fixture(autouse=True)
def _setup_auth(tmp_path):
from app.gateway import deps
from app.gateway.routers.auth import _SETUP_STATUS_COOLDOWN
from deerflow.persistence.engine import close_engine, init_engine
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
url = f"sqlite+aiosqlite:///{tmp_path}/auth_me.db"
asyncio.run(init_engine("sqlite", url=url, sqlite_dir=str(tmp_path)))
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
try:
yield
finally:
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
asyncio.run(close_engine())
@pytest.fixture()
def client(_setup_auth):
from app.gateway.app import create_app
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
yield TestClient(create_app())
def test_auth_me_returns_workspaces_list(client):
"""After registration, /auth/me lists the user's single personal workspace."""
client.post("/api/v1/auth/initialize", json={"email": "admin@example.com", "password": "Str0ng!Pass99"})
reg = client.post("/api/v1/auth/register", json={"email": "alice@example.com", "password": "Tr0ub4dor3a-strong!"})
user_id = reg.json()["id"]
resp = client.get("/api/v1/auth/me")
assert resp.status_code == 200, resp.text
body = resp.json()
assert body["id"] == user_id
assert body["email"] == "alice@example.com"
assert body["default_workspace_id"], "user should land with a default workspace"
assert len(body["workspaces"]) == 1, body
ws = body["workspaces"][0]
assert ws["id"] == body["default_workspace_id"]
assert ws["slug"] == "alice"
assert ws["role"] == "owner"
assert ws["name"] # whatever the helper picks — just sanity check it's non-empty
def test_auth_me_workspaces_isolated_per_user(client):
"""Two users see only their own workspaces in /auth/me."""
client.post("/api/v1/auth/initialize", json={"email": "admin@example.com", "password": "Str0ng!Pass99"})
client.post("/api/v1/auth/register", json={"email": "alice@example.com", "password": "Tr0ub4dor3a-strong!"})
a_id = client.get("/api/v1/auth/me").json()["id"]
a_workspaces = client.get("/api/v1/auth/me").json()["workspaces"]
client.cookies.clear()
client.post("/api/v1/auth/register", json={"email": "bob@example.com", "password": "Tr0ub4dor3a-strong!"})
b_id = client.get("/api/v1/auth/me").json()["id"]
b_workspaces = client.get("/api/v1/auth/me").json()["workspaces"]
assert a_id != b_id
assert {w["id"] for w in a_workspaces}.isdisjoint({w["id"] for w in b_workspaces})
assert {w["slug"] for w in a_workspaces} == {"alice"}
assert {w["slug"] for w in b_workspaces} == {"bob"}
@@ -0,0 +1,131 @@
"""AuthMiddleware injects the workspace ContextVar (Stage 0 PR4 T4.7).
After PR4 every authenticated request has a workspace bound on
``deerflow.runtime.workspace_context._current_workspace``. The middleware
populates it from the JWT's ``wid`` / ``role`` claims, mirrors what it
already does for ``user_context``, and tears both down in a single
``try/finally`` so leaks don't cross requests.
Legacy 4-field tokens (no ``wid``) are rejected upstream by
``decode_token`` (T4.6) those should never reach the workspace
injection branch; this file pins that the 401 they trigger carries
``AuthErrorCode.WORKSPACE_REQUIRED``.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
from unittest.mock import AsyncMock, patch
from uuid import uuid4
import jwt
import pytest
from fastapi import FastAPI
from starlette.testclient import TestClient
from app.gateway.auth import create_access_token
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.models import User
from app.gateway.auth_middleware import AuthMiddleware
from deerflow.runtime.workspace_context import get_current_workspace
@pytest.fixture(autouse=True)
def _stable_jwt_secret(monkeypatch):
monkeypatch.setenv("AUTH_JWT_SECRET", "test-secret-key-for-jwt-testing-minimum-32-chars")
yield
def _make_app() -> FastAPI:
"""App with AuthMiddleware + an inspect route that surfaces the contextvar."""
app = FastAPI()
app.add_middleware(AuthMiddleware)
@app.get("/api/v1/auth/setup-status") # public — never gates on wid
async def setup_status():
return {"needs_setup": False}
@app.get("/api/models") # protected — exercises wid injection
async def inspect_workspace():
ws = get_current_workspace()
if ws is None:
return {"workspace": None}
return {"workspace": {"id": ws.id, "role": ws.role}}
return app
def _make_user(uid: str) -> User:
return User(id=uid, email="t@example.com", password_hash="hash", token_version=0)
def _make_legacy_token() -> str:
"""Encode a pre-PR4 JWT (no wid/role) directly."""
now = datetime.now(UTC)
payload = {
"sub": str(uuid4()),
"exp": now + timedelta(hours=1),
"iat": now,
"ver": 0,
}
return jwt.encode(payload, get_auth_config().jwt_secret, algorithm="HS256")
def test_new_jwt_injects_workspace_into_contextvar() -> None:
"""Cookie with wid+role → route observes the workspace via the contextvar."""
uid = str(uuid4())
token = create_access_token(uid, workspace_id="ws-abc", role="owner")
with patch("app.gateway.deps.get_local_provider") as fn:
fn.return_value.get_user = AsyncMock(return_value=_make_user(uid))
client = TestClient(_make_app())
res = client.get("/api/models", cookies={"access_token": token})
assert res.status_code == 200, res.text
assert res.json() == {"workspace": {"id": "ws-abc", "role": "owner"}}
def test_legacy_jwt_rejected_with_workspace_required() -> None:
"""No-wid tokens get 401 with AuthErrorCode.WORKSPACE_REQUIRED, not generic token_invalid."""
client = TestClient(_make_app())
res = client.get("/api/models", cookies={"access_token": _make_legacy_token()})
assert res.status_code == 401
assert res.json()["detail"]["code"] == "workspace_required"
def test_public_path_skips_workspace_check() -> None:
"""Public whitelist (e.g. /api/v1/auth/setup-status) does not require wid."""
client = TestClient(_make_app())
res = client.get("/api/v1/auth/setup-status") # no cookie at all
assert res.status_code == 200
@pytest.mark.no_auto_workspace
def test_workspace_contextvar_resets_between_requests() -> None:
"""After dispatch returns the contextvar must be clear (no leak across requests).
Why we test this: if the try/finally is wired only for user_context but
not workspace_context, two back-to-back requests can see each other's
workspace under asyncio task switching.
"""
uid = str(uuid4())
token = create_access_token(uid, workspace_id="ws-first", role="owner")
# First request resolves to ws-first
with patch("app.gateway.deps.get_local_provider") as fn:
fn.return_value.get_user = AsyncMock(return_value=_make_user(uid))
client = TestClient(_make_app())
res1 = client.get("/api/models", cookies={"access_token": token})
assert res1.json() == {"workspace": {"id": "ws-first", "role": "owner"}}
# Outside the request scope the contextvar must be empty again.
assert get_current_workspace() is None
# Second request with a different workspace must not see ws-first.
token2 = create_access_token(uid, workspace_id="ws-second", role="owner")
with patch("app.gateway.deps.get_local_provider") as fn:
fn.return_value.get_user = AsyncMock(return_value=_make_user(uid))
client = TestClient(_make_app())
res2 = client.get("/api/models", cookies={"access_token": token2})
assert res2.json() == {"workspace": {"id": "ws-second", "role": "owner"}}
+367
View File
@@ -0,0 +1,367 @@
"""Tests for ``scripts/backfill_workspace_id.py`` (Stage 0 PR5).
Each step in the three-step backfill is exercised in isolation against
a SQLite-on-disk database. The script wires in ``app.gateway.auth.workspace_slug``,
so we get the same slug semantics that the registration flow uses.
Pattern mirrors :mod:`test_workspace_repo`: ``init_engine`` + per-test
tmp_path, with explicit ``close_engine`` teardown so the singleton
session factory does not leak across tests.
"""
from __future__ import annotations
import uuid
import pytest
from sqlalchemy import select
from deerflow.persistence.feedback.model import FeedbackRow
from deerflow.persistence.models.run_event import RunEventRow
from deerflow.persistence.run.model import RunRow
from deerflow.persistence.thread_meta.model import ThreadMetaRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
from scripts.backfill_workspace_id import (
LEGACY_WORKSPACE_ID,
_ensure_legacy_workspace,
_step1_create_workspaces_for_users,
_step2_update_table_from_users,
_step3_assign_legacy_workspace,
backfill,
)
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
_BUSINESS_ROWS = (ThreadMetaRow, RunRow, FeedbackRow, RunEventRow)
@pytest.fixture(autouse=True)
def _relax_workspace_id_nullable():
"""Simulate alembic 0002 (pre-backfill) state during these tests.
PR6 T5.11 flipped ``workspace_id`` to ``nullable=False`` on the four
business ORM models production correctness comes from alembic 0003.
The backfill script's job is precisely to fill the rows that were
inserted between 0002 (column added, nullable) and 0003 (NOT NULL),
so tests for it must be able to insert NULL rows. We mutate
``column.nullable`` for the four tables before ``create_all`` runs,
then restore on teardown so other tests see the production shape.
"""
saved: list[tuple] = []
for model in _BUSINESS_ROWS:
col = model.__table__.c.workspace_id
saved.append((col, col.nullable))
col.nullable = True
try:
yield
finally:
for col, original in saved:
col.nullable = original
async def _init_engine(tmp_path):
from sqlalchemy import delete
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
sf = get_session_factory()
# The PR6 conftest seeds an autouse user + workspace so business-row FKs
# resolve in the wider test suite. Backfill tests model "fresh DB needs
# backfill" semantics, so wipe those rows here. Order: clear the FK
# pointer first, then the rows.
async with sf() as session:
await session.execute(delete(WorkspaceMembershipRow))
await session.execute(delete(WorkspaceRow).where(WorkspaceRow.id == "test-workspace-autouse"))
await session.execute(delete(UserRow).where(UserRow.id == "test-user-autouse"))
await session.commit()
return sf
async def _close():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_user(sf, *, email: str, default_workspace_id: str | None = None, system_role: str = "user") -> str:
user_id = str(uuid.uuid4())
async with sf() as session:
session.add(UserRow(id=user_id, email=email, default_workspace_id=default_workspace_id, system_role=system_role))
await session.commit()
return user_id
async def _seed_business_row(sf, model, **fields) -> None:
async with sf() as session:
session.add(model(**fields))
await session.commit()
# ---------------------------------------------------------------------------
# Step 1: per-user workspace creation
# ---------------------------------------------------------------------------
async def test_creates_workspace_per_user_without_default(tmp_path):
"""Each user with NULL default_workspace_id gets a workspace + owner membership."""
sf = await _init_engine(tmp_path)
try:
u_alice = await _seed_user(sf, email="alice@example.com")
u_bob = await _seed_user(sf, email="bob+spam@example.com")
count = await _step1_create_workspaces_for_users(sf, dry_run=False)
assert count == 2
async with sf() as session:
workspaces = (await session.execute(select(WorkspaceRow))).scalars().all()
memberships = (await session.execute(select(WorkspaceMembershipRow))).scalars().all()
users = {u.id: u for u in (await session.execute(select(UserRow))).scalars().all()}
# 2 workspaces, each with exactly one owner membership matching its user.
assert len(workspaces) == 2
assert len(memberships) == 2
owners_by_ws = {m.workspace_id: m.user_id for m in memberships if m.role == "owner"}
assert {m.role for m in memberships} == {"owner"}
for ws in workspaces:
assert owners_by_ws[ws.id] == ws.owner_id
assert users[ws.owner_id].default_workspace_id == ws.id
# Slug semantics: alice@ → "alice", bob+spam@ → "bob-spam".
slugs = {ws.slug for ws in workspaces}
assert slugs == {"alice", "bob-spam"}
_ = u_alice, u_bob # captured for readability
finally:
await _close()
async def test_step1_is_idempotent(tmp_path):
"""Second run is a no-op when every user already has a default_workspace_id."""
sf = await _init_engine(tmp_path)
try:
await _seed_user(sf, email="carol@example.com")
first = await _step1_create_workspaces_for_users(sf, dry_run=False)
assert first == 1
# Re-running picks up the just-populated default_workspace_id, so the
# candidate set is empty.
second = await _step1_create_workspaces_for_users(sf, dry_run=False)
assert second == 0
async with sf() as session:
ws_count = len((await session.execute(select(WorkspaceRow))).scalars().all())
mem_count = len((await session.execute(select(WorkspaceMembershipRow))).scalars().all())
assert ws_count == 1
assert mem_count == 1
finally:
await _close()
async def test_backfill_updates_4_tables_from_users(tmp_path):
"""Step 2 propagates each user's default_workspace_id into 4 business tables."""
sf = await _init_engine(tmp_path)
try:
user_id = await _seed_user(sf, email="dave@example.com")
# Pre-seed business rows owned by the user with NULL workspace_id.
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-1", user_id=user_id)
await _seed_business_row(sf, RunRow, run_id="r-1", thread_id="t-1", user_id=user_id)
await _seed_business_row(sf, FeedbackRow, feedback_id="f-1", thread_id="t-1", run_id="r-1", user_id=user_id, rating=1)
await _seed_business_row(sf, RunEventRow, thread_id="t-1", run_id="r-1", user_id=user_id, event_type="lifecycle_started", category="lifecycle", seq=1)
# Step 1 first so users.default_workspace_id is populated.
await _step1_create_workspaces_for_users(sf, dry_run=False)
async with sf() as session:
ws_id = (await session.execute(select(WorkspaceRow.id))).scalar_one()
# Step 2 updates each table.
for table in ("threads_meta", "runs", "feedback", "run_events"):
count = await _step2_update_table_from_users(sf, table, dry_run=False)
assert count == 1, table
async with sf() as session:
tm = (await session.execute(select(ThreadMetaRow))).scalar_one()
run = (await session.execute(select(RunRow))).scalar_one()
fb = (await session.execute(select(FeedbackRow))).scalar_one()
ev = (await session.execute(select(RunEventRow))).scalar_one()
assert tm.workspace_id == ws_id
assert run.workspace_id == ws_id
assert fb.workspace_id == ws_id
assert ev.workspace_id == ws_id
# Re-running Step 2 is a no-op (filtered by workspace_id IS NULL).
for table in ("threads_meta", "runs", "feedback", "run_events"):
assert await _step2_update_table_from_users(sf, table, dry_run=False) == 0
finally:
await _close()
async def test_backfill_step2_isolates_per_user(tmp_path):
"""Two users with different default workspaces get their own threads tagged independently."""
sf = await _init_engine(tmp_path)
try:
u_eve = await _seed_user(sf, email="eve@example.com")
u_frank = await _seed_user(sf, email="frank@example.com")
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-eve", user_id=u_eve)
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-frank", user_id=u_frank)
await _step1_create_workspaces_for_users(sf, dry_run=False)
await _step2_update_table_from_users(sf, "threads_meta", dry_run=False)
async with sf() as session:
rows = {r.thread_id: r.workspace_id for r in (await session.execute(select(ThreadMetaRow))).scalars().all()}
users = {u.id: u.default_workspace_id for u in (await session.execute(select(UserRow))).scalars().all()}
assert rows["t-eve"] == users[u_eve]
assert rows["t-frank"] == users[u_frank]
assert rows["t-eve"] != rows["t-frank"]
finally:
await _close()
async def test_step1_skips_blacklisted_base_slug(tmp_path):
"""A user with email like admin@... gets bumped past the slug blacklist via the walker."""
sf = await _init_engine(tmp_path)
try:
await _seed_user(sf, email="admin@example.com")
await _step1_create_workspaces_for_users(sf, dry_run=False)
async with sf() as session:
ws = (await session.execute(select(WorkspaceRow))).scalar_one()
# The walker treats "admin" as taken (blacklisted), so it falls
# through to "admin-2" — the same behaviour the registration flow
# uses for reserved slugs.
assert ws.slug == "admin-2"
finally:
await _close()
# ---------------------------------------------------------------------------
# Step 3: orphan rows -> legacy_workspace
# ---------------------------------------------------------------------------
async def test_backfill_orphan_rows_go_to_legacy_workspace(tmp_path):
"""Rows with user_id=NULL get assigned the legacy_workspace UUID after Step 3."""
sf = await _init_engine(tmp_path)
try:
# Seed a platform admin so the legacy workspace has an owner.
await _seed_user(sf, email="admin@example.com", system_role="admin")
# Orphan business rows (user_id=NULL): legacy data from before auth.
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-orphan", user_id=None)
await _seed_business_row(sf, RunRow, run_id="r-orphan", thread_id="t-orphan", user_id=None)
await _seed_business_row(sf, FeedbackRow, feedback_id="f-orphan", thread_id="t-orphan", run_id="r-orphan", user_id=None, rating=1)
await _seed_business_row(sf, RunEventRow, thread_id="t-orphan", run_id="r-orphan", user_id=None, event_type="legacy", category="lifecycle", seq=1)
# Ensure the anchor + reassign per table.
created = await _ensure_legacy_workspace(sf, dry_run=False)
assert created is True
for table in ("threads_meta", "runs", "feedback", "run_events"):
count = await _step3_assign_legacy_workspace(sf, table, dry_run=False)
assert count == 1, table
# Re-running the anchor helper is a no-op.
assert await _ensure_legacy_workspace(sf, dry_run=False) is False
async with sf() as session:
tm = (await session.execute(select(ThreadMetaRow))).scalar_one()
run = (await session.execute(select(RunRow))).scalar_one()
fb = (await session.execute(select(FeedbackRow))).scalar_one()
ev = (await session.execute(select(RunEventRow))).scalar_one()
legacy = (await session.execute(select(WorkspaceRow).where(WorkspaceRow.id == LEGACY_WORKSPACE_ID))).scalar_one()
legacy_mem = (await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.workspace_id == LEGACY_WORKSPACE_ID))).scalar_one()
assert tm.workspace_id == LEGACY_WORKSPACE_ID
assert run.workspace_id == LEGACY_WORKSPACE_ID
assert fb.workspace_id == LEGACY_WORKSPACE_ID
assert ev.workspace_id == LEGACY_WORKSPACE_ID
assert legacy.slug == "legacy"
assert legacy_mem.role == "owner"
finally:
await _close()
async def test_ensure_legacy_workspace_refuses_when_no_users(tmp_path):
"""ensure_legacy_workspace raises a clear error if the DB has no users."""
sf = await _init_engine(tmp_path)
try:
with pytest.raises(RuntimeError, match="no users exist"):
await _ensure_legacy_workspace(sf, dry_run=False)
finally:
await _close()
async def test_backfill_dry_run_does_not_write(tmp_path):
"""``backfill(..., dry_run=True)`` reports counts but writes nothing."""
sf = await _init_engine(tmp_path)
try:
await _seed_user(sf, email="admin@example.com", system_role="admin")
user_id = await _seed_user(sf, email="helen@example.com")
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-owned", user_id=user_id)
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-orphan", user_id=None)
await _seed_business_row(sf, RunRow, run_id="r-owned", thread_id="t-owned", user_id=user_id)
# Snapshot row counts BEFORE the dry run so we can confirm
# nothing changed AFTER.
async with sf() as session:
ws_before = len((await session.execute(select(WorkspaceRow))).scalars().all())
mem_before = len((await session.execute(select(WorkspaceMembershipRow))).scalars().all())
users_with_default_before = len((await session.execute(select(UserRow).where(UserRow.default_workspace_id.is_not(None)))).scalars().all())
report = await backfill(sf, dry_run=True)
assert report["dry_run"] is True
# Step 1 reports 2 candidates (admin + helen, both without default).
assert report["users_workspaces_created"] == 2
# Step 2 reports 0 because Step 1 didn't actually populate
# users.default_workspace_id under dry_run — the JOIN comes up empty.
assert report["threads_meta_from_users"] == 0
assert report["runs_from_users"] == 0
# Step 3 reports the 3 NULL business rows (t-owned, t-orphan, r-owned).
assert report["legacy_workspace_created"] is True
assert report["threads_meta_legacy"] == 2
assert report["runs_legacy"] == 1
# State did not change.
async with sf() as session:
ws_after = len((await session.execute(select(WorkspaceRow))).scalars().all())
mem_after = len((await session.execute(select(WorkspaceMembershipRow))).scalars().all())
users_with_default_after = len((await session.execute(select(UserRow).where(UserRow.default_workspace_id.is_not(None)))).scalars().all())
rows = (await session.execute(select(ThreadMetaRow.workspace_id))).scalars().all()
assert ws_after == ws_before
assert mem_after == mem_before
assert users_with_default_after == users_with_default_before
assert all(w is None for w in rows)
finally:
await _close()
async def test_full_backfill_orchestrator(tmp_path):
"""End-to-end: backfill() runs all three steps and reports per-step counts."""
sf = await _init_engine(tmp_path)
try:
await _seed_user(sf, email="admin@example.com", system_role="admin")
user_id = await _seed_user(sf, email="gina@example.com")
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-owned", user_id=user_id)
await _seed_business_row(sf, ThreadMetaRow, thread_id="t-orphan", user_id=None)
report = await backfill(sf, dry_run=False)
assert report["dry_run"] is False
# Two users were missing a default workspace (admin too — we
# didn't pre-populate admin's default_workspace_id).
assert report["users_workspaces_created"] == 2
assert report["threads_meta_from_users"] == 1
assert report["legacy_workspace_created"] is True
assert report["threads_meta_legacy"] == 1
async with sf() as session:
rows = {r.thread_id: r.workspace_id for r in (await session.execute(select(ThreadMetaRow))).scalars().all()}
assert rows["t-orphan"] == LEGACY_WORKSPACE_ID
assert rows["t-owned"] != LEGACY_WORKSPACE_ID
assert rows["t-owned"] is not None
finally:
await _close()
@@ -0,0 +1,108 @@
"""``change_password`` and ``login_local`` re-issue JWTs carrying wid + role.
Stage 0 PR4 T4.12. The contract:
- After ``POST /auth/change-password`` the new session cookie's JWT must
still encode the user's workspace under ``wid`` (and ``role='owner'``),
with ``ver`` bumped. Dropping ``wid`` here would lock the user out of
every protected endpoint immediately after a password change.
- Same for ``POST /auth/login/local`` it issues a fresh JWT and must
also encode ``wid``.
"""
from __future__ import annotations
import asyncio
import os
import jwt
import pytest
from fastapi.testclient import TestClient
os.environ.setdefault("AUTH_JWT_SECRET", "test-secret-key-change-password-wid-32x")
from app.gateway.auth.config import AuthConfig, set_auth_config
_TEST_SECRET = "test-secret-key-change-password-wid-32x"
@pytest.fixture(autouse=True)
def _setup_auth(tmp_path):
from app.gateway import deps
from app.gateway.routers.auth import _SETUP_STATUS_COOLDOWN
from deerflow.persistence.engine import close_engine, init_engine
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
url = f"sqlite+aiosqlite:///{tmp_path}/change_pwd.db"
asyncio.run(init_engine("sqlite", url=url, sqlite_dir=str(tmp_path)))
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
try:
yield
finally:
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
asyncio.run(close_engine())
@pytest.fixture()
def client(_setup_auth):
from app.gateway.app import create_app
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
yield TestClient(create_app())
def _decode(token: str) -> dict:
return jwt.decode(token, _TEST_SECRET, algorithms=["HS256"])
def _bootstrap_user(client) -> tuple[str, dict]:
"""Register a user and return (user_id, initial claims)."""
client.post("/api/v1/auth/initialize", json={"email": "admin@example.com", "password": "Str0ng!Pass99"})
resp = client.post("/api/v1/auth/register", json={"email": "alice@example.com", "password": "Tr0ub4dor3a-strong!"})
assert resp.status_code == 201, resp.text
return resp.json()["id"], _decode(resp.cookies["access_token"])
def test_change_password_keeps_wid_and_bumps_ver(client):
"""change_password re-signs the JWT with wid + role; ver moves forward."""
user_id, initial_claims = _bootstrap_user(client)
# /register set the csrf cookie; the matching header is required on
# the change-password POST (Double Submit Cookie pattern).
csrf = client.cookies.get("csrf_token")
assert csrf, "register must have set csrf_token cookie"
resp = client.post(
"/api/v1/auth/change-password",
json={"current_password": "Tr0ub4dor3a-strong!", "new_password": "Tr0ub4dor3a-strong2!"},
headers={"X-CSRF-Token": csrf},
)
assert resp.status_code == 200, resp.text
new_claims = _decode(resp.cookies["access_token"])
assert new_claims["sub"] == user_id
assert new_claims["wid"] == initial_claims["wid"], "wid must survive a password change"
assert new_claims["role"] == "owner"
assert new_claims["ver"] == initial_claims["ver"] + 1, "token_version must advance"
def test_login_issues_wid_carrying_jwt(client):
"""/auth/login/local issues a JWT that the workspace middleware will accept."""
user_id, _ = _bootstrap_user(client)
# Clear the session cookie set by /register so the login response is observed in isolation.
client.cookies.clear()
resp = client.post(
"/api/v1/auth/login/local",
data={"username": "alice@example.com", "password": "Tr0ub4dor3a-strong!"},
)
assert resp.status_code == 200, resp.text
claims = _decode(resp.cookies["access_token"])
assert claims["sub"] == user_id
assert claims.get("wid"), "login must include wid"
assert claims["role"] == "owner"
@@ -0,0 +1,90 @@
"""Stage 0 PR2 · default backend regression + sqlite-still-works.
Two facts pinned here:
T2.3 explicit ``database.backend: sqlite`` still produces a working
config (backwards-compat for users who deliberately stay on SQLite).
T2.4 ``config.example.yaml`` default ``database.backend`` is ``postgres``
(Stage 0 PR2 flipped this from sqlite).
"""
from __future__ import annotations
import json
from pathlib import Path
import pytest
import yaml
from deerflow.config.app_config import AppConfig
REPO_ROOT = Path(__file__).resolve().parents[2]
def _write_extensions_config(path: Path) -> None:
path.write_text(json.dumps({"mcpServers": {}, "skills": {}}), encoding="utf-8")
# ---------------------------------------------------------------------------
# T2.3 · sqlite backend regression
# ---------------------------------------------------------------------------
def test_explicit_sqlite_backend_still_works(tmp_path, monkeypatch) -> None:
"""A config with explicit ``database.backend: sqlite`` must still parse.
Pin: PR2 made postgres the example default. This test guarantees that
users who copy the SQLite fallback block to their config.yaml do not
silently regress.
"""
config_path = tmp_path / "config.yaml"
extensions_path = tmp_path / "extensions_config.json"
_write_extensions_config(extensions_path)
config_path.write_text(
yaml.safe_dump(
{
"models": [
{
"name": "test",
"use": "langchain_openai:ChatOpenAI",
"model": "gpt-4",
}
],
"database": {
"backend": "sqlite",
"sqlite_dir": "/custom/sqlite/path",
},
"sandbox": {"use": "deerflow.sandbox.local:LocalSandboxProvider"},
}
),
encoding="utf-8",
)
monkeypatch.setenv("DEER_FLOW_EXTENSIONS_CONFIG_PATH", str(extensions_path))
config = AppConfig.from_file(str(config_path))
assert config.database.backend == "sqlite"
assert config.database.sqlite_dir == "/custom/sqlite/path"
# ---------------------------------------------------------------------------
# T2.4 · default config.example.yaml uses postgres
# ---------------------------------------------------------------------------
def test_config_example_default_backend_is_postgres(monkeypatch) -> None:
"""Pin Stage 0 PR2's commitment: the example config defaults to Postgres.
Loaded directly from the on-disk ``config.example.yaml`` so any future
accidental flip back to sqlite would fail this test.
"""
example_path = REPO_ROOT / "config.example.yaml"
if not example_path.exists():
pytest.skip(f"config.example.yaml not found at {example_path}")
raw = yaml.safe_load(example_path.read_text(encoding="utf-8")) or {}
db = raw.get("database") or {}
assert db.get("backend") == "postgres", f"config.example.yaml database.backend is {db.get('backend')!r}; PR2 requires 'postgres' as the default"
# The PG URL must come from the env (referenced as $DATABASE_URL),
# never hardcoded with credentials.
assert db.get("postgres_url") == "$DATABASE_URL", f"postgres_url should be '$DATABASE_URL' env reference, got {db.get('postgres_url')!r}"
@@ -0,0 +1,137 @@
"""Lifespan hook backfills missing workspaces for pre-PR4 admins.
Stage 0 PR4 T4.13. Production scenario: a deployment that pre-dates
PR4 has an admin user whose ``users.default_workspace_id`` is NULL.
After the upgrade, the first time the app boots, the lifespan hook
must create the admin's personal workspace + owner membership so the
admin can immediately log in without hitting the post-PR4 workspace
gate (T4.7).
This test directly invokes ``_ensure_admin_user`` against a fixture
SQLite DB, simulating the upgrade path.
"""
from __future__ import annotations
import asyncio
import pytest
from fastapi import FastAPI
from app.gateway.auth.config import AuthConfig, set_auth_config
_TEST_SECRET = "test-secret-key-admin-backfill-32-chars"
@pytest.fixture(autouse=True)
def _setup(tmp_path):
from app.gateway import deps
from deerflow.persistence.engine import close_engine, init_engine
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
url = f"sqlite+aiosqlite:///{tmp_path}/admin_backfill.db"
asyncio.run(init_engine("sqlite", url=url, sqlite_dir=str(tmp_path)))
deps._cached_local_provider = None
deps._cached_repo = None
try:
yield
finally:
deps._cached_local_provider = None
deps._cached_repo = None
asyncio.run(close_engine())
async def _seed_pre_pr4_admin(email: str = "admin@example.com") -> str:
"""Insert an admin user with default_workspace_id=NULL (pre-PR4 state)."""
from app.gateway.deps import get_local_provider
provider = get_local_provider()
user = await provider.create_user(email=email, password="Str0ng!Pass99", system_role="admin")
# Belt + suspenders: pretend this user pre-dates PR4 even if the
# provider added a workspace_id (it does not today, but explicit
# is better).
user.default_workspace_id = None
await provider.update_user(user)
return str(user.id)
async def _read_workspace_state(user_id: str) -> dict:
from sqlalchemy import select
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
sf = get_session_factory()
async with sf() as session:
user = await session.get(UserRow, user_id)
memberships = (await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.user_id == user_id))).scalars().all()
workspaces = []
if memberships:
workspaces = (await session.execute(select(WorkspaceRow).where(WorkspaceRow.id.in_([m.workspace_id for m in memberships])))).scalars().all()
return {
"default_workspace_id": user.default_workspace_id if user else None,
"memberships": [(m.workspace_id, m.role) for m in memberships],
"workspaces": [(w.id, w.slug) for w in workspaces],
}
def test_ensure_admin_user_creates_missing_workspace():
"""Pre-PR4 admin without default_workspace_id → lifespan backfills it."""
from app.gateway.app import _ensure_admin_user
admin_id = asyncio.run(_seed_pre_pr4_admin())
before = asyncio.run(_read_workspace_state(admin_id))
assert before["default_workspace_id"] is None
assert before["workspaces"] == []
asyncio.run(_ensure_admin_user(FastAPI()))
after = asyncio.run(_read_workspace_state(admin_id))
assert after["default_workspace_id"] is not None, "lifespan should set default_workspace_id"
assert len(after["workspaces"]) == 1
ws_id, _slug = after["workspaces"][0]
assert after["memberships"] == [(ws_id, "owner")]
def test_ensure_admin_user_is_idempotent():
"""Running the lifespan hook twice does not create duplicate workspaces."""
from app.gateway.app import _ensure_admin_user
admin_id = asyncio.run(_seed_pre_pr4_admin())
asyncio.run(_ensure_admin_user(FastAPI()))
state_after_first = asyncio.run(_read_workspace_state(admin_id))
asyncio.run(_ensure_admin_user(FastAPI()))
state_after_second = asyncio.run(_read_workspace_state(admin_id))
assert state_after_first == state_after_second, "second run must be a no-op"
assert len(state_after_second["workspaces"]) == 1
def test_ensure_admin_user_skips_when_admin_already_has_workspace():
"""An admin with a workspace already set should not get a second one."""
from app.gateway.app import _ensure_admin_user
from app.gateway.deps import get_local_provider
admin_id = asyncio.run(_seed_pre_pr4_admin())
async def _set_default(workspace_id: str):
provider = get_local_provider()
user = await provider.get_user(admin_id)
user.default_workspace_id = workspace_id
await provider.update_user(user)
# Run the lifespan hook once to seed a workspace, then re-run.
asyncio.run(_ensure_admin_user(FastAPI()))
state_seeded = asyncio.run(_read_workspace_state(admin_id))
assert len(state_seeded["workspaces"]) == 1
seeded_ws_id = state_seeded["workspaces"][0][0]
asyncio.run(_set_default(seeded_ws_id)) # ensure default is still pointing at it
asyncio.run(_ensure_admin_user(FastAPI()))
final = asyncio.run(_read_workspace_state(admin_id))
assert final["default_workspace_id"] == seeded_ws_id
assert [w[0] for w in final["workspaces"]] == [seeded_ws_id]
+134
View File
@@ -0,0 +1,134 @@
"""Schema tests for ``ExternalUserRow`` (Stage 0 PR8).
PR8 is schema-only no repository class, no API. Tests exercise raw
ORM behaviour: the composite UNIQUE constraint (service_account_id,
external_id) and CASCADE on service_account delete.
An external user is the end-user identity passed through by a
service_account whose ``identity_mode`` is ``external_passthrough`` or
``both``: every call carries an ``X-External-User-Id`` header which is
upserted into this table for audit / quota attribution.
"""
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from sqlalchemy import delete
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.external_user import ExternalUserRow
from deerflow.persistence.service_account import ServiceAccountRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def _setup(tmp_path):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
return get_session_factory()
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_parents(sf) -> None:
now = datetime.now(UTC)
async with sf() as session:
session.add(UserRow(id="u-alice", email="alice@example.com"))
await session.commit()
async with sf() as session:
session.add(WorkspaceRow(id="w-1", name="Alice WS", slug="alice", owner_id="u-alice"))
await session.commit()
async with sf() as session:
session.add(
ServiceAccountRow(
id="sa-1",
workspace_id="w-1",
name="passthrough-bot",
role="member",
identity_mode="external_passthrough",
status="active",
created_by="u-alice",
created_at=now,
updated_at=now,
)
)
await session.commit()
def _make_external_user(*, eu_id: str, external_id: str) -> ExternalUserRow:
now = datetime.now(UTC)
return ExternalUserRow(
id=eu_id,
workspace_id="w-1",
service_account_id="sa-1",
external_id=external_id,
display_name=None,
metadata_json={},
created_at=now,
last_seen_at=None,
)
# ---------------------------------------------------------------------------
# T8.5-1 — UNIQUE (service_account_id, external_id)
# ---------------------------------------------------------------------------
async def test_unique_service_account_id_plus_external_id(tmp_path):
"""The same external_id may be inserted twice only under different SAs."""
sf = await _setup(tmp_path)
try:
await _seed_parents(sf)
async with sf() as session:
session.add(_make_external_user(eu_id="eu-1", external_id="client-42"))
await session.commit()
with pytest.raises(IntegrityError):
async with sf() as session:
session.add(_make_external_user(eu_id="eu-2", external_id="client-42"))
await session.commit()
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# T8.5-2 — CASCADE on service_account delete
# ---------------------------------------------------------------------------
async def test_cascade_on_service_account_delete(tmp_path):
"""Deleting the parent service_account removes all child external_users rows."""
sf = await _setup(tmp_path)
try:
await _seed_parents(sf)
async with sf() as session:
session.add(_make_external_user(eu_id="eu-c1", external_id="endpoint-A"))
session.add(_make_external_user(eu_id="eu-c2", external_id="endpoint-B"))
await session.commit()
async with sf() as session:
await session.execute(delete(ServiceAccountRow).where(ServiceAccountRow.id == "sa-1"))
await session.commit()
async with sf() as session:
row1 = await session.get(ExternalUserRow, "eu-c1")
row2 = await session.get(ExternalUserRow, "eu-c2")
assert row1 is None
assert row2 is None
finally:
await _cleanup()
+16 -8
View File
@@ -74,7 +74,7 @@ def test_expired_jwt_raises_401():
def test_user_not_found_raises_401():
token = create_access_token("ghost")
token = create_access_token("ghost", workspace_id="ws-test", role="owner")
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(None)):
with pytest.raises(Auth.exceptions.HTTPException) as exc:
asyncio.run(authenticate(_req({"access_token": token})))
@@ -84,7 +84,7 @@ def test_user_not_found_raises_401():
def test_token_version_mismatch_raises_401():
user = _user(token_version=2)
token = create_access_token(str(user.id), token_version=1)
token = create_access_token(str(user.id), token_version=1, workspace_id="ws-test", role="owner")
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(user)):
with pytest.raises(Auth.exceptions.HTTPException) as exc:
asyncio.run(authenticate(_req({"access_token": token})))
@@ -94,7 +94,7 @@ def test_token_version_mismatch_raises_401():
def test_valid_token_returns_user_id():
user = _user(token_version=0)
token = create_access_token(str(user.id), token_version=0)
token = create_access_token(str(user.id), token_version=0, workspace_id="ws-test", role="owner")
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(user)):
result = asyncio.run(authenticate(_req({"access_token": token})))
assert result == str(user.id)
@@ -102,7 +102,7 @@ def test_valid_token_returns_user_id():
def test_valid_token_matching_version():
user = _user(token_version=5)
token = create_access_token(str(user.id), token_version=5)
token = create_access_token(str(user.id), token_version=5, workspace_id="ws-test", role="owner")
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(user)):
result = asyncio.run(authenticate(_req({"access_token": token})))
assert result == str(user.id)
@@ -113,7 +113,7 @@ def test_valid_token_matching_version():
def test_provider_exception_propagates():
"""Provider raises → should not be swallowed silently."""
token = create_access_token("user-1")
token = create_access_token("user-1", workspace_id="ws-test", role="owner")
p = AsyncMock()
p.get_user = AsyncMock(side_effect=RuntimeError("DB down"))
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=p):
@@ -126,7 +126,11 @@ def test_jwt_missing_ver_defaults_to_zero():
import jwt as pyjwt
uid = str(uuid4())
raw = pyjwt.encode({"sub": uid, "exp": 9999999999, "iat": 1000000000}, _JWT_SECRET, algorithm="HS256")
raw = pyjwt.encode(
{"sub": uid, "wid": "ws-test", "role": "owner", "exp": 9999999999, "iat": 1000000000},
_JWT_SECRET,
algorithm="HS256",
)
user = _user(user_id=uid, token_version=0)
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(user)):
result = asyncio.run(authenticate(_req({"access_token": raw})))
@@ -138,7 +142,11 @@ def test_jwt_missing_ver_rejected_when_user_version_nonzero():
import jwt as pyjwt
uid = str(uuid4())
raw = pyjwt.encode({"sub": uid, "exp": 9999999999, "iat": 1000000000}, _JWT_SECRET, algorithm="HS256")
raw = pyjwt.encode(
{"sub": uid, "wid": "ws-test", "role": "owner", "exp": 9999999999, "iat": 1000000000},
_JWT_SECRET,
algorithm="HS256",
)
user = _user(user_id=uid, token_version=1)
with patch("app.gateway.langgraph_auth.get_local_provider", return_value=_mock_provider(user)):
with pytest.raises(Auth.exceptions.HTTPException) as exc:
@@ -221,7 +229,7 @@ def test_filter_with_empty_metadata():
def test_shared_jwt_secret():
token = create_access_token("user-1", token_version=3)
token = create_access_token("user-1", token_version=3, workspace_id="ws-test", role="owner")
payload = decode_token(token)
from app.gateway.auth.errors import TokenError
+60
View File
@@ -0,0 +1,60 @@
"""Legacy 4-field JWT compatibility (Stage 0 PR4 T4.6).
Before PR4 every JWT carried only `{sub, exp, iat, ver}`. After PR4 the
server expects `wid` (workspace_id) on every protected request. Old
cookies in the wild must NOT collapse into ``TokenError.MALFORMED``
that hides the actual problem (workspace required) and prevents the
frontend from steering the user to ``/select-workspace``.
The contract: ``decode_token`` returns ``TokenError.WORKSPACE_MISSING``
specifically when the JWT signature checks out and the payload is
otherwise well-formed but does NOT carry a ``wid`` claim.
"""
from __future__ import annotations
from datetime import UTC, datetime, timedelta
import jwt
import pytest
from app.gateway.auth.config import get_auth_config
from app.gateway.auth.errors import TokenError
from app.gateway.auth.jwt import decode_token
@pytest.fixture(autouse=True)
def _stable_jwt_secret(monkeypatch):
monkeypatch.setenv("AUTH_JWT_SECRET", "test-secret-key-for-jwt-testing-minimum-32-chars")
yield
def _make_legacy_token(*, expired: bool = False) -> str:
"""Encode a pre-PR4 JWT directly (bypassing create_access_token)."""
now = datetime.now(UTC)
payload = {
"sub": "u-legacy",
"exp": now + (timedelta(seconds=-1) if expired else timedelta(hours=1)),
"iat": now,
"ver": 0,
}
return jwt.encode(payload, get_auth_config().jwt_secret, algorithm="HS256")
def test_decode_legacy_token_returns_workspace_missing_error() -> None:
"""4-field token (no wid) → TokenError.WORKSPACE_MISSING (not MALFORMED)."""
token = _make_legacy_token()
result = decode_token(token)
assert result == TokenError.WORKSPACE_MISSING
def test_decode_legacy_token_with_expired_still_reports_expired() -> None:
"""Expired legacy tokens keep reporting EXPIRED — that signal takes priority.
Why: an expired token must trigger /auth/refresh logic before we
decide it also lacks workspace; reporting WORKSPACE_MISSING on an
expired token would steer the user to /select-workspace instead.
"""
token = _make_legacy_token(expired=True)
result = decode_token(token)
assert result == TokenError.EXPIRED
@@ -0,0 +1,155 @@
"""PR6 T6.13 — workspace-path migration script tests.
Builds the legacy ``users/{uid}/...`` tree under a temp base dir,
points a populated sqlite ``users`` table at it via the conventional
``deer-flow.db`` location, and asserts the migration produces the new
``workspaces/{wid}/...`` layout. Covers:
- threads / memory.json / custom agents all rewritten under the workspace
- ``--dry-run`` writes nothing
- pre-existing destinations get diverted to ``migration-conflicts/``
- users without a ``default_workspace_id`` fall back to the explicit flag
- empty legacy dirs are cleaned up
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
import pytest
from deerflow.config.paths import Paths
from scripts.migrate_paths_to_workspace import (
LEGACY_WORKSPACE_FALLBACK,
_load_user_workspaces,
migrate,
)
def _build_legacy_tree(base: Path, *, user_id: str, thread_ids: tuple[str, ...] = (), with_memory: bool = False, agent_names: tuple[str, ...] = ()) -> None:
user_root = base / "users" / user_id
user_root.mkdir(parents=True, exist_ok=True)
for tid in thread_ids:
(user_root / "threads" / tid / "user-data" / "workspace").mkdir(parents=True, exist_ok=True)
(user_root / "threads" / tid / "user-data" / "workspace" / "marker.txt").write_text(f"{user_id}/{tid}", encoding="utf-8")
if with_memory:
(user_root / "memory.json").write_text(f'{{"user_id": "{user_id}"}}', encoding="utf-8")
for name in agent_names:
(user_root / "agents" / name).mkdir(parents=True, exist_ok=True)
(user_root / "agents" / name / "SOUL.md").write_text(f"# {name}", encoding="utf-8")
def _seed_db(base: Path, *, users: dict[str, str | None]) -> None:
db_path = base / "deer-flow.db"
conn = sqlite3.connect(str(db_path))
try:
conn.execute("CREATE TABLE users (id TEXT PRIMARY KEY, default_workspace_id TEXT)")
conn.executemany("INSERT INTO users (id, default_workspace_id) VALUES (?, ?)", list(users.items()))
conn.commit()
finally:
conn.close()
@pytest.fixture
def base(tmp_path: Path) -> Path:
return tmp_path
def test_migrate_threads_under_workspace(base: Path):
_build_legacy_tree(base, user_id="alice", thread_ids=("t1",))
_seed_db(base, users={"alice": "ws-alpha"})
paths = Paths(base)
report = migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=False)
assert (base / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "workspace" / "marker.txt").read_text(encoding="utf-8") == "alice/t1"
assert not (base / "users" / "alice" / "threads").exists()
assert {entry["asset"] for entry in report} == {"thread"}
def test_migrate_memory_and_agents_nested_under_workspace_and_user(base: Path):
_build_legacy_tree(base, user_id="alice", with_memory=True, agent_names=("code-reviewer",))
_seed_db(base, users={"alice": "ws-alpha"})
paths = Paths(base)
migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=False)
assert (base / "workspaces" / "ws-alpha" / "users" / "alice" / "memory.json").exists()
assert (base / "workspaces" / "ws-alpha" / "users" / "alice" / "agents" / "code-reviewer" / "SOUL.md").exists()
def test_dry_run_writes_nothing(base: Path):
_build_legacy_tree(base, user_id="alice", thread_ids=("t1",), with_memory=True, agent_names=("a1",))
_seed_db(base, users={"alice": "ws-alpha"})
paths = Paths(base)
report = migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=True)
# Source unchanged
assert (base / "users" / "alice" / "threads" / "t1" / "user-data" / "workspace" / "marker.txt").exists()
assert (base / "users" / "alice" / "memory.json").exists()
assert (base / "users" / "alice" / "agents" / "a1" / "SOUL.md").exists()
# No destination created
assert not (base / "workspaces").exists()
# Report still populated so operator sees what *would* happen
assert len(report) == 3
def test_fallback_workspace_used_when_user_has_no_default(base: Path):
_build_legacy_tree(base, user_id="alice", thread_ids=("t1",))
_seed_db(base, users={"alice": None})
paths = Paths(base)
migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace="legacy_workspace", dry_run=False)
assert (base / "workspaces" / "legacy_workspace" / "threads" / "t1" / "user-data" / "workspace" / "marker.txt").exists()
def test_conflict_routes_legacy_to_migration_conflicts(base: Path):
# Pre-create the destination with a different marker so the move sees a conflict.
_build_legacy_tree(base, user_id="alice", thread_ids=("t1",))
(base / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "workspace").mkdir(parents=True)
(base / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "workspace" / "marker.txt").write_text("preexisting", encoding="utf-8")
_seed_db(base, users={"alice": "ws-alpha"})
paths = Paths(base)
report = migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=False)
assert (base / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "workspace" / "marker.txt").read_text(encoding="utf-8") == "preexisting"
conflict_marker = base / "migration-conflicts" / "workspace-migration" / "threads/ws-alpha" / "t1"
assert conflict_marker.exists()
assert any("conflict" in entry["action"] for entry in report)
def test_empty_users_dir_removed_after_full_migration(base: Path):
_build_legacy_tree(base, user_id="alice", thread_ids=("t1",))
_seed_db(base, users={"alice": "ws-alpha"})
paths = Paths(base)
migrate(paths, user_workspaces=_load_user_workspaces(paths), fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=False)
assert not (base / "users").exists(), "Empty legacy users/ dir should be cleaned up"
def test_no_users_directory_is_noop(base: Path):
paths = Paths(base)
report = migrate(paths, user_workspaces={}, fallback_workspace=LEGACY_WORKSPACE_FALLBACK, dry_run=False)
assert report == []
def test_missing_db_returns_empty_mapping(base: Path):
paths = Paths(base)
assert _load_user_workspaces(paths) == {}
def test_db_without_users_table_returns_empty(base: Path):
db_path = base / "deer-flow.db"
conn = sqlite3.connect(str(db_path))
try:
conn.execute("CREATE TABLE other_table (x INTEGER)")
conn.commit()
finally:
conn.close()
paths = Paths(base)
assert _load_user_workspaces(paths) == {}
@@ -0,0 +1,45 @@
"""PR6 T6.14 — lifespan warns when legacy users/ tree still has content."""
from __future__ import annotations
import logging
from pathlib import Path
from unittest.mock import patch
import pytest
from app.gateway.app import _check_path_migration_pending
class _FakePaths:
def __init__(self, base: Path):
self.base_dir = base
@pytest.fixture
def base(tmp_path: Path) -> Path:
return tmp_path
def _patch_paths(base: Path):
return patch("deerflow.config.paths.get_paths", return_value=_FakePaths(base))
def test_warns_when_legacy_users_dir_has_content(base: Path, caplog: pytest.LogCaptureFixture):
(base / "users" / "alice" / "threads" / "t1").mkdir(parents=True)
with _patch_paths(base), caplog.at_level(logging.WARNING, logger="app.gateway.app"):
_check_path_migration_pending(app=None) # type: ignore[arg-type]
assert any("make migrate-paths" in rec.message for rec in caplog.records)
def test_silent_when_legacy_users_dir_missing(base: Path, caplog: pytest.LogCaptureFixture):
with _patch_paths(base), caplog.at_level(logging.WARNING, logger="app.gateway.app"):
_check_path_migration_pending(app=None) # type: ignore[arg-type]
assert not any("migrate-paths" in rec.message for rec in caplog.records)
def test_silent_when_legacy_users_dir_empty(base: Path, caplog: pytest.LogCaptureFixture):
(base / "users").mkdir()
with _patch_paths(base), caplog.at_level(logging.WARNING, logger="app.gateway.app"):
_check_path_migration_pending(app=None) # type: ignore[arg-type]
assert not any("migrate-paths" in rec.message for rec in caplog.records)
+115
View File
@@ -0,0 +1,115 @@
"""PR6 T6.9 / T6.10 — workspace-scoped path resolution.
The Paths class learns a new top-level dimension for multi-tenant
filesystems: ``{base_dir}/workspaces/{wid}/...``. Precedence:
- ``workspace_id`` given new shape ``workspaces/{wid}/threads/{tid}/...``
- only ``user_id`` given legacy shape ``users/{uid}/threads/{tid}/...``
- neither very-legacy shape ``threads/{tid}/...``
Per-user filesystem state (memory.json, custom agents) lives under the
workspace too: ``workspaces/{wid}/users/{uid}/memory.json`` etc. so a
user's memory cannot be reused across workspaces by mistake.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from deerflow.config.paths import Paths
@pytest.fixture
def paths(tmp_path: Path) -> Paths:
return Paths(tmp_path)
class TestValidateWorkspaceId:
def test_valid_workspace_id(self, paths: Paths):
d = paths.workspace_dir("ws-abc-123")
assert d == paths.base_dir / "workspaces" / "ws-abc-123"
def test_rejects_path_traversal(self, paths: Paths):
with pytest.raises(ValueError, match="Invalid workspace_id"):
paths.workspace_dir("../escape")
def test_rejects_slash(self, paths: Paths):
with pytest.raises(ValueError, match="Invalid workspace_id"):
paths.workspace_dir("ws/bar")
def test_rejects_empty(self, paths: Paths):
with pytest.raises(ValueError, match="Invalid workspace_id"):
paths.workspace_dir("")
class TestWorkspaceScopedThreadDir:
def test_workspace_takes_precedence_over_user(self, paths: Paths):
"""When both are given, workspace wins — user_id is recorded in the row, not the filesystem."""
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1"
assert paths.thread_dir("t1", workspace_id="ws-alpha", user_id="alice") == expected
def test_workspace_only(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1"
assert paths.thread_dir("t1", workspace_id="ws-alpha") == expected
def test_user_only_still_legacy(self, paths: Paths):
"""Legacy callers keep the user_id-only shape until migration runs."""
expected = paths.base_dir / "users" / "alice" / "threads" / "t1"
assert paths.thread_dir("t1", user_id="alice") == expected
def test_no_ids_very_legacy(self, paths: Paths):
expected = paths.base_dir / "threads" / "t1"
assert paths.thread_dir("t1") == expected
class TestEnsureThreadDirsWorkspace:
def test_creates_workspace_layout(self, paths: Paths):
paths.ensure_thread_dirs("t1", workspace_id="ws-alpha")
root = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1"
for sub in ("user-data/workspace", "user-data/uploads", "user-data/outputs", "acp-workspace"):
assert (root / sub).is_dir(), f"missing {sub}"
class TestSandboxDirsWorkspace:
def test_sandbox_work_dir(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "workspace"
assert paths.sandbox_work_dir("t1", workspace_id="ws-alpha") == expected
def test_sandbox_uploads_dir(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "uploads"
assert paths.sandbox_uploads_dir("t1", workspace_id="ws-alpha") == expected
def test_sandbox_outputs_dir(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "outputs"
assert paths.sandbox_outputs_dir("t1", workspace_id="ws-alpha") == expected
class TestUserMemoryUnderWorkspace:
def test_user_memory_file_under_workspace(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "users" / "alice" / "memory.json"
assert paths.user_memory_file("alice", workspace_id="ws-alpha") == expected
def test_user_memory_file_legacy_without_workspace(self, paths: Paths):
expected = paths.base_dir / "users" / "alice" / "memory.json"
assert paths.user_memory_file("alice") == expected
def test_user_agents_dir_under_workspace(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "users" / "alice" / "agents"
assert paths.user_agents_dir("alice", workspace_id="ws-alpha") == expected
def test_user_agent_memory_file_under_workspace(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "users" / "alice" / "agents" / "code-reviewer" / "memory.json"
assert paths.user_agent_memory_file("alice", "code-reviewer", workspace_id="ws-alpha") == expected
class TestVirtualPathResolutionWorkspace:
def test_resolve_virtual_path_workspace_scope(self, paths: Paths):
expected = paths.base_dir / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data" / "outputs" / "x.json"
actual = paths.resolve_virtual_path("t1", "/mnt/user-data/outputs/x.json", workspace_id="ws-alpha")
assert actual == expected.resolve()
def test_resolve_virtual_path_rejects_traversal_under_workspace(self, paths: Paths):
with pytest.raises(ValueError, match="path traversal"):
paths.resolve_virtual_path("t1", "/mnt/user-data/../../etc/passwd", workspace_id="ws-alpha")
+151
View File
@@ -0,0 +1,151 @@
"""Stage 0 PR1 · Postgres smoke tests via testcontainers.
These tests exercise the postgres_url fixture and verify that DeerFlow's
existing ``init_engine`` + ORM ``Base.metadata.create_all`` works against
Postgres exactly the same way it works against SQLite (no Stage 0
schema changes here that's PR3+).
All tests are gated by ``@pytest.mark.postgres`` and skip cleanly when
Docker is unavailable (the postgres_container fixture handles that).
"""
from __future__ import annotations
import secrets
import pytest
# Mark every test in this module as `postgres` so they only run when
# explicitly requested with ``pytest -m postgres``.
pytestmark = [pytest.mark.postgres, pytest.mark.anyio]
@pytest.fixture
def anyio_backend() -> str:
"""anyio uses asyncio backend (matches DeerFlow's runtime)."""
return "asyncio"
# ---------------------------------------------------------------------------
# T1.4 · Fixture self-test — verify per-test isolation
# ---------------------------------------------------------------------------
async def test_postgres_url_creates_isolated_database(postgres_url: str) -> None:
"""The fixture should yield a usable URL pointing at a unique database."""
import asyncpg
# The URL form is `postgresql+asyncpg://user:pass@host:port/test_<hex>`.
assert postgres_url.startswith("postgresql+asyncpg://")
assert "/test_" in postgres_url
# Connect with raw asyncpg (strip SQLAlchemy dialect prefix) and confirm
# we landed in the named test DB.
raw = postgres_url.replace("postgresql+asyncpg://", "postgresql://")
conn = await asyncpg.connect(raw)
try:
current = await conn.fetchval("SELECT current_database()")
assert current.startswith("test_"), f"expected test_*, got {current!r}"
finally:
await conn.close()
async def test_postgres_url_isolates_between_tests(postgres_container, postgres_url: str) -> None:
"""Two invocations of the fixture should yield two distinct databases.
Hand-rolls a second database via the same recipe to prove isolation
without depending on pytest's own per-test invocation timing.
"""
import asyncpg
import psycopg
from psycopg import sql
# Read what DB we're in.
raw = postgres_url.replace("postgresql+asyncpg://", "postgresql://")
conn = await asyncpg.connect(raw)
try:
db_a = await conn.fetchval("SELECT current_database()")
finally:
await conn.close()
# Create a *second* DB on the same container directly.
container_url = postgres_container.get_connection_url().replace("postgresql+psycopg2://", "postgresql://")
parent_url = container_url.rsplit("/", 1)[0] + "/postgres"
db_b_name = f"test_{secrets.token_hex(8)}"
with psycopg.connect(parent_url, autocommit=True) as c:
c.execute(sql.SQL("CREATE DATABASE {}").format(sql.Identifier(db_b_name)))
try:
assert db_a != db_b_name, "fixture should not reuse a DB name across tests"
finally:
with psycopg.connect(parent_url, autocommit=True) as c:
c.execute(sql.SQL("DROP DATABASE IF EXISTS {}").format(sql.Identifier(db_b_name)))
# ---------------------------------------------------------------------------
# T1.5 · init_engine smoke — Base.metadata.create_all() works on Postgres
# ---------------------------------------------------------------------------
async def test_init_engine_postgres_creates_tables(postgres_url: str) -> None:
"""``init_engine`` against Postgres must auto-create existing ORM tables.
Verifies the four current business tables (users, threads_meta, runs,
feedback) plus run_events appear in information_schema after init
proving that DeerFlow's ``Base.metadata.create_all()`` path works
identically on PG and SQLite.
"""
from sqlalchemy import text
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
await init_engine("postgres", url=postgres_url)
try:
sf = get_session_factory()
async with sf() as session:
result = await session.execute(text("SELECT table_name FROM information_schema.tables WHERE table_schema = 'public'"))
tables = {row[0] for row in result.all()}
# Existing tables before any Stage 0 schema additions:
expected = {"users", "threads_meta", "runs", "feedback", "run_events"}
missing = expected - tables
assert not missing, f"create_all() did not produce {missing}; got {tables}"
finally:
await close_engine()
# ---------------------------------------------------------------------------
# T1.6 · Repository round-trip on Postgres
# ---------------------------------------------------------------------------
async def test_thread_meta_repo_postgres_round_trip(postgres_url: str) -> None:
"""ThreadMetaRepository.create + get must behave identically on PG.
Pin: this is a regression net for any SQLAlchemy / asyncpg surprise
that diverges from SQLite behavior in dict shape, default values,
or timestamp precision.
"""
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
from deerflow.persistence.thread_meta import ThreadMetaRepository
await init_engine("postgres", url=postgres_url)
try:
repo = ThreadMetaRepository(get_session_factory())
# Note: conftest's `_auto_user_context` autouse fixture has already
# injected `id="test-user-autouse"` so create() picks it via AUTO.
created = await repo.create(
thread_id="thread-1",
display_name="hello",
metadata={"k": "v"},
)
assert created["thread_id"] == "thread-1"
assert created["user_id"] == "test-user-autouse"
assert created["display_name"] == "hello"
assert created["metadata"] == {"k": "v"}
fetched = await repo.get("thread-1")
assert fetched is not None
assert fetched["thread_id"] == "thread-1"
assert fetched["user_id"] == "test-user-autouse"
finally:
await close_engine()
@@ -0,0 +1,42 @@
"""Acceptance test for PR8: ``Base.metadata.create_all()`` automatically
provisions ``service_accounts`` / ``api_keys`` / ``external_users``.
The harness layer registers all ORM models through ``deerflow.persistence.models``
(imported for side effects from ``engine.init_engine``). This test guards
against a row class being defined but accidentally left out of the
registration entry point a class table that never gets created at
``init_engine`` time would otherwise silently break Stage 1 once the
API-key auth layer starts inserting rows.
"""
from __future__ import annotations
import pytest
from sqlalchemy import inspect
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def test_pr8_tables_present_after_init_engine(tmp_path):
from deerflow.persistence.engine import close_engine, get_engine, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
try:
engine = get_engine()
assert engine is not None
def _table_names(sync_conn):
return set(inspect(sync_conn).get_table_names())
async with engine.connect() as conn:
tables = await conn.run_sync(_table_names)
assert {"service_accounts", "api_keys", "external_users"}.issubset(tables), f"PR8 tables missing from create_all: have {sorted(tables)}"
finally:
await close_engine()
@@ -0,0 +1,172 @@
"""Registration endpoints (POST /initialize, POST /register) auto-create a workspace.
Stage 0 PR4 T4.8 + T4.9. Every newly registered user must end up with:
- a single ``workspaces`` row (their personal workspace),
- a single ``workspace_memberships`` row with ``role='owner'``,
- ``users.default_workspace_id`` pointing at that workspace,
- a session cookie whose JWT carries the workspace as the ``wid`` claim.
Tests run against a per-test SQLite engine bootstrapped by the
fixture; the registration router is exercised through the real
TestClient so the full handler + DB transaction path is covered.
"""
from __future__ import annotations
import asyncio
import os
import jwt
import pytest
from fastapi.testclient import TestClient
os.environ.setdefault("AUTH_JWT_SECRET", "test-secret-key-register-workspace-32+")
from app.gateway.auth.config import AuthConfig, set_auth_config
_TEST_SECRET = "test-secret-key-register-workspace-32+"
@pytest.fixture(autouse=True)
def _setup_auth(tmp_path):
from app.gateway import deps
from app.gateway.routers.auth import _SETUP_STATUS_COOLDOWN
from deerflow.persistence.engine import close_engine, init_engine
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
url = f"sqlite+aiosqlite:///{tmp_path}/register_ws.db"
asyncio.run(init_engine("sqlite", url=url, sqlite_dir=str(tmp_path)))
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
try:
yield
finally:
deps._cached_local_provider = None
deps._cached_repo = None
_SETUP_STATUS_COOLDOWN.clear()
asyncio.run(close_engine())
@pytest.fixture()
def client(_setup_auth):
from app.gateway.app import create_app
set_auth_config(AuthConfig(jwt_secret=_TEST_SECRET))
app = create_app()
yield TestClient(app)
def _init_payload(**extra):
return {"email": "admin@example.com", "password": "Str0ng!Pass99", **extra}
def _register_payload(email: str = "alice@example.com", **extra):
return {"email": email, "password": "Tr0ub4dor3a-strong!", **extra}
def _decode(token: str) -> dict:
"""Decode a JWT (signature-checked) and return the raw payload."""
return jwt.decode(token, _TEST_SECRET, algorithms=["HS256"])
async def _read_workspace_state(user_id: str) -> dict:
"""Inspect the per-user workspace state after a registration call.
Returns a dict with the workspace row, the owner membership row and
the user's default_workspace_id, so each test can pick what it
cares about.
"""
from sqlalchemy import select
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
sf = get_session_factory()
async with sf() as session:
user = await session.get(UserRow, user_id)
memberships = (await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.user_id == user_id))).scalars().all()
workspaces = []
if memberships:
workspaces = (await session.execute(select(WorkspaceRow).where(WorkspaceRow.id.in_([m.workspace_id for m in memberships])))).scalars().all()
return {
"user_default_workspace_id": getattr(user, "default_workspace_id", None) if user else None,
"memberships": [(m.workspace_id, m.user_id, m.role) for m in memberships],
"workspaces": [(w.id, w.slug, w.owner_id) for w in workspaces],
}
# ---------- T4.8 — /initialize ---------------------------------------------
def test_initialize_creates_admin_with_default_workspace(client):
"""POST /initialize → admin + workspace + owner membership + wid cookie."""
resp = client.post("/api/v1/auth/initialize", json=_init_payload())
assert resp.status_code == 201, resp.text
user_id = resp.json()["id"]
state = asyncio.run(_read_workspace_state(user_id))
assert len(state["workspaces"]) == 1, state
ws_id, ws_slug, ws_owner = state["workspaces"][0]
assert ws_owner == user_id
# base slug "admin" is reserved, walker bumps to first free suffix
assert ws_slug == "admin-2"
assert state["memberships"] == [(ws_id, user_id, "owner")]
assert state["user_default_workspace_id"] == ws_id
token = resp.cookies["access_token"]
claims = _decode(token)
assert claims["wid"] == ws_id
assert claims["role"] == "owner"
# ---------- T4.9 — /register -----------------------------------------------
def test_register_creates_user_with_default_workspace(client):
"""POST /register → user + workspace + owner membership + wid cookie."""
# Initialize an admin first so the system is past first-boot.
client.post("/api/v1/auth/initialize", json=_init_payload())
resp = client.post("/api/v1/auth/register", json=_register_payload())
assert resp.status_code == 201, resp.text
user_id = resp.json()["id"]
state = asyncio.run(_read_workspace_state(user_id))
assert len(state["workspaces"]) == 1, state
ws_id, ws_slug, ws_owner = state["workspaces"][0]
assert ws_owner == user_id
assert ws_slug == "alice"
assert state["memberships"] == [(ws_id, user_id, "owner")]
assert state["user_default_workspace_id"] == ws_id
token = resp.cookies["access_token"]
claims = _decode(token)
assert claims["wid"] == ws_id
assert claims["role"] == "owner"
def test_two_registrations_isolate_workspaces_and_avoid_slug_collision(client):
"""Two users with colliding email local-parts → distinct workspaces, slug suffix bump."""
client.post("/api/v1/auth/initialize", json=_init_payload())
r1 = client.post("/api/v1/auth/register", json=_register_payload(email="alice@example.com"))
r2 = client.post("/api/v1/auth/register", json=_register_payload(email="alice@somewhere.else"))
assert r1.status_code == 201, r1.text
assert r2.status_code == 201, r2.text
s1 = asyncio.run(_read_workspace_state(r1.json()["id"]))
s2 = asyncio.run(_read_workspace_state(r2.json()["id"]))
ws1 = s1["workspaces"][0]
ws2 = s2["workspaces"][0]
assert ws1[0] != ws2[0], "workspaces must be distinct"
assert ws1[1] == "alice"
assert ws2[1] == "alice-2", "slug collision walker should land on -2"
@@ -0,0 +1,136 @@
"""PR6 T6.6 — `@require_permission(owner_check=True)` workspace upgrade.
The decorator must:
1. Pull workspace_id from ``get_effective_workspace_id()`` (set by
AuthMiddleware per request) and pass it as the third positional to
``ThreadMetaStore.check_access``.
2. Raise **HTTPException 404** when check_access returns False never
403 so a cross-workspace request cannot distinguish "thread exists
in another tenant" from "thread does not exist".
These tests build a fake router with the same decorator usage as the
production code and verify the decorator's behaviour via a Mock
``thread_store`` whose ``check_access`` call we inspect, plus a TestClient
exercising the full HTTP boundary.
"""
from __future__ import annotations
from collections.abc import Callable
import pytest
from _router_auth_helpers import make_authed_test_app
from fastapi import APIRouter, Request
from fastapi.testclient import TestClient
from app.gateway.auth.models import ActiveWorkspace
from app.gateway.authz import require_permission
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
def _make_workspace(wid: str) -> Callable[[], ActiveWorkspace]:
"""Factory closure stable per call so the middleware reinjects the same id."""
def _factory() -> ActiveWorkspace:
return ActiveWorkspace(id=wid, role="owner")
return _factory
def _mount_routes(app):
router = APIRouter()
@router.delete("/probe/{thread_id}")
@require_permission("threads", "delete", owner_check=True, require_existing=True)
async def _delete_probe(thread_id: str, request: Request): # noqa: ARG001
return {"ok": True, "thread_id": thread_id}
@router.get("/probe/{thread_id}")
@require_permission("threads", "read", owner_check=True)
async def _get_probe(thread_id: str, request: Request): # noqa: ARG001
return {"ok": True, "thread_id": thread_id}
app.include_router(router)
return app
def test_cross_workspace_returns_404():
"""check_access returning False surfaces as 404, never 403."""
app = make_authed_test_app(
workspace_factory=_make_workspace("ws-alpha"),
owner_check_passes=False,
)
_mount_routes(app)
with TestClient(app) as client:
response = client.delete("/probe/t1")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_same_workspace_delete_allowed():
"""check_access returning True lets the route execute."""
app = make_authed_test_app(
workspace_factory=_make_workspace("ws-alpha"),
owner_check_passes=True,
)
_mount_routes(app)
with TestClient(app) as client:
response = client.delete("/probe/t1")
assert response.status_code == 200
assert response.json()["ok"] is True
def test_workspace_id_passed_to_check_access():
"""check_access receives the contextvar workspace_id as the 3rd positional."""
app = make_authed_test_app(
workspace_factory=_make_workspace("ws-alpha"),
owner_check_passes=True,
)
_mount_routes(app)
with TestClient(app) as client:
client.delete("/probe/t1")
call = app.state.thread_store.check_access.call_args
assert call is not None
args = call.args
# (thread_id, user_id, workspace_id)
assert args[0] == "t1"
assert args[2] == "ws-alpha"
@pytest.mark.no_auto_workspace
def test_no_workspace_in_context_falls_back_to_default():
"""No-auth dev mode (no workspace contextvar) uses DEFAULT_WORKSPACE_ID."""
app = make_authed_test_app(workspace_factory=None, owner_check_passes=True)
_mount_routes(app)
with TestClient(app) as client:
client.delete("/probe/t1")
args = app.state.thread_store.check_access.call_args.args
assert args[2] == "default"
def test_get_route_also_uses_workspace_id():
"""Read-style routes (require_existing=False) also pass workspace_id through."""
app = make_authed_test_app(
workspace_factory=_make_workspace("ws-beta"),
owner_check_passes=True,
)
_mount_routes(app)
with TestClient(app) as client:
client.get("/probe/t-read")
args = app.state.thread_store.check_access.call_args.args
assert args[2] == "ws-beta"
@pytest.fixture
def _reset_ws():
"""Helper for direct-call paths that mutate the contextvar."""
tokens: list = []
yield lambda wid: tokens.append(set_current_workspace(ActiveWorkspace(id=wid, role="owner")))
for token in reversed(tokens):
reset_current_workspace(token)
@@ -0,0 +1,181 @@
"""Tests for Run/Feedback/RunEvent repository workspace_id filtering (PR6 T6.5)."""
from __future__ import annotations
from datetime import UTC, datetime
from types import SimpleNamespace
import pytest
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
async def _init_engine(tmp_path, *, workspaces: tuple[str, ...] = ()):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
for wid in workspaces:
await _seed_workspace(wid)
return get_session_factory()
async def _seed_workspace(wid: str) -> None:
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.workspace.model import WorkspaceRow
factory = get_session_factory()
async with factory() as session:
if await session.get(WorkspaceRow, wid) is not None:
return
now = datetime.now(UTC)
session.add(
WorkspaceRow(
id=wid,
name=f"WS {wid}",
slug=wid.replace("_", "-")[:32],
status="active",
owner_id="test-user-autouse",
created_at=now,
updated_at=now,
)
)
await session.commit()
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
def _use_workspace(wid: str):
return set_current_workspace(SimpleNamespace(id=wid, role="owner"))
class TestRunRepositoryWorkspace:
@pytest.mark.anyio
async def test_put_records_workspace_id(self, tmp_path):
from deerflow.persistence.run import RunRepository
sf = await _init_engine(tmp_path, workspaces=("ws-alpha",))
repo = RunRepository(sf)
token = _use_workspace("ws-alpha")
try:
await repo.put("r1", thread_id="t1", user_id="alice")
record = await repo.get("r1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert record["workspace_id"] == "ws-alpha"
@pytest.mark.anyio
async def test_get_filters_cross_workspace(self, tmp_path):
from deerflow.persistence.run import RunRepository
sf = await _init_engine(tmp_path, workspaces=("ws-alpha", "ws-beta"))
repo = RunRepository(sf)
token = _use_workspace("ws-alpha")
try:
await repo.put("r1", thread_id="t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
assert await repo.get("r1", user_id="alice") is None
finally:
reset_current_workspace(token)
await _cleanup()
@pytest.mark.anyio
async def test_list_by_thread_filters_workspace(self, tmp_path):
from deerflow.persistence.run import RunRepository
sf = await _init_engine(tmp_path, workspaces=("ws-alpha", "ws-beta"))
repo = RunRepository(sf)
token = _use_workspace("ws-alpha")
try:
await repo.put("r1", thread_id="t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.put("r2", thread_id="t1", user_id="alice")
rows = await repo.list_by_thread("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert [r["run_id"] for r in rows] == ["r2"]
class TestFeedbackRepositoryWorkspace:
@pytest.mark.anyio
async def test_create_records_workspace_id(self, tmp_path):
from deerflow.persistence.feedback.sql import FeedbackRepository
sf = await _init_engine(tmp_path, workspaces=("ws-alpha",))
repo = FeedbackRepository(sf)
token = _use_workspace("ws-alpha")
try:
row = await repo.create(run_id="r1", thread_id="t1", rating=1, user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert row["workspace_id"] == "ws-alpha"
@pytest.mark.anyio
async def test_list_by_thread_filters_workspace(self, tmp_path):
from deerflow.persistence.feedback.sql import FeedbackRepository
sf = await _init_engine(tmp_path, workspaces=("ws-alpha", "ws-beta"))
repo = FeedbackRepository(sf)
token = _use_workspace("ws-alpha")
try:
await repo.create(run_id="r1", thread_id="t1", rating=1, user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
rows = await repo.list_by_thread("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert rows == []
class TestRunEventStoreWorkspace:
@pytest.mark.anyio
async def test_put_records_workspace_id(self, tmp_path):
from deerflow.runtime.events.store.db import DbRunEventStore
sf = await _init_engine(tmp_path, workspaces=("ws-alpha",))
store = DbRunEventStore(sf)
token = _use_workspace("ws-alpha")
try:
row = await store.put(thread_id="t1", run_id="r1", event_type="msg", category="message", content="hi")
finally:
reset_current_workspace(token)
await _cleanup()
assert row["workspace_id"] == "ws-alpha"
@pytest.mark.anyio
async def test_list_messages_filters_cross_workspace(self, tmp_path):
from deerflow.runtime.events.store.db import DbRunEventStore
sf = await _init_engine(tmp_path, workspaces=("ws-alpha", "ws-beta"))
store = DbRunEventStore(sf)
token = _use_workspace("ws-alpha")
try:
await store.put(thread_id="t1", run_id="r1", event_type="msg", category="message", content="from-alpha")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
rows = await store.list_messages("t1", user_id="test-user-autouse")
finally:
reset_current_workspace(token)
await _cleanup()
assert rows == []
@@ -0,0 +1,176 @@
"""Schema tests for ``ServiceAccountRow`` (Stage 0 PR8).
Pattern mirrors :mod:`test_workspace_repo`: ephemeral SQLite per test via
``tmp_path``, no Postgres required at this layer.
PR8 is schema-only no repository class, no API. Tests exercise raw
ORM behaviour: insert smoke, CASCADE on workspace delete, and RESTRICT
on the ``created_by`` user FK.
"""
from __future__ import annotations
from datetime import UTC, datetime
import pytest
from sqlalchemy import delete
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.service_account import ServiceAccountRow
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace.model import WorkspaceRow
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def _setup(tmp_path):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
return get_session_factory()
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_user(sf, user_id: str = "u-alice", email: str = "alice@example.com") -> None:
async with sf() as session:
session.add(UserRow(id=user_id, email=email))
await session.commit()
async def _seed_workspace(sf, workspace_id: str = "w-1", owner_id: str = "u-alice", slug: str = "alice") -> None:
async with sf() as session:
session.add(WorkspaceRow(id=workspace_id, name="Alice's WS", slug=slug, owner_id=owner_id))
await session.commit()
# ---------------------------------------------------------------------------
# T8.1 — insert smoke
# ---------------------------------------------------------------------------
async def test_insert_smoke(tmp_path):
"""A minimal service_account row can be inserted and read back."""
sf = await _setup(tmp_path)
try:
await _seed_user(sf)
await _seed_workspace(sf)
now = datetime.now(UTC)
async with sf() as session:
session.add(
ServiceAccountRow(
id="sa-1",
workspace_id="w-1",
name="ci-bot",
role="member",
identity_mode="collapsed",
status="active",
created_by="u-alice",
created_at=now,
updated_at=now,
)
)
await session.commit()
async with sf() as session:
row = await session.get(ServiceAccountRow, "sa-1")
assert row is not None
assert row.workspace_id == "w-1"
assert row.name == "ci-bot"
assert row.role == "member"
assert row.identity_mode == "collapsed"
assert row.status == "active"
assert row.created_by == "u-alice"
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# T8.2 — CASCADE on workspace delete
# ---------------------------------------------------------------------------
async def test_cascade_on_workspace_delete(tmp_path):
"""Deleting the parent workspace removes the service_account row (FK CASCADE)."""
sf = await _setup(tmp_path)
try:
await _seed_user(sf)
await _seed_workspace(sf)
now = datetime.now(UTC)
async with sf() as session:
session.add(
ServiceAccountRow(
id="sa-2",
workspace_id="w-1",
name="bot",
role="member",
identity_mode="collapsed",
status="active",
created_by="u-alice",
created_at=now,
updated_at=now,
)
)
await session.commit()
async with sf() as session:
await session.execute(delete(WorkspaceRow).where(WorkspaceRow.id == "w-1"))
await session.commit()
async with sf() as session:
row = await session.get(ServiceAccountRow, "sa-2")
assert row is None
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# T8.3 — RESTRICT on created_by user delete
# ---------------------------------------------------------------------------
async def test_restrict_on_created_by_user_delete(tmp_path):
"""Deleting the creator user is blocked while their service_account survives."""
sf = await _setup(tmp_path)
try:
await _seed_user(sf)
await _seed_workspace(sf)
now = datetime.now(UTC)
async with sf() as session:
session.add(
ServiceAccountRow(
id="sa-3",
workspace_id="w-1",
name="bot",
role="member",
identity_mode="collapsed",
status="active",
created_by="u-alice",
created_at=now,
updated_at=now,
)
)
await session.commit()
with pytest.raises(IntegrityError):
async with sf() as session:
await session.execute(delete(UserRow).where(UserRow.id == "u-alice"))
await session.commit()
# The service_account is still there after the rollback.
async with sf() as session:
row = await session.get(ServiceAccountRow, "sa-3")
assert row is not None
finally:
await _cleanup()
@@ -0,0 +1,88 @@
"""PR6 T6.11 — ThreadDataMiddleware writes under workspace layout."""
from __future__ import annotations
from types import SimpleNamespace
from unittest.mock import patch
import pytest
from deerflow.agents.middlewares.thread_data_middleware import ThreadDataMiddleware
from deerflow.config.paths import Paths
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
class _FakeRuntime:
def __init__(self, *, thread_id: str = "t1", run_id: str = "r1"):
self.context = {"thread_id": thread_id, "run_id": run_id}
def test_paths_resolve_under_workspace(tmp_path):
paths = Paths(tmp_path)
middleware = ThreadDataMiddleware(base_dir=str(tmp_path), lazy_init=True)
middleware._paths = paths
token = set_current_workspace(SimpleNamespace(id="ws-alpha", role="owner"))
try:
out = middleware.before_agent({"messages": []}, _FakeRuntime())
finally:
reset_current_workspace(token)
expected_root = tmp_path / "workspaces" / "ws-alpha" / "threads" / "t1" / "user-data"
assert out["thread_data"]["workspace_path"] == str(expected_root / "workspace")
assert out["thread_data"]["uploads_path"] == str(expected_root / "uploads")
assert out["thread_data"]["outputs_path"] == str(expected_root / "outputs")
assert out["thread_data"]["workspace_id"] == "ws-alpha"
@pytest.mark.no_auto_workspace
def test_falls_back_to_default_workspace(tmp_path):
"""Without a workspace contextvar, `get_effective_workspace_id` returns 'default'."""
paths = Paths(tmp_path)
middleware = ThreadDataMiddleware(base_dir=str(tmp_path), lazy_init=True)
middleware._paths = paths
out = middleware.before_agent({"messages": []}, _FakeRuntime())
expected_root = tmp_path / "workspaces" / "default" / "threads" / "t1" / "user-data"
assert out["thread_data"]["workspace_path"] == str(expected_root / "workspace")
assert out["thread_data"]["workspace_id"] == "default"
def test_eager_creates_directories_under_workspace(tmp_path):
paths = Paths(tmp_path)
middleware = ThreadDataMiddleware(base_dir=str(tmp_path), lazy_init=False)
middleware._paths = paths
token = set_current_workspace(SimpleNamespace(id="ws-beta", role="owner"))
try:
middleware.before_agent({"messages": []}, _FakeRuntime(thread_id="t2"))
finally:
reset_current_workspace(token)
root = tmp_path / "workspaces" / "ws-beta" / "threads" / "t2" / "user-data"
assert (root / "workspace").is_dir()
assert (root / "uploads").is_dir()
assert (root / "outputs").is_dir()
def test_get_config_fallback_still_workspace_scoped(tmp_path):
"""Thread_id resolution via LangGraph config still routes through workspace."""
paths = Paths(tmp_path)
middleware = ThreadDataMiddleware(base_dir=str(tmp_path), lazy_init=True)
middleware._paths = paths
class _Runtime:
context: dict = {}
with patch("deerflow.agents.middlewares.thread_data_middleware.get_config", return_value={"configurable": {"thread_id": "t-cfg"}}):
token = set_current_workspace(SimpleNamespace(id="ws-gamma", role="owner"))
try:
out = middleware.before_agent({"messages": []}, _Runtime())
finally:
reset_current_workspace(token)
assert "workspaces/ws-gamma/threads/t-cfg/user-data/workspace" in out["thread_data"]["workspace_path"]
+8 -8
View File
@@ -64,21 +64,21 @@ class TestThreadMetaRepository:
@pytest.mark.anyio
async def test_check_access_no_record_allows(self, tmp_path):
repo = await _make_repo(tmp_path)
assert await repo.check_access("unknown", "user1") is True
assert await repo.check_access("unknown", "user1", "test-workspace-autouse") is True
await _cleanup()
@pytest.mark.anyio
async def test_check_access_owner_matches(self, tmp_path):
repo = await _make_repo(tmp_path)
await repo.create("t1", user_id="user1")
assert await repo.check_access("t1", "user1") is True
assert await repo.check_access("t1", "user1", "test-workspace-autouse") is True
await _cleanup()
@pytest.mark.anyio
async def test_check_access_owner_mismatch(self, tmp_path):
repo = await _make_repo(tmp_path)
await repo.create("t1", user_id="user1")
assert await repo.check_access("t1", "user2") is False
assert await repo.check_access("t1", "user2", "test-workspace-autouse") is False
await _cleanup()
@pytest.mark.anyio
@@ -87,7 +87,7 @@ class TestThreadMetaRepository:
# Explicit user_id=None to bypass the new AUTO default that
# would otherwise pick up the test user from the autouse fixture.
await repo.create("t1", user_id=None)
assert await repo.check_access("t1", "anyone") is True
assert await repo.check_access("t1", "anyone", "test-workspace-autouse") is True
await _cleanup()
@pytest.mark.anyio
@@ -99,21 +99,21 @@ class TestThreadMetaRepository:
caller "claim" it as untracked. The strict mode demands a row.
"""
repo = await _make_repo(tmp_path)
assert await repo.check_access("never-existed", "user1", require_existing=True) is False
assert await repo.check_access("never-existed", "user1", "test-workspace-autouse", require_existing=True) is False
await _cleanup()
@pytest.mark.anyio
async def test_check_access_strict_owner_match_allowed(self, tmp_path):
repo = await _make_repo(tmp_path)
await repo.create("t1", user_id="user1")
assert await repo.check_access("t1", "user1", require_existing=True) is True
assert await repo.check_access("t1", "user1", "test-workspace-autouse", require_existing=True) is True
await _cleanup()
@pytest.mark.anyio
async def test_check_access_strict_owner_mismatch_denied(self, tmp_path):
repo = await _make_repo(tmp_path)
await repo.create("t1", user_id="user1")
assert await repo.check_access("t1", "user2", require_existing=True) is False
assert await repo.check_access("t1", "user2", "test-workspace-autouse", require_existing=True) is False
await _cleanup()
@pytest.mark.anyio
@@ -126,7 +126,7 @@ class TestThreadMetaRepository:
"""
repo = await _make_repo(tmp_path)
await repo.create("t1", user_id=None)
assert await repo.check_access("t1", "anyone", require_existing=True) is True
assert await repo.check_access("t1", "anyone", "test-workspace-autouse", require_existing=True) is True
await _cleanup()
@pytest.mark.anyio
@@ -0,0 +1,296 @@
"""Tests for ThreadMetaRepository workspace_id filtering (PR6 T6.1-T6.4).
The repository's three-state ``workspace_id`` semantics mirror ``user_id``:
- :data:`AUTO` (default): read from workspace contextvar
- Explicit ``str``: use the provided id
- Explicit ``None``: bypass workspace filter (migration / CLI)
Cross-workspace access (a thread in workspace A queried with workspace B)
must return ``None``, never the row. This is the load-bearing isolation
boundary tested here.
"""
from __future__ import annotations
from datetime import UTC, datetime
from types import SimpleNamespace
import pytest
from deerflow.persistence.thread_meta import ThreadMetaRepository
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
async def _seed_workspace(wid: str, *, owner_id: str = "test-user-autouse") -> None:
"""Insert a workspace row so threads_meta.workspace_id FK resolves."""
from deerflow.persistence.engine import get_session_factory
from deerflow.persistence.workspace.model import WorkspaceRow
factory = get_session_factory()
async with factory() as session:
existing = await session.get(WorkspaceRow, wid)
if existing is not None:
return
now = datetime.now(UTC)
session.add(
WorkspaceRow(
id=wid,
name=f"WS {wid}",
slug=wid.replace("_", "-")[:32],
status="active",
owner_id=owner_id,
created_at=now,
updated_at=now,
)
)
await session.commit()
async def _make_repo(tmp_path, *, workspaces: tuple[str, ...] = ()):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
for wid in workspaces:
await _seed_workspace(wid)
return ThreadMetaRepository(get_session_factory())
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
def _use_workspace(wid: str, role: str = "owner"):
"""Replace the autouse workspace contextvar inside a single test."""
return set_current_workspace(SimpleNamespace(id=wid, role=role))
class TestCreateWorkspace:
@pytest.mark.anyio
async def test_create_uses_workspace_context(self, tmp_path):
"""AUTO sentinel pulls workspace_id from the contextvar."""
repo = await _make_repo(tmp_path, workspaces=("ws-alpha",))
token = _use_workspace("ws-alpha")
try:
record = await repo.create("t1")
assert record["workspace_id"] == "ws-alpha"
finally:
reset_current_workspace(token)
await _cleanup()
@pytest.mark.anyio
async def test_create_explicit_workspace_overrides_context(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
record = await repo.create("t1", workspace_id="ws-beta")
assert record["workspace_id"] == "ws-beta"
finally:
reset_current_workspace(token)
await _cleanup()
@pytest.mark.anyio
async def test_create_workspace_none_rejected_by_orm(self, tmp_path):
"""After T5.11 (ORM nullable=False) explicit None creates fail at the DB layer.
Pre-PR6 the migration scripts relied on `workspace_id=None` to insert
orphan rows; that use is now restricted to **read** paths (filter
bypass). Writes must always carry a workspace.
"""
import sqlalchemy
repo = await _make_repo(tmp_path)
with pytest.raises((sqlalchemy.exc.IntegrityError, sqlalchemy.exc.DBAPIError)):
await repo.create("t1", workspace_id=None)
await _cleanup()
class TestGetWorkspace:
@pytest.mark.anyio
async def test_get_filters_by_workspace(self, tmp_path):
"""Cross-workspace get returns None even when user_id matches."""
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
assert await repo.get("t1", user_id="alice") is None
finally:
reset_current_workspace(token)
await _cleanup()
@pytest.mark.anyio
async def test_get_returns_row_in_same_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha",))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
record = await repo.get("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert record is not None
assert record["thread_id"] == "t1"
assert record["workspace_id"] == "ws-alpha"
@pytest.mark.anyio
async def test_get_workspace_none_bypasses_filter(self, tmp_path):
"""Explicit workspace_id=None lets migration scripts see any row."""
repo = await _make_repo(tmp_path, workspaces=("ws-alpha",))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
assert await repo.get("t1", user_id=None, workspace_id=None) is not None
finally:
reset_current_workspace(token)
await _cleanup()
class TestSearchUpdateDeleteWorkspace:
@pytest.mark.anyio
async def test_search_only_returns_current_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.create("t2", user_id="alice")
rows = await repo.search(user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
ids = {r["thread_id"] for r in rows}
assert ids == {"t2"}
@pytest.mark.anyio
async def test_update_status_blocked_across_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.update_status("t1", "busy", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-alpha")
try:
row = await repo.get("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert row["status"] == "idle"
@pytest.mark.anyio
async def test_update_display_name_blocked_across_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice", display_name="A")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.update_display_name("t1", "B", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-alpha")
try:
row = await repo.get("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert row["display_name"] == "A"
@pytest.mark.anyio
async def test_update_metadata_blocked_across_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice", metadata={"k": "alpha"})
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.update_metadata("t1", {"k": "beta"}, user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-alpha")
try:
row = await repo.get("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert row["metadata"] == {"k": "alpha"}
@pytest.mark.anyio
async def test_check_access_cross_workspace_false(self, tmp_path):
"""`check_access` returns False for cross-workspace, even with matching user_id."""
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
try:
assert await repo.check_access("t1", "alice", "ws-beta") is False
assert await repo.check_access("t1", "alice", "ws-alpha") is True
finally:
await _cleanup()
@pytest.mark.anyio
async def test_check_access_strict_cross_workspace_false(self, tmp_path):
"""require_existing=True path also denies cross-workspace."""
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
try:
assert await repo.check_access("t1", "alice", "ws-beta", require_existing=True) is False
assert await repo.check_access("t1", "alice", "ws-alpha", require_existing=True) is True
finally:
await _cleanup()
@pytest.mark.anyio
async def test_delete_blocked_across_workspace(self, tmp_path):
repo = await _make_repo(tmp_path, workspaces=("ws-alpha", "ws-beta"))
token = _use_workspace("ws-alpha")
try:
await repo.create("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-beta")
try:
await repo.delete("t1", user_id="alice")
finally:
reset_current_workspace(token)
token = _use_workspace("ws-alpha")
try:
row = await repo.get("t1", user_id="alice")
finally:
reset_current_workspace(token)
await _cleanup()
assert row is not None and row["thread_id"] == "t1"
@@ -0,0 +1,59 @@
"""PR6 T6.7 — POST /api/threads writes workspace_id from contextvar.
The `routers/threads.py:create_thread` path delegates to
``ThreadMetaStore.create`` *without* an explicit ``workspace_id`` it
relies on the AUTO sentinel pulling the value from the active workspace
contextvar that AuthMiddleware (or the test stub) sets. This test
covers the integration through the FastAPI TestClient stack: post a
thread under workspace A, then verify the persisted record carries
``workspace_id="ws-alpha"``.
"""
from __future__ import annotations
from collections.abc import Callable
from _router_auth_helpers import make_authed_test_app
from fastapi.testclient import TestClient
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from app.gateway.auth.models import ActiveWorkspace
from app.gateway.routers import threads
from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
def _workspace_factory(wid: str) -> Callable[[], ActiveWorkspace]:
def _factory() -> ActiveWorkspace:
return ActiveWorkspace(id=wid, role="owner")
return _factory
def _build_app(workspace_id: str):
app = make_authed_test_app(workspace_factory=_workspace_factory(workspace_id))
store = InMemoryStore()
checkpointer = InMemorySaver()
app.state.store = store
app.state.checkpointer = checkpointer
app.state.thread_store = MemoryThreadMetaStore(store)
app.include_router(threads.router)
return app, store
def test_post_thread_stamps_workspace_id_from_contextvar():
app, store = _build_app("ws-alpha")
with TestClient(app) as client:
response = client.post("/api/threads", json={"thread_id": "t1", "metadata": {}})
assert response.status_code == 200, response.text
item = store.get(("threads",), "t1")
assert item is not None
assert item.value["workspace_id"] == "ws-alpha"
def test_post_thread_under_different_workspace():
app, store = _build_app("ws-beta")
with TestClient(app) as client:
client.post("/api/threads", json={"thread_id": "t2", "metadata": {}})
assert store.get(("threads",), "t2").value["workspace_id"] == "ws-beta"
+6 -6
View File
@@ -27,21 +27,21 @@ class _PermissiveThreadMetaStore(MemoryThreadMetaStore):
timestamp wire format.
"""
async def _get_owned_record(self, thread_id, user_id, method_name): # type: ignore[override]
async def _get_owned_record(self, thread_id, user_id, workspace_id, method_name): # type: ignore[override]
item = await self._store.aget(THREADS_NS, thread_id)
return dict(item.value) if item is not None else None
async def check_access(self, thread_id, user_id, *, require_existing=False): # type: ignore[override]
async def check_access(self, thread_id, user_id, workspace_id, *, require_existing=False): # type: ignore[override]
item = await self._store.aget(THREADS_NS, thread_id)
if item is None:
return not require_existing
return True
async def create(self, thread_id, *, assistant_id=None, user_id=None, display_name=None, metadata=None): # type: ignore[override]
return await super().create(thread_id, assistant_id=assistant_id, user_id=None, display_name=display_name, metadata=metadata)
async def create(self, thread_id, *, assistant_id=None, user_id=None, workspace_id=None, display_name=None, metadata=None): # type: ignore[override]
return await super().create(thread_id, assistant_id=assistant_id, user_id=None, workspace_id=None, display_name=display_name, metadata=metadata)
async def search(self, *, metadata=None, status=None, limit=100, offset=0, user_id=None): # type: ignore[override]
return await super().search(metadata=metadata, status=status, limit=limit, offset=offset, user_id=None)
async def search(self, *, metadata=None, status=None, limit=100, offset=0, user_id=None, workspace_id=None): # type: ignore[override]
return await super().search(metadata=metadata, status=status, limit=limit, offset=offset, user_id=None, workspace_id=None)
def _build_thread_app() -> tuple[FastAPI, InMemoryStore, InMemorySaver]:
+127
View File
@@ -0,0 +1,127 @@
"""Boundary check: only allowlisted modules may directly import LangGraph
checkpoint/saver clients.
Direct use of the LangGraph checkpoint API anywhere outside the gateway thread
plumbing and the harness checkpointer factory is a workspace-isolation hazard:
arbitrary code paths could otherwise reach across threads/workspaces by
constructing their own savers. PR7 enforces this with an AST static scan.
Imports inside ``if TYPE_CHECKING:`` blocks are intentionally ignored they
never execute at runtime and therefore cannot bypass the boundary.
Allowlist lives in ``tests/boundary_allowlist.toml`` as a plain list of paths
relative to ``backend/``. Adding a new legitimate importer means appending a
line there in the same PR that introduces the import.
"""
from __future__ import annotations
import ast
import tomllib
from pathlib import Path
BACKEND_ROOT = Path(__file__).parent.parent # backend/
ALLOWLIST_FILE = Path(__file__).parent / "boundary_allowlist.toml"
# Match any submodule of these top-level packages.
TARGET_MODULE_PREFIXES: tuple[str, ...] = (
"langgraph.checkpoint",
"langgraph_checkpoint_postgres",
"langgraph_checkpoint_sqlite",
)
# Directories under backend/ that are not part of the running app.
EXCLUDED_TOP_LEVEL = ("tests", ".venv", "build", "dist", "docs", ".pytest_cache", "node_modules")
def _matches_target(name: str) -> bool:
return any(name == prefix or name.startswith(prefix + ".") for prefix in TARGET_MODULE_PREFIXES)
def _build_parent_map(tree: ast.AST) -> dict[int, ast.AST]:
parents: dict[int, ast.AST] = {}
for parent in ast.walk(tree):
for child in ast.iter_child_nodes(parent):
parents[id(child)] = parent
return parents
def _is_type_checking_test(test: ast.expr) -> bool:
"""Return True for ``TYPE_CHECKING`` or ``typing.TYPE_CHECKING`` conditions."""
if isinstance(test, ast.Name) and test.id == "TYPE_CHECKING":
return True
if isinstance(test, ast.Attribute) and test.attr == "TYPE_CHECKING":
return True
return False
def _inside_type_checking(node: ast.AST, parents: dict[int, ast.AST]) -> bool:
current = parents.get(id(node))
while current is not None:
if isinstance(current, ast.If) and _is_type_checking_test(current.test):
return True
current = parents.get(id(current))
return False
def collect_runtime_checkpoint_imports(filepath: Path) -> list[tuple[int, str]]:
"""Return ``(lineno, module_path)`` for every runtime import that targets a banned module."""
source = filepath.read_text(encoding="utf-8")
try:
tree = ast.parse(source, filename=str(filepath))
except SyntaxError:
return []
parents = _build_parent_map(tree)
hits: list[tuple[int, str]] = []
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom):
module = node.module or ""
if _matches_target(module) and not _inside_type_checking(node, parents):
hits.append((node.lineno, module))
elif isinstance(node, ast.Import):
for alias in node.names:
if _matches_target(alias.name) and not _inside_type_checking(node, parents):
hits.append((node.lineno, alias.name))
return hits
def _iter_backend_py_files() -> list[Path]:
candidates: list[Path] = []
for path in sorted(BACKEND_ROOT.rglob("*.py")):
rel_parts = path.relative_to(BACKEND_ROOT).parts
if rel_parts and rel_parts[0] in EXCLUDED_TOP_LEVEL:
continue
if "__pycache__" in rel_parts:
continue
candidates.append(path)
return candidates
def _load_allowlist() -> set[str]:
data = tomllib.loads(ALLOWLIST_FILE.read_text(encoding="utf-8"))
raw = data.get("langgraph_checkpoint_importers", [])
return set(raw)
def scan_violations(allowlist: set[str]) -> list[str]:
"""Return formatted violation lines (one per banned import outside the allowlist)."""
violations: list[str] = []
for py in _iter_backend_py_files():
rel = py.relative_to(BACKEND_ROOT).as_posix()
for lineno, module in collect_runtime_checkpoint_imports(py):
if rel in allowlist:
continue
violations.append(f" {rel}:{lineno} imports {module}")
return violations
def test_only_allowlisted_modules_import_langgraph_checkpoint() -> None:
allowlist = _load_allowlist()
violations = scan_violations(allowlist)
assert not violations, (
"Unauthorized direct imports of langgraph.checkpoint.* detected. "
"Either route the access through `app.gateway.deps.get_checkpointer` "
"or, if this is a legitimate new importer, add the path to "
"tests/boundary_allowlist.toml in the same PR.\n" + "\n".join(violations)
)
@@ -0,0 +1,93 @@
"""Self-tests for the boundary scanner in ``test_workspace_boundary``.
These tests guard against the scanner silently passing because it cannot
detect anything. They feed synthetic ``.py`` snippets to the per-file
collector and assert it produces (or correctly suppresses) the expected
hits so we know the integration test in ``test_workspace_boundary.py``
would actually fail if a real violation appeared.
"""
from __future__ import annotations
from pathlib import Path
from test_workspace_boundary import collect_runtime_checkpoint_imports
def _write(tmp_path: Path, body: str) -> Path:
target = tmp_path / "sample.py"
target.write_text(body, encoding="utf-8")
return target
def test_flags_direct_from_import(tmp_path: Path) -> None:
target = _write(
tmp_path,
"from langgraph.checkpoint.postgres import AsyncPostgresSaver\n",
)
hits = collect_runtime_checkpoint_imports(target)
assert hits == [(1, "langgraph.checkpoint.postgres")]
def test_flags_bare_module_import(tmp_path: Path) -> None:
target = _write(
tmp_path,
"import langgraph.checkpoint.memory # noqa: F401\n",
)
hits = collect_runtime_checkpoint_imports(target)
assert hits == [(1, "langgraph.checkpoint.memory")]
def test_flags_third_party_checkpoint_package(tmp_path: Path) -> None:
target = _write(
tmp_path,
"from langgraph_checkpoint_postgres import PostgresSaver\nfrom langgraph_checkpoint_sqlite import SqliteSaver\n",
)
hits = collect_runtime_checkpoint_imports(target)
assert (1, "langgraph_checkpoint_postgres") in hits
assert (2, "langgraph_checkpoint_sqlite") in hits
def test_skips_type_checking_block_name_form(tmp_path: Path) -> None:
target = _write(
tmp_path,
"from typing import TYPE_CHECKING\n\nif TYPE_CHECKING:\n from langgraph.checkpoint.base import BaseCheckpointSaver # noqa: F401\n",
)
assert collect_runtime_checkpoint_imports(target) == []
def test_skips_type_checking_block_attribute_form(tmp_path: Path) -> None:
target = _write(
tmp_path,
"import typing\n\nif typing.TYPE_CHECKING:\n from langgraph.checkpoint.base import BaseCheckpointSaver # noqa: F401\n",
)
assert collect_runtime_checkpoint_imports(target) == []
def test_skips_nested_type_checking_block(tmp_path: Path) -> None:
target = _write(
tmp_path,
"from typing import TYPE_CHECKING\n\nif True:\n if TYPE_CHECKING:\n from langgraph.checkpoint.postgres import AsyncPostgresSaver # noqa: F401\n",
)
assert collect_runtime_checkpoint_imports(target) == []
def test_does_not_flag_unrelated_imports(tmp_path: Path) -> None:
target = _write(
tmp_path,
"import os\nfrom langgraph.graph.state import CompiledStateGraph # noqa: F401\nfrom app.gateway.deps import get_checkpointer # noqa: F401\n",
)
assert collect_runtime_checkpoint_imports(target) == []
def test_does_not_flag_string_literal_with_module_name(tmp_path: Path) -> None:
target = _write(
tmp_path,
'TARGET = "langgraph.checkpoint.postgres"\n',
)
assert collect_runtime_checkpoint_imports(target) == []
def test_handles_syntax_error_gracefully(tmp_path: Path) -> None:
target = _write(tmp_path, "def broken(:\n")
assert collect_runtime_checkpoint_imports(target) == []
+176
View File
@@ -0,0 +1,176 @@
"""Tests for runtime.workspace_context — workspace contextvar semantics.
Mirrors :mod:`test_user_context` but for the workspace contextvar
introduced in Stage 0 PR3. No autouse workspace fixture exists yet
(PR4 will add it together with the AuthMiddleware injection), so these
tests run against a clean contextvar.
"""
import uuid
from types import SimpleNamespace
import pytest
from deerflow.runtime.workspace_context import (
AUTO,
DEFAULT_WORKSPACE_ID,
CurrentWorkspace,
get_current_workspace,
get_effective_workspace_id,
require_current_workspace,
reset_current_workspace,
resolve_workspace_id,
set_current_workspace,
)
# ---------------------------------------------------------------------------
# get_current_workspace / require_current_workspace / set+reset round-trip
# ---------------------------------------------------------------------------
@pytest.mark.no_auto_workspace
def test_default_is_none():
"""Before any set, contextvar returns None."""
assert get_current_workspace() is None
@pytest.mark.no_auto_workspace
def test_set_and_reset_roundtrip():
"""set_current_workspace returns a token that reset restores."""
workspace = SimpleNamespace(id="ws-1", role="owner")
token = set_current_workspace(workspace)
try:
assert get_current_workspace() is workspace
finally:
reset_current_workspace(token)
assert get_current_workspace() is None
@pytest.mark.no_auto_workspace
def test_require_current_workspace_raises_when_unset():
"""require_current_workspace raises RuntimeError if contextvar is unset."""
assert get_current_workspace() is None
with pytest.raises(RuntimeError, match="without workspace context"):
require_current_workspace()
def test_require_current_workspace_returns_workspace_when_set():
"""require_current_workspace returns the workspace when contextvar is set."""
workspace = SimpleNamespace(id="ws-2", role="admin")
token = set_current_workspace(workspace)
try:
assert require_current_workspace() is workspace
finally:
reset_current_workspace(token)
# ---------------------------------------------------------------------------
# CurrentWorkspace Protocol — must require BOTH .id and .role
# ---------------------------------------------------------------------------
def test_protocol_accepts_id_and_role():
"""CurrentWorkspace is satisfied by any object with .id and .role."""
workspace = SimpleNamespace(id="ws-3", role="member")
assert isinstance(workspace, CurrentWorkspace)
def test_protocol_rejects_missing_role():
"""An object with only .id (no .role) is NOT a workspace."""
user_shaped = SimpleNamespace(id="ws-4")
assert not isinstance(user_shaped, CurrentWorkspace)
def test_protocol_rejects_no_id():
"""An object without .id does not satisfy CurrentWorkspace."""
not_a_workspace = SimpleNamespace(role="owner")
assert not isinstance(not_a_workspace, CurrentWorkspace)
# ---------------------------------------------------------------------------
# get_effective_workspace_id / DEFAULT_WORKSPACE_ID tests
# ---------------------------------------------------------------------------
def test_default_workspace_id_is_default():
assert DEFAULT_WORKSPACE_ID == "default"
@pytest.mark.no_auto_workspace
def test_effective_workspace_id_returns_default_when_no_workspace():
"""No workspace in context -> fallback to DEFAULT_WORKSPACE_ID."""
assert get_effective_workspace_id() == "default"
def test_effective_workspace_id_returns_workspace_id_when_set():
workspace = SimpleNamespace(id="ws-abc-123", role="owner")
token = set_current_workspace(workspace)
try:
assert get_effective_workspace_id() == "ws-abc-123"
finally:
reset_current_workspace(token)
def test_effective_workspace_id_coerces_to_str():
"""workspace.id might be a UUID object; must come back as str."""
wid = uuid.uuid4()
workspace = SimpleNamespace(id=wid, role="owner")
token = set_current_workspace(workspace)
try:
assert get_effective_workspace_id() == str(wid)
finally:
reset_current_workspace(token)
# ---------------------------------------------------------------------------
# resolve_workspace_id three-state semantics
# ---------------------------------------------------------------------------
def test_resolve_auto_reads_from_contextvar():
workspace = SimpleNamespace(id="ws-resolve-1", role="owner")
token = set_current_workspace(workspace)
try:
assert resolve_workspace_id(AUTO) == "ws-resolve-1"
finally:
reset_current_workspace(token)
@pytest.mark.no_auto_workspace
def test_resolve_auto_raises_when_unset():
assert get_current_workspace() is None
with pytest.raises(RuntimeError, match="workspace_id=AUTO but no workspace"):
resolve_workspace_id(AUTO, method_name="TestRepo.search")
def test_resolve_explicit_str_overrides_contextvar():
workspace = SimpleNamespace(id="ws-ctx", role="owner")
token = set_current_workspace(workspace)
try:
# Explicit value beats contextvar — admin override / test path.
assert resolve_workspace_id("ws-explicit") == "ws-explicit"
finally:
reset_current_workspace(token)
def test_resolve_explicit_none_means_no_filter():
workspace = SimpleNamespace(id="ws-ctx-2", role="owner")
token = set_current_workspace(workspace)
try:
# Explicit None opts out of workspace filtering (migration scripts).
assert resolve_workspace_id(None) is None
finally:
reset_current_workspace(token)
def test_resolve_auto_coerces_uuid_to_str():
"""resolve_workspace_id with AUTO returns str even if workspace.id is UUID."""
wid = uuid.uuid4()
workspace = SimpleNamespace(id=wid, role="owner")
token = set_current_workspace(workspace)
try:
resolved = resolve_workspace_id(AUTO)
assert resolved == str(wid)
assert isinstance(resolved, str)
finally:
reset_current_workspace(token)
@@ -0,0 +1,127 @@
"""PR6 T6.8 — cross-workspace isolation boundary (e2e).
Wires a ``MemoryThreadMetaStore`` (LangGraph BaseStore backed) into a
stub-authed FastAPI app and asserts that any request from workspace B
against a thread created in workspace A returns **404**, regardless of
matching user_id. The cross-workspace block fires in
``ThreadMetaStore.check_access`` and is converted to 404 by
``@require_permission(owner_check=True)``.
We use the memory-backed implementation so the test stays in the test
event loop end-to-end (the SQL engine binds to whatever loop owns
``init_engine`` and the TestClient spins its own loop, which would
collide). The decorator path it exercises is the same as production;
the SQL repository's identical workspace filter is unit-covered by
``test_thread_meta_workspace_filter.py``.
Covers:
- ``GET /api/threads/{tid}`` read (require_existing=False)
- ``DELETE /api/threads/{tid}`` destructive (require_existing=True)
- ``PATCH /api/threads/{tid}`` destructive write
- positive control: same-workspace GET still succeeds
"""
from __future__ import annotations
from collections.abc import Callable
from types import SimpleNamespace
from uuid import uuid4
from _router_auth_helpers import make_authed_test_app
from fastapi.testclient import TestClient
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore
from app.gateway.auth.models import ActiveWorkspace, User
from app.gateway.routers import threads
from deerflow.persistence.thread_meta.memory import MemoryThreadMetaStore
from deerflow.runtime.workspace_context import (
reset_current_workspace,
set_current_workspace,
)
def _workspace_factory(wid: str) -> Callable[[], ActiveWorkspace]:
def _factory() -> ActiveWorkspace:
return ActiveWorkspace(id=wid, role="owner")
return _factory
def _user_factory(uid: str) -> Callable[[], User]:
def _factory() -> User:
return User(email=f"{uid}@example.com", password_hash="x", system_role="user", id=uid)
return _factory
def _seed_thread(store, *, thread_id: str, user_id: str, workspace_id: str) -> None:
"""Insert a thread record under a specific workspace, bypassing the autouse fixture."""
import asyncio
async def _go():
meta_store = MemoryThreadMetaStore(store)
token = set_current_workspace(SimpleNamespace(id=workspace_id, role="owner"))
try:
await meta_store.create(thread_id, user_id=user_id)
finally:
reset_current_workspace(token)
asyncio.run(_go())
def _build_app(*, user_id: str, workspace_id: str):
app = make_authed_test_app(
user_factory=_user_factory(user_id),
workspace_factory=_workspace_factory(workspace_id),
override_user_contextvar=True,
)
store = InMemoryStore()
app.state.store = store
app.state.checkpointer = InMemorySaver()
app.state.thread_store = MemoryThreadMetaStore(store)
app.include_router(threads.router)
return app, store
def test_cross_workspace_get_returns_404():
user_id = str(uuid4())
app, store = _build_app(user_id=user_id, workspace_id="ws-beta")
_seed_thread(store, thread_id="t1", user_id=user_id, workspace_id="ws-alpha")
with TestClient(app) as client:
response = client.get("/api/threads/t1")
assert response.status_code == 404
assert "not found" in response.json()["detail"].lower()
def test_cross_workspace_delete_returns_404():
user_id = str(uuid4())
app, store = _build_app(user_id=user_id, workspace_id="ws-beta")
_seed_thread(store, thread_id="t1", user_id=user_id, workspace_id="ws-alpha")
with TestClient(app) as client:
response = client.delete("/api/threads/t1")
assert response.status_code == 404
def test_cross_workspace_patch_returns_404():
user_id = str(uuid4())
app, store = _build_app(user_id=user_id, workspace_id="ws-beta")
_seed_thread(store, thread_id="t1", user_id=user_id, workspace_id="ws-alpha")
with TestClient(app) as client:
response = client.patch("/api/threads/t1", json={"metadata": {"k": "v"}})
assert response.status_code == 404
def test_same_workspace_get_succeeds():
"""Positive control: when the workspace matches, the row is returned."""
user_id = str(uuid4())
app, store = _build_app(user_id=user_id, workspace_id="ws-alpha")
_seed_thread(store, thread_id="t1", user_id=user_id, workspace_id="ws-alpha")
with TestClient(app) as client:
response = client.get("/api/threads/t1")
assert response.status_code == 200
assert response.json()["thread_id"] == "t1"
@@ -0,0 +1,216 @@
"""Tests for WorkspaceMembershipRepository (Stage 0 PR3 T3.7)."""
from __future__ import annotations
import pytest
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace import WorkspaceRepository
from deerflow.persistence.workspace_membership import (
MembershipValidationError,
WorkspaceMembershipRepository,
)
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def _setup(tmp_path):
"""Create both repos against a fresh SQLite DB."""
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
sf = get_session_factory()
return (
WorkspaceRepository(sf),
WorkspaceMembershipRepository(sf),
sf,
)
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_user(sf, user_id: str, email: str) -> None:
async with sf() as session:
session.add(UserRow(id=user_id, email=email))
await session.commit()
# ---------------------------------------------------------------------------
# add / remove smoke
# ---------------------------------------------------------------------------
async def test_add_then_get_role(tmp_path):
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
ws = await ws_repo.create(name="W1", slug="w-1", owner_id="u-1")
added = await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="owner")
assert added["role"] == "owner"
role = await m_repo.get_role(workspace_id=ws["id"], user_id="u-1")
assert role == "owner"
finally:
await _cleanup()
async def test_remove_returns_true_when_deleted_false_when_missing(tmp_path):
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
ws = await ws_repo.create(name="W1", slug="rm-w", owner_id="u-1")
await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="owner")
assert await m_repo.remove(workspace_id=ws["id"], user_id="u-1") is True
# second remove of same row -> nothing to delete
assert await m_repo.remove(workspace_id=ws["id"], user_id="u-1") is False
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# partial unique: exactly one owner per workspace
# ---------------------------------------------------------------------------
async def test_cannot_add_second_owner(tmp_path):
"""Inserting a second role='owner' in the same workspace must raise IntegrityError."""
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
await _seed_user(sf, "u-2", "u2@example.com")
ws = await ws_repo.create(name="W", slug="one-owner", owner_id="u-1")
await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="owner")
with pytest.raises(IntegrityError):
await m_repo.add(workspace_id=ws["id"], user_id="u-2", role="owner")
finally:
await _cleanup()
async def test_admin_and_member_dont_trigger_partial_unique(tmp_path):
"""Multiple admin/member rows in one workspace are fine (Stage 2 forward compat)."""
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
await _seed_user(sf, "u-2", "u2@example.com")
await _seed_user(sf, "u-3", "u3@example.com")
ws = await ws_repo.create(name="W", slug="multi-admin", owner_id="u-1")
await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="owner")
await m_repo.add(workspace_id=ws["id"], user_id="u-2", role="admin")
# Second admin OK
await m_repo.add(workspace_id=ws["id"], user_id="u-3", role="admin")
members = await m_repo.list_by_workspace(workspace_id=ws["id"])
assert len(members) == 3
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# CASCADE: deleting a user wipes their memberships
# ---------------------------------------------------------------------------
async def test_cascade_delete_user_removes_memberships(tmp_path):
"""FK ON DELETE CASCADE on user_id."""
from sqlalchemy import delete
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-keep", "keep@example.com")
await _seed_user(sf, "u-purge", "purge@example.com")
ws = await ws_repo.create(name="W", slug="cascade", owner_id="u-keep")
await m_repo.add(workspace_id=ws["id"], user_id="u-keep", role="owner")
await m_repo.add(workspace_id=ws["id"], user_id="u-purge", role="admin")
async with sf() as session:
await session.execute(delete(UserRow).where(UserRow.id == "u-purge"))
await session.commit()
members = await m_repo.list_by_workspace(workspace_id=ws["id"])
member_ids = [m["user_id"] for m in members]
assert "u-purge" not in member_ids
assert "u-keep" in member_ids
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# list_by_user ordering (most recent joined_at first)
# ---------------------------------------------------------------------------
async def test_list_by_user_orders_recent_first(tmp_path):
import asyncio
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
await _seed_user(sf, "u-2", "u2@example.com")
# u-1 owns workspace A
ws_a = await ws_repo.create(name="A", slug="ord-a", owner_id="u-1")
await m_repo.add(workspace_id=ws_a["id"], user_id="u-1", role="owner")
await asyncio.sleep(0.01) # ensure distinct joined_at
# u-1 later joins workspace C as a member (u-2 owns it)
ws_c = await ws_repo.create(name="C", slug="ord-c", owner_id="u-2")
await m_repo.add(workspace_id=ws_c["id"], user_id="u-1", role="member")
memberships = await m_repo.list_by_user(user_id="u-1")
slugs_in_order = [(m["workspace_id"], m["role"]) for m in memberships]
# 'ws_c member' joined AFTER 'ws_a owner' → ws_c first
assert slugs_in_order[0] == (ws_c["id"], "member")
assert slugs_in_order[1] == (ws_a["id"], "owner")
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# role validation + change_role
# ---------------------------------------------------------------------------
async def test_add_rejects_unknown_role(tmp_path):
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
ws = await ws_repo.create(name="W", slug="invrole", owner_id="u-1")
with pytest.raises(MembershipValidationError, match="allowed set"):
await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="viewer")
finally:
await _cleanup()
async def test_change_role_admin_to_member(tmp_path):
ws_repo, m_repo, sf = await _setup(tmp_path)
try:
await _seed_user(sf, "u-1", "u1@example.com")
await _seed_user(sf, "u-2", "u2@example.com")
ws = await ws_repo.create(name="W", slug="chg", owner_id="u-1")
await m_repo.add(workspace_id=ws["id"], user_id="u-1", role="owner")
await m_repo.add(workspace_id=ws["id"], user_id="u-2", role="admin")
ok = await m_repo.change_role(workspace_id=ws["id"], user_id="u-2", new_role="member")
assert ok is True
assert await m_repo.get_role(workspace_id=ws["id"], user_id="u-2") == "member"
# change_role on a non-member returns False
miss = await m_repo.change_role(workspace_id=ws["id"], user_id="u-nonexistent", new_role="member")
assert miss is False
finally:
await _cleanup()
@@ -0,0 +1,85 @@
"""Partial unique index `idx_one_owner_per_workspace` works on Postgres.
The same assertion against SQLite is in
:mod:`test_workspace_membership_repo::test_cannot_add_second_owner`.
This file adds the Postgres twin via the @pytest.mark.postgres
testcontainers fixture from PR1.
Why duplicate the test:
- SQLite and Postgres parse ``WHERE`` clauses differently. We declare
both ``sqlite_where`` and ``postgresql_where`` on the Index; this
test pins that Postgres genuinely enforces the partial-unique
constraint, not just that SQLAlchemy emits the DDL.
"""
from __future__ import annotations
import pytest
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace import WorkspaceRepository
from deerflow.persistence.workspace_membership import WorkspaceMembershipRepository
pytestmark = [pytest.mark.postgres, pytest.mark.anyio]
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
async def test_partial_unique_on_owner_enforced_on_postgres(postgres_url: str) -> None:
"""Postgres: inserting a 2nd owner must raise IntegrityError, same as SQLite."""
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
await init_engine("postgres", url=postgres_url)
try:
sf = get_session_factory()
ws_repo = WorkspaceRepository(sf)
m_repo = WorkspaceMembershipRepository(sf)
# Seed two users
async with sf() as session:
session.add(UserRow(id="pg-u1", email="pg1@example.com"))
session.add(UserRow(id="pg-u2", email="pg2@example.com"))
await session.commit()
ws = await ws_repo.create(name="PG W", slug="pg-one-owner", owner_id="pg-u1")
await m_repo.add(workspace_id=ws["id"], user_id="pg-u1", role="owner")
with pytest.raises(IntegrityError):
await m_repo.add(workspace_id=ws["id"], user_id="pg-u2", role="owner")
finally:
await close_engine()
async def test_multiple_admins_allowed_on_postgres(postgres_url: str) -> None:
"""Postgres: partial unique on owner must NOT block multiple admins."""
from deerflow.persistence.engine import close_engine, get_session_factory, init_engine
await init_engine("postgres", url=postgres_url)
try:
sf = get_session_factory()
ws_repo = WorkspaceRepository(sf)
m_repo = WorkspaceMembershipRepository(sf)
async with sf() as session:
session.add(UserRow(id="pg-a1", email="a1@example.com"))
session.add(UserRow(id="pg-a2", email="a2@example.com"))
session.add(UserRow(id="pg-a3", email="a3@example.com"))
await session.commit()
ws = await ws_repo.create(name="PG W", slug="pg-multi-admin", owner_id="pg-a1")
await m_repo.add(workspace_id=ws["id"], user_id="pg-a1", role="owner")
await m_repo.add(workspace_id=ws["id"], user_id="pg-a2", role="admin")
await m_repo.add(workspace_id=ws["id"], user_id="pg-a3", role="admin")
members = await m_repo.list_by_workspace(workspace_id=ws["id"])
assert {(m["user_id"], m["role"]) for m in members} == {
("pg-a1", "owner"),
("pg-a2", "admin"),
("pg-a3", "admin"),
}
finally:
await close_engine()
+296
View File
@@ -0,0 +1,296 @@
"""Tests for WorkspaceRepository (Stage 0 PR3).
Pattern mirrors :mod:`test_feedback`: SQLite ephemeral DB per test via
tmp_path, no real Postgres needed at the unit-test layer. Partial-unique
double-driver validation lives in :mod:`test_workspace_partial_unique`
(T3.8, runs against both backends).
"""
from __future__ import annotations
from types import SimpleNamespace
import pytest
from sqlalchemy.exc import IntegrityError
from deerflow.persistence.user.model import UserRow
from deerflow.persistence.workspace import WorkspaceRepository, WorkspaceValidationError
async def _make_repo(tmp_path):
from deerflow.persistence.engine import get_session_factory, init_engine
url = f"sqlite+aiosqlite:///{tmp_path / 'test.db'}"
await init_engine("sqlite", url=url, sqlite_dir=str(tmp_path))
return WorkspaceRepository(get_session_factory())
async def _cleanup():
from deerflow.persistence.engine import close_engine
await close_engine()
async def _seed_user(repo, user_id: str = "u-alice", email: str = "alice@example.com") -> None:
"""Create a user row so workspace.owner_id FK is satisfied."""
async with repo._sf() as session:
session.add(UserRow(id=user_id, email=email))
await session.commit()
pytestmark = pytest.mark.anyio
@pytest.fixture
def anyio_backend() -> str:
return "asyncio"
# ---------------------------------------------------------------------------
# create / get_by_slug round-trip
# ---------------------------------------------------------------------------
async def test_create_then_lookup_by_slug(tmp_path):
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
created = await repo.create(name="Alice's Workspace", slug="alice", owner_id="u-alice")
assert created["slug"] == "alice"
assert created["status"] == "active"
assert created["owner_id"] == "u-alice"
assert len(created["id"]) == 36 # UUID v4
fetched = await repo.get_by_slug("alice")
assert fetched is not None
assert fetched["id"] == created["id"]
finally:
await _cleanup()
async def test_get_by_slug_returns_none_when_missing(tmp_path):
repo = await _make_repo(tmp_path)
try:
assert await repo.get_by_slug("nonexistent") is None
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# slug uniqueness + format + blacklist
# ---------------------------------------------------------------------------
async def test_create_rejects_duplicate_slug(tmp_path):
"""Two workspaces with the same slug — second raises IntegrityError."""
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
await repo.create(name="A", slug="dup", owner_id="u-alice")
with pytest.raises(IntegrityError):
await repo.create(name="B", slug="dup", owner_id="u-alice")
finally:
await _cleanup()
@pytest.mark.parametrize(
"bad_slug",
[
"ab", # too short
"x" * 33, # too long
"UPPER", # uppercase
"has space", # space
"-start-with-dash", # bad start
"end-with-dash-", # bad end
"double--dash", # consecutive dashes
"underscore_not_ok", # underscore
],
)
async def test_create_rejects_invalid_slug_pattern(tmp_path, bad_slug):
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
with pytest.raises(WorkspaceValidationError, match="(pattern|length)"):
await repo.create(name="x", slug=bad_slug, owner_id="u-alice")
finally:
await _cleanup()
@pytest.mark.parametrize("reserved", ["admin", "api", "auth", "settings", "billing", "select-workspace"])
async def test_create_rejects_reserved_slug(tmp_path, reserved):
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
with pytest.raises(WorkspaceValidationError, match="reserved"):
await repo.create(name="x", slug=reserved, owner_id="u-alice")
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# status state machine
# ---------------------------------------------------------------------------
async def test_status_state_transitions(tmp_path):
"""active → suspended → deleted are all accepted."""
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
ws = await repo.create(name="x", slug="trans", owner_id="u-alice")
assert ws["status"] == "active"
await repo.update_status(ws["id"], "suspended")
async with repo._sf() as session:
from deerflow.persistence.workspace.model import WorkspaceRow
row = await session.get(WorkspaceRow, ws["id"])
assert row.status == "suspended"
await repo.update_status(ws["id"], "deleted")
async with repo._sf() as session:
from deerflow.persistence.workspace.model import WorkspaceRow
row = await session.get(WorkspaceRow, ws["id"])
assert row.status == "deleted"
finally:
await _cleanup()
async def test_update_status_rejects_unknown_value(tmp_path):
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
ws = await repo.create(name="x", slug="rejstat", owner_id="u-alice")
with pytest.raises(WorkspaceValidationError, match="allowed set"):
await repo.update_status(ws["id"], "weird-state")
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# CASCADE: workspace.delete() drops dependent memberships
# ---------------------------------------------------------------------------
async def test_delete_cascades_to_memberships(tmp_path):
"""Deleting a workspace removes its membership rows (FK CASCADE)."""
from sqlalchemy import select
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo)
ws = await repo.create(name="x", slug="casc", owner_id="u-alice")
# Insert an owner membership manually (repository pattern is single-
# responsibility; registration flow will normally insert both rows
# in one transaction).
async with repo._sf() as session:
session.add(WorkspaceMembershipRow(workspace_id=ws["id"], user_id="u-alice", role="owner"))
await session.commit()
await repo.delete(ws["id"])
async with repo._sf() as session:
remaining = (await session.execute(select(WorkspaceMembershipRow).where(WorkspaceMembershipRow.workspace_id == ws["id"]))).scalars().all()
assert remaining == [], "memberships should be CASCADE-deleted with workspace"
finally:
await _cleanup()
# ---------------------------------------------------------------------------
# membership-aware get + list_by_user
# ---------------------------------------------------------------------------
@pytest.mark.no_auto_user
async def test_get_returns_none_for_non_member(tmp_path):
"""User-A creates a workspace; User-B's `get(wsA)` returns None."""
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
from deerflow.runtime.user_context import reset_current_user, set_current_user
repo = await _make_repo(tmp_path)
try:
# Seed two users
async with repo._sf() as session:
session.add(UserRow(id="u-A", email="a@example.com"))
session.add(UserRow(id="u-B", email="b@example.com"))
await session.commit()
# User A creates workspace + becomes owner
ws = await repo.create(name="A's WS", slug="a-ws", owner_id="u-A")
async with repo._sf() as session:
session.add(WorkspaceMembershipRow(workspace_id=ws["id"], user_id="u-A", role="owner"))
await session.commit()
# User B attempts to read it via contextvar
user_b = SimpleNamespace(id="u-B")
token = set_current_user(user_b)
try:
assert await repo.get(ws["id"]) is None
finally:
reset_current_user(token)
# User A's own get succeeds
user_a = SimpleNamespace(id="u-A")
token = set_current_user(user_a)
try:
row = await repo.get(ws["id"])
assert row is not None
assert row["slug"] == "a-ws"
finally:
reset_current_user(token)
finally:
await _cleanup()
@pytest.mark.no_auto_user
async def test_list_by_user_excludes_other_workspaces(tmp_path):
from deerflow.persistence.workspace_membership.model import WorkspaceMembershipRow
from deerflow.runtime.user_context import reset_current_user, set_current_user
repo = await _make_repo(tmp_path)
try:
async with repo._sf() as session:
session.add(UserRow(id="u-A", email="a@example.com"))
session.add(UserRow(id="u-B", email="b@example.com"))
await session.commit()
ws_a = await repo.create(name="A", slug="ws-a", owner_id="u-A")
ws_b = await repo.create(name="B", slug="ws-b", owner_id="u-B")
async with repo._sf() as session:
session.add(WorkspaceMembershipRow(workspace_id=ws_a["id"], user_id="u-A", role="owner"))
session.add(WorkspaceMembershipRow(workspace_id=ws_b["id"], user_id="u-B", role="owner"))
await session.commit()
token = set_current_user(SimpleNamespace(id="u-A"))
try:
workspaces = await repo.list_by_user()
assert [w["slug"] for w in workspaces] == ["ws-a"]
finally:
reset_current_user(token)
finally:
await _cleanup()
async def test_list_by_user_bypass_returns_all(tmp_path):
"""user_id=None opts out of membership filter (migration path)."""
repo = await _make_repo(tmp_path)
try:
await _seed_user(repo, "u-A", "a@example.com")
await _seed_user(repo, "u-B", "b@example.com")
await repo.create(name="A", slug="all-a", owner_id="u-A")
await repo.create(name="B", slug="all-b", owner_id="u-B")
workspaces = await repo.list_by_user(user_id=None)
# PR6 conftest auto-seeds an "autouse-test" workspace via the
# ``Base.metadata.after_create`` hook so business-row FKs resolve.
# ``user_id=None`` bypasses the membership filter, so it surfaces
# alongside the two rows the test inserted — that is the intended
# "no filter" behaviour. Assert the inserted ones are present.
slugs = sorted(w["slug"] for w in workspaces)
assert "all-a" in slugs
assert "all-b" in slugs
finally:
await _cleanup()
+98
View File
@@ -0,0 +1,98 @@
"""Slug helpers for registration / initialize (Stage 0 PR4 T4.10).
Two surfaces under test:
- ``auto_slug_from_email`` pure transform; no DB dependency.
- ``next_available_slug`` async collision walker; we stub the
``exists_check`` callable so the test stays a unit test.
"""
from __future__ import annotations
import re
import pytest
from app.gateway.auth.workspace_slug import auto_slug_from_email, next_available_slug
# Mirror of the schema's slug pattern. Keeping it inline keeps this
# test self-contained — if the schema regex changes we want this test
# to refuse to lie about validity.
_SLUG_PATTERN = re.compile(r"^[a-z0-9](-?[a-z0-9])*$")
@pytest.mark.parametrize(
("email", "expected"),
[
("foo@example.com", "foo"),
("foo.bar@example.com", "foo-bar"),
("foo+spam@example.com", "foo-spam"),
("foo_bar@example.com", "foo-bar"),
("Foo.Bar@example.com", "foo-bar"),
("foo.bar+spam@example.com", "foo-bar-spam"),
("aaaaaaaaaabbbbbbbbbbccccccccccddddd@example.com", "aaaaaaaaaabbbbbbbbbbccccccccccdd"),
],
)
def test_auto_slug_known_inputs(email: str, expected: str) -> None:
"""Deterministic mapping for the inputs called out in the design doc."""
assert auto_slug_from_email(email) == expected
assert _SLUG_PATTERN.fullmatch(auto_slug_from_email(email)), "schema regex must accept the output"
@pytest.mark.parametrize(
"email",
[
"@example.com", # no local part
"a@example.com", # too short
"ab@example.com", # still too short
"...+_+...@example.com", # only separators
"---@example.com", # only hyphens
"🎉@example.com", # non-ASCII
],
)
def test_auto_slug_falls_back_when_unusable(email: str) -> None:
"""Pathological emails fall back to ``user-{token}`` so the slug is always valid."""
slug = auto_slug_from_email(email)
assert slug.startswith("user-"), f"expected fallback, got {slug!r}"
assert _SLUG_PATTERN.fullmatch(slug)
assert 3 <= len(slug) <= 32
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_next_available_slug_returns_base_when_free(anyio_backend) -> None:
"""No collision → ``base`` is returned unchanged."""
seen: set[str] = set()
async def exists(s: str) -> bool:
return s in seen
assert await next_available_slug("foo", exists_check=exists) == "foo"
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_next_available_slug_walks_through_collisions(anyio_backend) -> None:
"""``foo``, ``foo-2`` taken → walker lands on ``foo-3``."""
seen = {"foo", "foo-2"}
async def exists(s: str) -> bool:
return s in seen
assert await next_available_slug("foo", exists_check=exists) == "foo-3"
@pytest.mark.anyio
@pytest.mark.parametrize("anyio_backend", ["asyncio"])
async def test_next_available_slug_truncates_base_to_fit_suffix(anyio_backend) -> None:
"""A 32-char base + ``-2`` would exceed the limit → base is shortened."""
base = "a" * 32 # exactly at the limit
seen = {base}
async def exists(s: str) -> bool:
return s in seen
result = await next_available_slug(base, exists_check=exists)
assert len(result) <= 32
assert result.endswith("-2")
assert _SLUG_PATTERN.fullmatch(result)
+45 -2
View File
@@ -766,6 +766,9 @@ dependencies = [
postgres = [
{ name = "deerflow-harness", extra = ["postgres"] },
]
postgres-test = [
{ name = "deerflow-harness", extra = ["postgres-test"] },
]
[package.dev-dependencies]
dev = [
@@ -780,6 +783,7 @@ requires-dist = [
{ name = "bcrypt", specifier = ">=4.0.0" },
{ name = "deerflow-harness", editable = "packages/harness" },
{ name = "deerflow-harness", extras = ["postgres"], marker = "extra == 'postgres'", editable = "packages/harness" },
{ name = "deerflow-harness", extras = ["postgres-test"], marker = "extra == 'postgres-test'", editable = "packages/harness" },
{ name = "dingtalk-stream", specifier = ">=0.24.3" },
{ name = "email-validator", specifier = ">=2.0.0" },
{ name = "fastapi", specifier = ">=0.115.0" },
@@ -795,7 +799,7 @@ requires-dist = [
{ name = "uvicorn", extras = ["standard"], specifier = ">=0.34.0" },
{ name = "wecom-aibot-python-sdk", specifier = ">=0.1.6" },
]
provides-extras = ["postgres"]
provides-extras = ["postgres", "postgres-test"]
[package.metadata.requires-dev]
dev = [
@@ -854,6 +858,13 @@ postgres = [
{ name = "psycopg", extra = ["binary"] },
{ name = "psycopg-pool" },
]
postgres-test = [
{ name = "asyncpg" },
{ name = "langgraph-checkpoint-postgres" },
{ name = "psycopg", extra = ["binary"] },
{ name = "psycopg-pool" },
{ name = "testcontainers" },
]
pymupdf = [
{ name = "pymupdf4llm" },
]
@@ -866,6 +877,7 @@ requires-dist = [
{ name = "alembic", specifier = ">=1.13" },
{ name = "asyncpg", marker = "extra == 'postgres'", specifier = ">=0.29" },
{ name = "ddgs", specifier = ">=9.10.0" },
{ name = "deerflow-harness", extras = ["postgres"], marker = "extra == 'postgres-test'", editable = "packages/harness" },
{ name = "dotenv", specifier = ">=0.9.9" },
{ name = "duckdb", specifier = ">=1.4.4" },
{ name = "exa-py", specifier = ">=1.0.0" },
@@ -897,9 +909,10 @@ requires-dist = [
{ name = "readabilipy", specifier = ">=0.3.0" },
{ name = "sqlalchemy", extras = ["asyncio"], specifier = ">=2.0,<3.0" },
{ name = "tavily-python", specifier = ">=0.7.17" },
{ name = "testcontainers", extras = ["postgres"], marker = "extra == 'postgres-test'", specifier = ">=4.0" },
{ name = "tiktoken", specifier = ">=0.8.0" },
]
provides-extras = ["ollama", "postgres", "pymupdf"]
provides-extras = ["ollama", "postgres", "postgres-test", "pymupdf"]
[[package]]
name = "defusedxml"
@@ -941,6 +954,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ba/5a/18ad964b0086c6e62e2e7500f7edc89e3faa45033c71c1893d34eed2b2de/dnspython-2.8.0-py3-none-any.whl", hash = "sha256:01d9bbc4a2d76bf0db7c1f729812ded6d912bd318d3b1cf81d30c0f845dbf3af", size = 331094, upload-time = "2025-09-07T18:57:58.071Z" },
]
[[package]]
name = "docker"
version = "7.1.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pywin32", marker = "sys_platform == 'win32'" },
{ name = "requests" },
{ name = "urllib3" },
]
sdist = { url = "https://files.pythonhosted.org/packages/91/9b/4a2ea29aeba62471211598dac5d96825bb49348fa07e906ea930394a83ce/docker-7.1.0.tar.gz", hash = "sha256:ad8c70e6e3f8926cb8a92619b832b4ea5299e2831c14284663184e200546fa6c", size = 117834, upload-time = "2024-05-23T11:13:57.216Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/e3/26/57c6fb270950d476074c087527a558ccb6f4436657314bfb6cdf484114c4/docker-7.1.0-py3-none-any.whl", hash = "sha256:c96b93b7f0a746f9e77d325bcfb87422a3d8bd4f03136ae8a85b37f1898d5fc0", size = 147774, upload-time = "2024-05-23T11:13:55.01Z" },
]
[[package]]
name = "docstring-parser"
version = "0.18.0"
@@ -4124,6 +4151,22 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/d7/c1/eb8f9debc45d3b7918a32ab756658a0904732f75e555402972246b0b8e71/tenacity-9.1.4-py3-none-any.whl", hash = "sha256:6095a360c919085f28c6527de529e76a06ad89b23659fa881ae0649b867a9d55", size = 28926, upload-time = "2026-02-07T10:45:32.24Z" },
]
[[package]]
name = "testcontainers"
version = "4.14.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "docker" },
{ name = "python-dotenv" },
{ name = "typing-extensions" },
{ name = "urllib3" },
{ name = "wrapt" },
]
sdist = { url = "https://files.pythonhosted.org/packages/ca/ac/a597c3a0e02b26cbed6dd07df68be1e57684766fd1c381dee9b170a99690/testcontainers-4.14.2.tar.gz", hash = "sha256:1340ccf16fe3acd9389a6c9e1d9ab21d9fe99a8afdf8165f89c3e69c1967d239", size = 166841, upload-time = "2026-03-18T05:19:16.696Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/13/2d/26b8b30067d94339afee62c3edc9b803a6eb9332f521ba77d8aaab5de873/testcontainers-4.14.2-py3-none-any.whl", hash = "sha256:0d0522c3cd8f8d9627cda41f7a6b51b639fa57bdc492923c045117933c668d68", size = 125712, upload-time = "2026-03-18T05:19:15.29Z" },
]
[[package]]
name = "tiktoken"
version = "0.12.0"
+11 -8
View File
@@ -15,7 +15,7 @@
# ============================================================================
# Bump this number when the config schema changes.
# Run `make config-upgrade` to merge new fields into your local config.yaml.
config_version: 9
config_version: 10
# ============================================================================
# Logging
@@ -878,16 +878,19 @@ skill_evolution:
# NOTE: When both `checkpointer` and `database` are configured,
# `checkpointer` takes precedence for LangGraph state persistence.
# If you use `database`, you can remove the `checkpointer` section.
# Stage 0+ default is postgres (parity with production, room for future RLS).
# Set DATABASE_URL in .env. SQLite is preserved as an offline-dev fallback —
# uncomment the SQLite block below and comment out the Postgres block to use it.
#
# Postgres (default):
database:
backend: postgres
postgres_url: $DATABASE_URL
# SQLite fallback (offline dev):
# database:
# backend: sqlite
# sqlite_dir: .deer-flow/data
#
# database:
# backend: postgres
# postgres_url: $DATABASE_URL
database:
backend: sqlite
sqlite_dir: .deer-flow/data
# ============================================================================
# Run Events Configuration
+30
View File
@@ -2,6 +2,7 @@
# Usage: docker-compose -f docker-compose-dev.yaml up --build
#
# Services:
# - postgres: PostgreSQL 16 (port 5432) — Stage 0+ default DB backend
# - nginx: Reverse proxy (port 2026)
# - frontend: Frontend Next.js dev server (port 3000)
# - gateway: Backend Gateway API + agent runtime (port 8001)
@@ -13,6 +14,30 @@
# Access: http://localhost:2026
services:
# ── Database (Stage 0+ default backend) ────────────────────────────────
# Postgres for local dev. Production may point DATABASE_URL at a remote RDS;
# this service can then be omitted via `docker compose --profile <other>`.
postgres:
image: postgres:16-alpine
container_name: deer-flow-postgres
environment:
POSTGRES_USER: ${POSTGRES_USER:-deerflow}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:-deerflow_dev}
POSTGRES_DB: ${POSTGRES_DB:-deerflow}
ports:
- "${POSTGRES_PORT:-5432}:5432"
volumes:
- postgres-data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER:-deerflow} -d ${POSTGRES_DB:-deerflow}"]
interval: 5s
timeout: 3s
retries: 5
start_period: 10s
networks:
- deer-flow-dev
restart: unless-stopped
# ── Sandbox Provisioner ────────────────────────────────────────────────
# Manages per-sandbox Pod + Service lifecycle in the host Kubernetes
# cluster via the K8s API.
@@ -125,6 +150,9 @@ services:
UV_IMAGE: ${UV_IMAGE:-ghcr.io/astral-sh/uv:0.7.20}
UV_INDEX_URL: ${UV_INDEX_URL:-https://pypi.org/simple}
container_name: deer-flow-gateway
depends_on:
postgres:
condition: service_healthy
command: sh -c "{ cd backend && (uv sync || (echo '[startup] uv sync failed; recreating .venv and retrying once' && uv venv --allow-existing .venv && uv sync)) && PYTHONPATH=. uv run uvicorn app.gateway.app:app --host 0.0.0.0 --port 8001 --reload --reload-include='*.yaml .env'; } > /app/logs/gateway.log 2>&1"
volumes:
- ../backend/:/app/backend/
@@ -178,6 +206,8 @@ volumes:
# image build are not shadowed by the host backend/ directory mount.
gateway-venv:
gateway-uv-cache:
# Persist Postgres data across container restarts.
postgres-data:
networks:
deer-flow-dev:
@@ -0,0 +1,188 @@
# DeerFlow 整体架构鸟瞰
按"自外向内、自顶向下"分层讲,并指出每一层对应的代码位置,方便后续深入。
## 一、进程与部署拓扑
DeerFlow 表面是 4 个端口,本质是 **3 个进程 + 1 个反向代理**
```
┌──────────────────────┐
浏览器 / IM ─────────▶│ nginx :2026 │ 统一入口
│ (含 CORS、SSE 透传) │
└──────────┬───────────┘
┌──────────────────┴──────────────────┐
│ │
▼ ▼
┌────────────────────┐ ┌──────────────────────────┐
│ Frontend (Next.js) │ │ Gateway (uvicorn) │
│ :3000 │ │ :8001 │
│ pnpm dev / preview │ │ ┌──────────────────────┐ │
└────────────────────┘ │ │ FastAPI 路由层 │ │
│ │ /api/models, /skills │ │
│ │ /threads, /runs ... │ │
│ ├──────────────────────┤ │
│ │ LangGraph Runtime │ │
│ │ (RunManager, │ │
│ │ StreamBridge, │ │
│ │ Checkpointer) │ │
│ ├──────────────────────┤ │
│ │ lead_agent 图 │ │
│ │ + 18 个中间件 │ │
│ │ + Sandbox / Tools │ │
│ └──────────────────────┘ │
└────────────┬─────────────┘
┌──────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
│ Sandbox │ │ MCP Servers │ │ LLM 提供商 │
│ Local / AIO Docker│ │ (stdio/sse/http) │ │ OpenAI/Anthropic │
│ 提供 bash/fs │ │ │ │ /vLLM/Codex CLI │
└──────────────────┘ └──────────────────┘ └──────────────────┘
```
关键事实(容易踩坑):
- **LangGraph 运行时不是独立进程**,而是嵌在 Gateway 这同一个 uvicorn 进程里。`scripts/serve.sh:225` 只起了一个后端进程:`uvicorn app.gateway.app:app`
- nginx 把 `/api/langgraph/*` 重写成 `/api/*` 再代理到 Gateway`docker/nginx/nginx.local.conf:49-73`),所以前端用的"标准 LangGraph SDK 协议"和 DeerFlow 自己的 REST 是同一个 8001 端口。
- 路由协议在 nginx 里特意为 SSE 关闭了缓冲(`proxy_buffering off; X-Accel-Buffering no`),否则流式响应会被吃掉。
## 二、后端代码二分:Harness vs App
后端最重要的边界,**决定新写代码该放哪里**。
```
backend/
├── packages/harness/deerflow/ ← 可发布的智能体框架(import: deerflow.*
│ ├── agents/ lead_agent + memory + middlewares + ThreadState
│ ├── runtime/ checkpointer / runs / stream_bridge / events / store
│ ├── sandbox/ Sandbox 抽象 + local 实现 + 文件/bash 工具
│ ├── subagents/ 子代理注册表 + 后台执行池
│ ├── tools/ 内建工具(present_files / ask_clarification / view_image
│ ├── mcp/ MultiServerMCPClient + 缓存 + OAuth
│ ├── skills/ SKILL.md 加载、工具白名单
│ ├── models/ 模型工厂、vLLM/Codex/Claude 自定义 provider
│ ├── community/ tavily / jina / firecrawl / aio_sandbox(可选实现)
│ ├── memory/ 长期记忆(事实抽取、debounce 队列)
│ ├── persistence/ SQLAlchemy 模型(用户、运行、事件、反馈)
│ ├── guardrails/ 工具调用前置鉴权(可插拔 provider)
│ ├── tracing/ LangSmith / Langfuse callback
│ ├── reflection/ "module:variable" 字符串 → 实例(配置驱动的关键)
│ ├── uploads/ 上传文件转换 markdown
│ └── client.py DeerFlowClient(嵌入式 Python 客户端)
└── app/ ← 应用代码(import: app.*
├── gateway/
│ ├── app.py FastAPI 入口 + lifespan
│ ├── auth_middleware.py 会话/Token 鉴权
│ ├── csrf_middleware.py 双重 cookie CSRF
│ ├── langgraph_auth.py 注入到 langgraph.json 的鉴权钩子
│ └── routers/ ↓ 下表
└── channels/ IMSlack/Telegram/Feishu/DingTalk/微信/企微)
```
**铁律:app 可以 import deerflowdeerflow 不能 import app**CI 用 `tests/test_harness_boundary.py` 强制)。这意味着 harness 必须自给自足——任何"agent 运行时需要的能力"都要在 harness 里完成抽象,app 层只做 HTTP/IM 适配。
## 三、Gateway 路由总览
`backend/app/gateway/routers/` 14 个路由文件,分三类职责:
| 类别 | 路由 | 干什么 |
|---|---|---|
| **配置/资源管理** | `models` `skills` `mcp` `memory` `agents` | 列出/启停 LLM 模型、技能、MCP、自定义 agent |
| **会话/数据** | `threads` `uploads` `artifacts` `suggestions` | 管理线程、上传文件、产物下载、追问建议 |
| **运行(核心)** | `thread_runs` `runs` `feedback` `assistants_compat` | 创建运行、SSE 流、消息分页、反馈打分、LangGraph 兼容协议 |
| **横切** | `auth` `channels` | 用户登录注册、IM 渠道状态 |
`assistants_compat.py` 是关键:它把前端用的 LangGraph SDK 协议(`POST /threads/{id}/runs/stream``messages-tuple` 流模式等)翻译成 DeerFlow 内部的 `RunManager` 调用——这就是 nginx 那条 `/api/langgraph/*` 重写规则的接收端。
## 四、一次对话的完整生命周期
把上面所有零件串起来——用户在前端输入一句话,会发生这些事:
```
1. 前端 useThreadStream hook
└─▶ LangGraph SDK 调用 POST /api/langgraph/threads/{id}/runs/stream
(stream_mode=["values","messages-tuple","custom"])
2. nginx 重写 → /api/threads/{id}/runs/stream → Gateway
3. Gateway thread_runs 路由
├─▶ AuthMiddleware 解析 session → user_id 注入到 user_contextcontextvar
├─▶ CSRFMiddleware 校验
└─▶ runtime.RunManager 创建 Run → 落库(runs / run_events 表)
4. RunManager 调用 lead_agent 图(langgraph.json: deerflow.agents:make_lead_agent
├─▶ 解析 configurable: model_name / thinking_enabled / is_plan_mode / subagent_enabled
├─▶ create_chat_model() 实例化 LLMreflection 从 "module:Class" 字符串实例化)
└─▶ create_agent(model, tools, middlewares, state_schema=ThreadState)
5. 18 个中间件按顺序拦截每一轮 model→tool→model
ThreadDataMiddleware 创建 .deer-flow/users/{uid}/threads/{tid}/...
UploadsMiddleware 注入新上传文件
SandboxMiddleware acquire 沙箱,state.sandbox_id 写入
DanglingToolCall 修复中断的 tool_call 序列
LLMErrorHandling LLM 报错降级
Guardrail 工具调用前鉴权(可选)
SandboxAudit 记录 bash/fs 操作
ToolErrorHandling tool 异常 → ToolMessage 不中断
Summarization token 接近上限时压缩历史
TodoList plan_mode 才挂
TokenUsage 累计 token
Title 首轮后自动起标题
Memory 队列异步抽取记忆
ViewImage 视觉模型注入 base64
DeferredToolFilter 需要时才暴露 tool schema
SubagentLimit 限制 task 并发到 3
LoopDetection 检测重复工具循环
Clarification ask_clarification 触发 interrupt(END)
6. Tools 由 get_available_tools() 拼装:
├─ Sandbox 工具:bash / ls / read_file / write_file / str_replace
├─ 内建工具:present_files / ask_clarification / view_image / setup_agent
├─ MCP 工具:从 extensions_config.json 启用的 server 拉取
├─ Community 工具:tavily / jina / firecrawl / image_search(按 config.yaml
└─ task 工具(可选):派遣 subagent
7. StreamBridge 把图执行的事件流转换成 SSE:
- "values" 完整状态快照
- "messages-tuple" 增量 token / 工具调用 / 工具返回
- "custom" StreamWriter 自定义事件
- "end" 收尾,附 token usage
8. 前端 LangGraph SDK 接 SSE,按 message id 累加 delta,更新 UI
9. 运行结束后,MemoryMiddleware 后台 30s debounce 抽取记忆事实写入
.deer-flow/users/{uid}/memory.json
```
## 五、状态与持久化的几条线
DeerFlow 的状态被有意拆成"快/慢/历史"三层,因为它要同时支持长会话、跨进程恢复、文件级产物:
| 状态 | 位置 | 谁写 |
|---|---|---|
| **会话状态(messages, todos, artifacts** | LangGraph checkpointer(内置 SQLite/PG,路径在 `runtime/checkpointer/async_provider.py` | 每个 step 自动 |
| **运行元数据/事件流** | `persistence/` 下 SQLAlchemy 模型(`runs``run_events``feedback``threads_meta` | RunManager + StreamBridge |
| **每用户每线程文件** | `.deer-flow/users/{uid}/threads/{tid}/user-data/{workspace,uploads,outputs}` | ThreadDataMiddleware + 沙箱工具 |
| **长期记忆** | `.deer-flow/users/{uid}/memory.json`(可叠加 per-agent | MemoryMiddleware(异步) |
| **配置** | `config.yaml`(模型、工具、沙箱、记忆…) + `extensions_config.json`MCP、技能开关) | `make setup` 或 Gateway PUT |
agent 看到的永远是 **虚拟路径** `/mnt/user-data/...``/mnt/skills/...`,由 `sandbox/tools.py``replace_virtual_path()` 翻译成上面物理路径。这层抽象让"本地沙箱"和"Docker 沙箱"对 agent 完全透明。
## 六、前端架构(一行总结)
`frontend/src/core/threads/hooks.ts` 里的 `useThreadStream` / `useSubmitThread` / `useThreads` 是整个前端的"主动脉"——它们包了 LangGraph SDK 单例(`core/api/`),所有 UI 组件订阅 thread 状态做渲染。Server Components 默认,需要交互的才 `"use client"``core/` 下其它子目录(artifacts/skills/mcp/memory/settings)都是为这条主动脉提供周边能力。
## 七、一图记住"它在做什么"
DeerFlow 本质上是一个 **"LangGraph 智能体 + 18 段切面 + 沙箱 + 记忆"** 的组合:
- **LangGraph** 提供图执行、checkpoint、stream 协议
- **18 个中间件** 是 DeerFlow 自己加的"切面层",每个解决一个具体的健壮性/能力问题(错误恢复、上下文压缩、记忆、子代理限流……)
- **沙箱+技能+MCP+工具** 是 agent 的"手脚"
- **Gateway + IM Channels + 嵌入式 Client** 是同一个 agent 的三种暴露方式(HTTP/聊天/Python 直调)
> 想继续往里钻的话,建议下一步选三个之一:(a)走读 lead_agent + 中间件链,理解 agent 一轮 think/act 的完整代码路径;(b)走读 sandbox + tools,理解虚拟路径和工具拼装;(c)走读 runtime + StreamBridge,理解 SSE 协议怎么映射回 LangGraph SDK。
@@ -0,0 +1,239 @@
# ADR-001 · 数据隔离模型
| 项目 | 内容 |
|---|---|
| 状态 | 草稿(Draft · 2026-05-09 据 spike 结果修订 §4.1.1 / §4.1.2 / §4.2 |
| 决策日期 | TBD |
| 决策者 | CTO + 架构 + 后端 lead |
| 关联 ADR | ADR-004 租户层级、ADR-005 存储拓扑、ADR-006 运行时与渠道 |
| 关联 spike / 审计 | [adr-vs-code-audit](./adr-vs-code-audit.zh-CN.md) · [langgraph-postgres spike](./adr-spike-langgraph-postgres.zh-CN.md) |
| 代码命名 | 本 ADR 写 `tenant_id`,落代码统一读作 `workspace_id`(详 [workspace-schema-design §1](./workspace-schema-design.zh-CN.md#1-命名约定--workspace-vs-tenant) |
---
## 1. 背景
DeerFlow 当前是 **"多用户单租户"** 模型:所有用户的会话、运行、记忆、产物都共用同一套表,仓储层用 `user_id` WHERE 过滤做个人空间隔离(`runtime/user_context.py:138-167`)。这套机制设计良好——repository 用 ContextVar + `AUTO` 哨兵自动注入当前用户,下层不依赖上层(harness 不能 import app)。
多租户化需要在 `user_id` 之上再加一层 `tenant_id`。问题是:**用什么物理隔离强度?**
三种主流方案:
| 维度 | 行级(tenant_id WHERE | per-tenant schema | per-tenant DB |
|---|---|---|---|
| 实现成本 | 低 | 中 | 高 |
| 跨租户 bug 爆炸半径 | 高 | 中 | 极低 |
| 备份/恢复粒度 | 全量 | 按 schema | 按 DB |
| 合规友好度(SOC2/HIPAA | 一般 | 好 | 最好 |
| 跨租户分析查询 | 容易 | 中 | 难 |
| 升级 schema | 一次完成 | 要遍历所有 schema | 要遍历所有 DB |
| 适用客户规模 | <10k 租户 | 10k100 大客户 | <100 大客户 |
| 运维复杂度 | 低 | 中 | 高 |
---
## 2. 决策
**采用 行级 `tenant_id` + Postgres Row-Level SecurityRLS** 作为双保险。
理由:
1. **DeerFlow 仓储层现状几乎平行扩展**——已经有 `resolve_user_id()` 哨兵模式,把 `tenant_id` 按同样模式补一遍,改造面集中、风险可控。
2. **Postgres RLS 是 DB 层兜底**——即使应用层有 bug 漏写 `WHERE tenant_id = ...`DB 也会强制过滤,第二道防线。
3. **覆盖目标客户规模**:B2B 中小客户为主、租户数 1k–10k,行级方案足够。
4. **不放弃跨租户分析能力**:平台需要做用量统计、监控、健康检查,单库行级最方便。
---
## 3. 备选方案与拒绝理由
### A. per-tenant schema(同库不同 schema
**拒绝。** 看似比行级更隔离,实际坑很多:
- **schema 数量爆炸**1000 个租户 = 1000 个 schema × 每张表,pg_class 体积膨胀,连接池里 search_path 切换有性能抖动
- **schema migration 痛苦**:每发布一次 schema 改动要遍历所有 schema 跑 migration,失败回滚极复杂
- **跨租户查询难**:要写 `UNION ALL` 跨所有 schema,运营仪表盘几乎无法实现
- **依然需要应用层过滤**:连接进哪个 schema 仍由应用层决定,没真正消除"应用层 bug 跨租户"
### B. per-tenant database(独立物理库)
**拒绝(默认场景)。** 隔离最强但成本极高:
- **运维负担**1000 个 DB = 1000 套备份/恢复/监控/连接池
- **冷启动延迟**:每个租户新建 DB 时间从秒级飙到分钟级
- **跨租户操作不可能**:平台级查询、聚合、迁移全部失效
- **连接池复杂度爆炸**:每租户独立连接池或者共用动态切库,都是噩梦
**仅在两种情况切换到此方案:** ① 拿到强合规客户(金融/医疗/政府),合同里写明物理数据隔离;② 客户付费足够覆盖每租户独立 DB 的运维成本(典型企业级订阅)。
---
## 4. 落地影响
### 4.1 表结构改造
所有业务表加 `tenant_id` 列 + 复合索引(`tenant_id` 作为前导列):
```sql
ALTER TABLE threads_meta ADD COLUMN tenant_id UUID NOT NULL;
CREATE INDEX idx_threads_meta_tenant_user ON threads_meta (tenant_id, user_id, updated_at DESC);
ALTER TABLE runs ADD COLUMN tenant_id UUID NOT NULL;
CREATE INDEX idx_runs_tenant_created ON runs (tenant_id, created_at DESC);
ALTER TABLE run_events ADD COLUMN tenant_id UUID NOT NULL;
CREATE INDEX idx_run_events_tenant_run ON run_events (tenant_id, run_id, seq);
ALTER TABLE feedback ADD COLUMN tenant_id UUID NOT NULL;
CREATE INDEX idx_feedback_tenant_run ON feedback (tenant_id, run_id);
-- ADR-005 引入的新表也要带 tenant_id(建表时就有)
-- agent_configs, memory_facts, memory_context,
-- tenant_skill_state, tenant_mcp_configs, tenant_secrets, tenant_quotas
```
**关键索引原则**:每个 `tenant_id` 都必须是复合索引的**第一列**——RLS policy 走的就是这条路径,前导列错了 RLS 会全表扫。
#### 4.1.1 LangGraph 自有表(checkpoints / checkpoint_writes / checkpoint_blobs / checkpoint_migrations
`runtime/checkpointer/async_provider.py` 用的是 LangGraph 内置 `AsyncPostgresSaver`**表结构不在 DeerFlow 控制下**。原稿讨论过两条路(subquery RLS / 列升级),spike[adr-spike-langgraph-postgres](./adr-spike-langgraph-postgres.zh-CN.md))验证后我们改用 **两层隔离模型**
| 表归属 | 隔离机制 | 防线性质 |
|---|---|---|
| **DeerFlow 自有表**threads_meta、runs、run_events、feedback、users、tenant_* | RLS + `SET LOCAL app.tenant_id` via SQLAlchemy sessionDeerFlow 完全控制 conn pool | DB 强约束 |
| **LangGraph checkpoint 表** | **应用层强校验**——入口路由在调 LangGraph 前必查 `threads_meta` 上的 `(tenant_id, thread_id)` 归属 | 应用层强约束 + 表 unique constraint 兜底 |
**为何对 LangGraph 表放弃 RLS**
- `langgraph-checkpoint-postgres==3.0.5` **不存在 `connection_factory` 参数**(spike §2 实测);其连接池注入路径只有 `__init__(conn=AsyncConnectionPool)` 这一个口子,且 `psycopg_pool` 自带的 `configure` callback 只在物理连接首次创建时跑——拿不到运行期 ContextVar 里的 tenant_id
- 子类化 `AsyncConnectionPool` 重写 `getconn` 注入 `SET app.tenant_id`/`RESET` 是可行的 hack,但侵入 psycopg-pool 内部,库升级风险高(spike §3.1)
- 给 LangGraph 表 ALTER 加 `tenant_id` 列同样不可取——LangGraph 用 `MIGRATIONS` 数组管理 schema,每次升级都要 diff 防漏(spike §3.4
**LangGraph 表的安全模型**(接受的 trade-off):
- 安全等级从"DB 强约束"降级为"应用层强约束 + 表 unique constraint"
- 强约束点是 **`threads_meta` 表上的 `UNIQUE (tenant_id, thread_id)` 复合索引** + 入口路由的强校验:任何代码路径要写 LangGraph 表前必须先在 `threads_meta` 找到对应行,且行的 `tenant_id` 与当前 ContextVar 一致
- CI 加 boundary 测试,禁止任何路径绕过 `threads.py` / `thread_runs.py` 直连 LangGraph saver(包括 LangGraph Studio 必须走相同入口或显式审批)
- 平台 admin 路径走 `BYPASSRLS` role 时同样必须经过应用层 audit,不直接跳过 thread 归属检查
**未来可升级路径**(不阻塞 phase-0):若上游接受 PR 加入 `connection_factory`,可平滑切回"DeerFlow 表 + LangGraph 表统一 RLS"模型。
#### 4.1.2 第一道防线:thread_id ↔ tenant_id 校验
LangGraph 调用入口在 **`app/gateway/routers/threads.py`**thread CRUD)和 **`app/gateway/routers/thread_runs.py`**(run 创建/恢复/事件流)。这两个路由在调用 LangGraph 之前**必须先用 `threads_meta` 校验 `(tenant_id, thread_id)` 归属**
- **创建路径**:先在 `threads_meta` 写入 `(tenant_id=current, thread_id, user_id=current)`,依赖 `UNIQUE (tenant_id, thread_id)` 防重;再调 LangGraph 创建对应 thread
- **读/写路径**:先用 `(current_tenant_id, requested_thread_id)` SELECT `threads_meta`,未命中即 404;命中后才允许调 LangGraph
> 注:原稿写"在 `AssistantsCompat` 路由强制校验"是错的——`assistants_compat.py:1-50` 只服务 `assistants.search/get` 静态 stub**不**触达 thread 入口(审计报告 §ADR-001 已修正)。
应用层校验是第一道防线、`UNIQUE` 约束是 DB 层兜底——任何对 LangGraph 表的访问都经过这一关。
### 4.2 RLS policy 模板
所有带 `tenant_id` 的表都加同样形态的 policy
```sql
ALTER TABLE threads_meta ENABLE ROW LEVEL SECURITY;
ALTER TABLE threads_meta FORCE ROW LEVEL SECURITY; -- 即使表所有者也走 policy
CREATE POLICY tenant_isolation ON threads_meta
USING (tenant_id = current_setting('app.tenant_id', true)::uuid)
WITH CHECK (tenant_id = current_setting('app.tenant_id', true)::uuid);
```
应用层在每次拿到连接时,先 `SET LOCAL app.tenant_id = '<uuid>'`
```python
# packages/harness/deerflow/persistence/engine.py
async def _set_session_tenant(session: AsyncSession, tenant_id: str) -> None:
"""Bind session to tenant; RLS policy enforces filter."""
await session.execute(text("SET LOCAL app.tenant_id = :tid"), {"tid": tenant_id})
```
每个仓储方法的开头自动调用,从 ContextVar 取 tenant_id(仿照现有 `resolve_user_id` 模式)。
**LangGraph 自有连接池不参与 SET LOCAL**DeerFlow 仓储和 LangGraph checkpointer 是**两套连接池**——前者是 SQLAlchemy `AsyncSession`DeerFlow 控制),后者是 LangGraph 自己持有的 psycopg 池(DeerFlow 不可控)。按 §4.1.1 的两层模型:
- **DeerFlow 自有表**:上面 `_set_session_tenant` helper 在每次仓储调用前注入 `SET LOCAL`RLS 兜底
- **LangGraph 表****不注入 `SET LOCAL`**——LangGraph 表上不启用 RLS,租户隔离靠应用层强校验(§4.1.2)实现。`AsyncPostgresSaver` 仍按现状用 `from_conn_string`,无侵入
> 原稿设想的"自定义 `connection_factory` 注入到 saver 构造"在 `langgraph-checkpoint-postgres==3.0.5` 不可行——库不存在该参数([spike](./adr-spike-langgraph-postgres.zh-CN.md) §2.2/2.4)。详细备选方案与拒绝理由见 spike §3。
**关键约束**:所有"会查 DeerFlow 自有表"的代码路径都必须保证调用栈上已注入 `app.tenant_id`,否则 RLS 会把整个会话过滤成空集。CI 加冒烟测试确认这点。LangGraph 表上的访问则必须经过 §4.1.2 的入口校验。
### 4.3 ContextVar 扩展
`runtime/user_context.py` 旁边加 `tenant_context.py`
```python
_current_tenant: Final[ContextVar[CurrentTenant | None]] = ContextVar("deerflow_current_tenant", default=None)
class _AutoSentinel: ... # 同 user_id 模式
AUTO: Final[_AutoSentinel] = _AutoSentinel()
def resolve_tenant_id(value, *, method_name) -> str:
"""与 resolve_user_id 同款三态:AUTO / 显式 str / 显式 None。
SaaS 模式下 None 是 forbidden(除非显式 admin override)。"""
```
`AuthMiddleware` 在解析完 JWT 后两个 ContextVar 同时注入。
### 4.4 SQLite → Postgres 迁移
**SQLite 不支持 RLS**,多租户上线必须切 Postgres。迁移路径:
1. 第 0 阶段:在 dev 环境同时跑 SQLite 和 Postgres,用 `aiosqlite` / `asyncpg` 双驱动
2. 第 1 阶段:生产切 Postgres,老数据用 `pg_loader` 导入;DeerFlow 现有的 `Base.metadata.create_all()` 直接接 Postgres
3. 老用户的 `user_id` 在没有 tenant_id 时归到一个"legacy_tenant",迁移脚本同步把 `tenant_id` 回填
### 4.5 跨租户操作(平台后台)
平台 admin / 运维需要跨租户查询时,不能简单"绕过 RLS"——而是用一个**专用 role** 配 `BYPASSRLS`,只给受限运维账号使用,操作审计入库:
```sql
CREATE ROLE deerflow_admin BYPASSRLS;
-- 应用层 admin 路由用这个 role 的连接池,并强制审计日志
```
**绝不允许应用主连接池有 `BYPASSRLS`。**
---
## 5. 风险与缓解
| 风险 | 缓解 |
|---|---|
| 应用层漏写 tenant_id WHERE | RLS 是兜底(DeerFlow 表);CI 加静态检查(detect SQL 不带 tenant_id |
| **LangGraph 表无 RLS,仅应用层强约束**(§4.1.1 trade-off | `threads_meta` `UNIQUE (tenant_id, thread_id)` 兜底;CI boundary 测试禁止绕过 `threads.py` / `thread_runs.py` 直连 saver;定期审计任何新增的 LangGraph 直连路径 |
| 索引前导列错了走全表扫 | DBA 评审所有 EXPLAIN;上线前压测 |
| `current_setting('app.tenant_id')` 没设导致 RLS 全过滤掉 | 应用层 fail-closed;监控空集查询率 |
| 跨租户分析需求多 | 提供受控的 admin role + 审计日志 |
| SQLite 开发 vs Postgres 生产差异 | 测试集成层用 testcontainers 跑 Postgres;不允许用 SQLite 跑 RLS 相关测试 |
| **当前不存在 Postgres 测试夹具基础设施**(审计报告 §ADR-001 highest-risk gap | phase-0 必须先落 testcontainers + RLS 冒烟测试,再做仓储改造;否则 RLS bug 进生产 |
| 单 DB 容量上限(>1TB 后维护困难) | 监控 DB 体积;超过阈值切 per-tenant DB(推翻方案) |
---
## 6. 推翻条件
切换到 **per-tenant DB** 当且仅当:
1. 拿到强合规客户(金融/医疗/政府),合同要求物理数据隔离
2. 单 DB 容量 / 写 TPS 触顶,垂直扩展不经济
3. 出现一次跨租户数据泄露事故,董事会要求最强隔离
---
## 7. 默认假设
| 项 | 默认 |
|---|---|
| 数据库 | PostgreSQL 16+ |
| RLS 启用 | 所有带 `tenant_id` 的业务表 |
| 主键 | UUID v7(时间排序) |
| 索引前导列 | `tenant_id` |
| Connection pool | 每应用进程 2050 connsPgBouncer transaction mode |
| 备份 | 每日全量 + WAL streaming,保留 30 天 |
| 跨租户查询 | 仅通过 `deerflow_admin` role + 审计 |
@@ -0,0 +1,372 @@
# ADR-002 · 沙箱隔离模型
| 项目 | 内容 |
|---|---|
| 状态 | 草稿(Draft |
| 决策日期 | TBD |
| 决策者 | 安全 + 架构 + SRE |
| 关联 ADR | ADR-001 数据隔离、ADR-005 存储拓扑 |
| 关联审计 | [adr-vs-code-audit](./adr-vs-code-audit.zh-CN.md) — 注意:现有 `AioSandboxProvider` 出网/资源/cosign 缺位;K8sSandboxProvider 几乎从零开工(实际工作量大于本 ADR §5 估算) |
| 代码命名 | 本 ADR 写 `tenant_id` / `tenant-{tenant_id}` namespace,落代码统一读作 `workspace_id` / `ws-{workspace_id}`(详 [workspace-schema-design §1](./workspace-schema-design.zh-CN.md#1-命名约定--workspace-vs-tenant) |
---
## 0. 概念前提
本 ADR 反复出现 **K8s Namespace + gVisor/Kata 运行时 + NetworkPolicy** 三件套——它们在不同层把租户的代码运行环境关起来,缺一不可。先用一段话讲清楚是什么、防什么、不防什么,再读后面的决策细节会顺很多。
### 0.1 K8s Namespace —— 资源/视图隔离
Kubernetes 的逻辑分区。一个集群里跑多租户,每租户分一个 namespace(如 `tenant-acme``tenant-bigco`),其中的 Pod / Service / Secret / ConfigMap 互相看不见。配套:
- **ResourceQuota**:限制 namespace 总用量(CPU、内存、Pod 数、存储)
- **LimitRange**:单 Pod 兜底(默认 request/limit、单 Pod 上限)
- **RBAC**:把租户管理员权限只绑到自己的 namespace
⚠️ **不是安全边界**。namespace 只让你"看不到",不是"碰不到"。两租户的 Pod 若都跑在默认 `runc` 上、共享同一个 Linux 内核,**任何一个内核 0day 都能让 A 容器逃逸到宿主,进而看到 B 容器**。所以需要下一层。
> 类比:办公楼里不同公司的门禁卡。同事进不了别人的工位,但墙不会自己变厚。
### 0.2 gVisor / Kata —— 内核级隔离
把容器从宿主内核上"再隔一层"。DeerFlow 沙箱要跑租户上传的任意 Python 代码,必须比 runc 更硬。
**gVisorGoogle**
- 用户态实现一个沙箱内核(Sentry),拦截容器所有 syscall 自己模拟,再用极少几个 syscall 跟真内核打交道
- 攻击面:先攻破 Sentry,再攻破真内核——多一道
- 代价:每个 syscall 中转,IO 密集型 workload 慢 ~10-30%
- 集成:Pod spec 写 `runtimeClassName: gvisor`
**Kata Containers**
- 给每个 Pod 起一个轻量虚拟机(QEMU 或 Firecracker 后端),容器跑在 VM 内独立内核里
- 隔离强度 ≈ 真 VM,启动几百毫秒
- 代价:每 Pod 多占 ~50-150 MB 内存、冷启动比 gVisor 慢一点
- 集成:`runtimeClassName: kata-qemu` / `kata-fc`
> 本 ADR 取舍:默认 **gVisor**(性价比平衡),监管/付费档位切 **Kata-Firecracker**(接近 VM 强度,单独定价)。配套 Cosign 镜像签名 + 只读根文件系统 + drop ALL caps 是纵深防御。
### 0.3 NetworkPolicy —— 出/入流量白名单
K8s 原生防火墙,按 namespace / Pod label 控制谁能跟谁通信。**默认拒绝 + 显式放行**是标准姿势:
```yaml
spec:
podSelector: {} # 命中 namespace 内所有 Pod
policyTypes: [Egress]
egress: [] # 空白名单 = 全部禁止
```
为什么对 DeerFlow 是关键——租户代码可能尝试访问:
| 目标 | 风险 |
|---|---|
| `169.254.169.254`(云元数据) | 偷 IAM 凭据、节点 token,直接拿下集群 |
| 平台内网 DB / Redis | 横向打到其他租户的数据 |
| 其他租户的 namespace IP | 跨租户监听/嗅探 |
| 互联网 C2 服务器 | 数据外泄、挖矿、僵尸网络 |
默认全部拒绝后,仅放行:DNS(CoreDNS)+ 出口走 **Egress Gateway**Envoy/Squid 做域名白名单,允许 `api.openai.com``pypi.org` 等,拒绝其余)。
> 执行靠 CNI 插件(Calico / Cilium)。Cilium 还支持 L7 策略(如"允许 GET /v1/chat/completions、禁 POST /admin"),是更强的备选。
### 0.4 三件套合起来看
```
租户的 Python 代码
┌────────────────────────────────────────┐
│ Pod (tenant-acme namespace) │ ← K8s Namespace:逻辑隔离 + 配额
│ ├─ runtimeClassName: gvisor │ ← gVisor:内核级攻击面隔离
│ ├─ readOnlyRootFilesystem │
│ └─ capabilities.drop: ["ALL"] │
└────────────────────────────────────────┘
│ egress
┌────────────────────────────────────────┐
│ NetworkPolicy: default-deny │ ← NetworkPolicy:网络层白名单
│ → 仅允许 Egress Gateway / DNS │
└────────────────────────────────────────┘
Egress Gateway(域名白名单)
```
- 没有 namespace:租户互相能看见对方的资源对象
- 没有 gVisor:一个内核 0day 全集群陪葬
- 没有 NetworkPolicy:租户代码 `curl 169.254.169.254` 就能拿走节点凭据
三层都套上,才是本 ADR 想要的"对抗任意租户代码"的最低防御姿势。
---
## 1. 背景
> **分期落地提示**:本 ADR 描述的 K8s + gVisor + NetworkPolicy 全套架构是**目标态**。按 [phased-rollout-by-scale](../02-rollout/phased-rollout-by-scale.zh-CN.md) 实际落地节奏:
> - **Stage 1**:仅做 §3 威胁模型的"出网默认禁 + cgroup CPU/memory 限额",落到现有 `AioSandboxProvider` 上(轻量补丁版)。**不上 K8s**。
> - **Stage 3**:才换 `K8sSandboxProvider`,引入 namespace + gVisor + NetworkPolicy + Pod Security Standard 全套(§5 全文落地)。
> - **Stage 4 / premium**:按合同切 Kata-Firecracker 或独立 nodepool。
>
> 读 §2~§9 时把它当作"Stage 3 完成态"Stage 1 落地时只摘 §3 出网/资源那两行就好。
沙箱是多租户里**爆炸半径最大**的组件:客户的 agent 可以跑任意 bash 命令、读写文件、调用 MCP 工具。如果隔离不够强,一个客户能:
- **读到其他客户的数据**(容器逃逸 / 共享卷误用)
- **薅云元数据**`curl http://169.254.169.254/...` 偷 IAM 凭证)
- **横向移动**(同 namespace 的其他容器、同节点的 hostNetwork
- **耗尽资源**(fork bomb、无限循环、磁盘填满)
DeerFlow 现状有两个 sandbox provider
| Provider | 强度 | 多租户可用 |
|---|---|---|
| `LocalSandboxProvider` | bash/fs **直接落主机**,零隔离 | ❌ 绝对不能用 |
| `AioSandboxProvider` | Docker 容器(社区实现,`packages/harness/deerflow/community/aio_sandbox/` | ⚠️ 当前配置不够 |
`AioSandboxProvider` 起点不错(每 thread 一个容器、虚拟路径翻译已有),但默认配置缺少多租户必需的几条隔离:默认出网未禁、CPU/内存 limits 未强制、根文件系统未只读、镜像未签名校验。
---
## 2. 决策
**采用 K8s + 强隔离运行时(gVisor 或 Kata Containers+ NetworkPolicy 默认禁出网 + per-tenant Namespace。**
| 层 | 作用 |
|---|---|
| **K8s Namespace per tenant** | 资源逻辑隔离;NetworkPolicy 起效边界 |
| **gVisor (runsc) 运行时** | 用户态系统调用拦截,容器逃逸到宿主难度大幅提升;性能损失 ~5-15%(多数 agent 任务可接受) |
| **NetworkPolicy 默认 DENY** | 出网白名单:只允许到 LLM endpoint、配置的 MCP servers、搜索 API |
| **ResourceQuota + LimitRange** | per-namespace CPU/内存上限;单 pod CPU/内存上限 |
| **Pod Security Standard: restricted** | 禁 root、禁 privileged、只读根文件系统、drop ALL caps |
| **emptyDir 临时卷** | 数据靠 ADR-005 同步对象存储,pod 销毁即清 |
| **镜像签名校验**Cosign | 启动 pod 前验证 sandbox 镜像签名,防供应链攻击 |
**Premium 客户档位**:在此之上再加 per-tenant **物理节点池** + **Firecracker microVM**kata-fc),把"租户 X 的沙箱永远不和别人共享物理节点"做到 SLA 里。
---
## 3. 威胁模型
| 攻击场景 | 共享 Docker(现状) | per-tenant K8s NS + gVisor(决策) | per-tenant Firecracker |
|---|---|---|---|
| 容器逃逸到宿主 | 全员沦陷 | 单租户沦陷(gVisor 大幅降低成功率) | 单租户沦陷(VM 边界,逃逸难度极高) |
| 容器间横向移动(同节点) | 可行 | NetworkPolicy 禁止 + namespace 隔离 | 不可能(独立 VM) |
| 出网到云元数据 169.254.169.254 | 默认可行 | NetworkPolicy 禁 + IMDSv2 强制 token | 同左 |
| 出网到内部服务(DB / 内网) | 可行 | NetworkPolicy 禁 + egress gateway 白名单 | 同左 |
| 侧信道(CPU 缓存 / Spectre | 可行 | 减弱(gVisor 隔离系统调用,但 CPU 共享仍有风险) | 显著减弱(独立 VM、独立 vCPU) |
| 资源耗尽(fork bomb / OOM | 影响同节点全部容器 | LimitRange 强制 cgroup;超限 OOMKill | VM 内独立调度 |
| 持久化攻击(写定时任务) | 可写主机 cron | 只读根文件系统 + ephemeral pod 重建即清 | 同左 |
| 提权 | 看 Docker 配置(默认 root | restricted PSS 禁 root、禁 capabilities | 同左 |
| 镜像被替换(供应链) | 不校验 | Cosign 验证签名 + admission controller 拦截 | 同左 |
---
## 4. 备选方案与拒绝理由
### A. 共享 Docker(保留 AioSandboxProvider 当前模式)
**拒绝。** 即使加固到极致,根本问题仍在:
- 所有租户共享 dockerd / containerd,一次容器逃逸 = 全员沦陷
- Docker 的 NetworkPolicy 等价物(user-defined network)粒度粗
- 资源限制依赖 cgroup v1/v2 配置一致,运维难统一
### B. per-tenant Firecracker microVM(默认)
**拒绝(作为默认)。** 隔离最强但成本最高:
- 冷启动比 K8s pod 慢 2-5×(500ms vs 几秒)
- 运维需要专门团队(kata-fc / cloud-hypervisor 都不是开箱即用)
- AWS / 阿里云的部分托管 K8s 不直接支持 Firecracker,需要自建 nodepool
**保留作为 premium 档位**:把 Firecracker 当付费 SLA 卖给监管类客户。
### C. 共享 K8s Namespace + 仅靠 NetworkPolicy + cgroup
**拒绝。** namespace 不分隔意味着:
- pod-to-pod 通信默认开放(NetworkPolicy 是白名单制,漏一条就全开)
- ServiceAccount 共享,越权读 secret 风险大
- ResourceQuota 是 namespace 级别,没法精细分配到租户
---
## 5. 落地影响
### 5.1 新建 `K8sSandboxProvider`
```python
# packages/harness/deerflow/sandbox/k8s/provider.py
class K8sSandboxProvider(SandboxProvider):
"""每 thread 一个 Pod,按 tenant 落到对应 Namespace。
生命周期:
acquire(thread_id) → 创建 PodgVisor runtime, restricted PSS, NetworkPolicy 已挂)
get(sandbox_id) → 返回与运行中 Pod 通信的客户端(kubectl exec / WebSocket
release(sandbox_id) → delete PodemptyDir 自动回收)
"""
```
替换 `LocalSandboxProvider`(开发/测试用)和 `AioSandboxProvider`(保留作为单机部署 fallback)。
### 5.2 K8s 资源(每租户 Namespace 一份)
```yaml
# tenant onboarding 时自动渲染、apply
apiVersion: v1
kind: Namespace
metadata:
name: tenant-{tenant_id}
labels:
pod-security.kubernetes.io/enforce: restricted
deerflow.io/tenant-id: {tenant_id}
---
apiVersion: v1
kind: ResourceQuota
metadata:
namespace: tenant-{tenant_id}
spec:
hard:
cpu: "16"
memory: 32Gi
pods: "20"
requests.storage: 100Gi
---
apiVersion: v1
kind: LimitRange
metadata:
namespace: tenant-{tenant_id}
spec:
limits:
- type: Container
default: { cpu: "500m", memory: 1Gi }
defaultRequest: { cpu: "100m", memory: 256Mi }
max: { cpu: "2", memory: 4Gi }
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all-egress
namespace: tenant-{tenant_id}
spec:
podSelector: {}
policyTypes: [Egress]
egress: # 仅允许下面这些
- to:
- namespaceSelector: { matchLabels: { name: kube-system } }
podSelector: { matchLabels: { k8s-app: kube-dns } }
ports: [{ protocol: UDP, port: 53 }]
- to: # 通过 egress gateway 出外网
- podSelector: { matchLabels: { app: deerflow-egress-gateway } }
```
### 5.3 Egress gateway
放一个集中的出口代理(envoy / squid)做:
- LLM endpoint 白名单(OpenAI / Anthropic / vLLM 内网)
- MCP server 白名单(per-tenant 启用列表)
- 搜索 API 白名单(Tavily / Jina / Brave / DuckDuckGo
- **黑名单**169.254.169.254cloud metadata)、10.0.0.0/8 / 172.16.0.0/12 / 192.168.0.0/16(内部网络,除非白名单)
- 全量审计(按租户记录每次出网)
### 5.4 Pod Spec 关键字段
```yaml
spec:
runtimeClassName: gvisor # 或 kata-fcpremium
automountServiceAccountToken: false # 沙箱不该有 SA token
securityContext:
runAsNonRoot: true
runAsUser: 65532
seccompProfile:
type: RuntimeDefault
containers:
- name: sandbox
image: registry.deerflow.io/sandbox:v2.3.4@sha256:... # 强制 digest pin
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities: { drop: [ALL] }
volumeMounts:
- { name: workspace, mountPath: /mnt/user-data }
- { name: skills, mountPath: /mnt/skills, readOnly: true }
volumes:
- { name: workspace, emptyDir: { sizeLimit: 10Gi } }
- { name: skills, emptyDir: { sizeLimit: 5Gi } } # 启动时从 S3 拉
```
### 5.5 镜像签名
- CI 用 cosign sign 给 sandbox 镜像签名
- K8s admission controllerpolicy-controller / Kyverno)启动 pod 前验证 cosign signature
- 拒绝任何未签名 / 签名不匹配的镜像
### 5.6 SandboxAuditMiddleware 增强
现有 `SandboxAuditMiddleware` 记录了工具调用,多租户必须:
- 每条审计带 `(tenant_id, user_id, thread_id, tool_name, args_hash)`
- 异步推到独立审计存储(不与业务库共用)
- 保留至少 90 天(合规要求常见值)
---
## 6. 性能影响
| 指标 | 估计 |
|---|---|
| Pod 冷启动(gVisor + 镜像 pre-warm | 1.5-3s |
| Pod 冷启动(Firecracker | 3-8s |
| Bash 命令延迟(vs 宿主) | gVisor +5-15%Firecracker +10-30% |
| 文件 I/O(小文件) | gVisor 显著慢(系统调用拦截);用 emptyDir tmpfs 缓解 |
| 网络(egress gateway | +1-3ms 单跳 |
**冷启动是最敏感的指标**——靠两条路径优化:
1. **Pod prewarm**:每个 namespace 维护 N 个空闲 pod 池(`PodReadinessProbe` 通过即可服用,按需绑定 thread)
2. **镜像层缓存**:每节点预拉镜像(DaemonSet image-puller
---
## 7. 风险与缓解
| 风险 | 缓解 |
|---|---|
| gVisor 与某些 syscall 不兼容(agent bash 跑不动某些工具) | sandbox 镜像里预装常用工具;CI 跑兼容性测试集 |
| 出网白名单维护负担 | Per-tenant MCP/搜索配置自动生成 NetworkPolicy;运维 admin UI 一键加 |
| Firecracker 运维复杂 | 仅作为 premium 档位,不强制全量上 |
| 节点 noisy neighborCPU 共享导致侧信道) | 高敏租户走专属 nodepooltaints/tolerations |
| Pod prewarm 池资源浪费 | 按租户活跃度动态调节池大小;闲置超过阈值缩到 0 |
| 镜像供应链攻击 | Cosign 强制 + SBOM + 漏洞扫描 |
---
## 8. 推翻条件
切换到 **per-tenant Firecracker(默认)** 当且仅当:
1. 实测 gVisor 在某条关键 syscall 上有不可绕过的兼容性问题(且 sandbox 镜像无法预装替代品)
2. 拿到合同要求"强物理隔离"的监管客户,付费档位要求覆盖运维成本
3. 出现一次容器逃逸 PoC 影响多租户
切换到 **共享 Docker(极端简化)** 当且仅当:
- 公司决定退回单租户产品形态——多租户上线后基本不应回退
---
## 9. 默认假设
| 项 | 默认 |
|---|---|
| 集群 | EKS / ACK / GKEK8s 1.29+ |
| 沙箱 runtime | gVisor (runsc) |
| Premium runtime | Kata Containers + Firecracker |
| Pod 隔离粒度 | per-thread(不复用) |
| 冷启动 SLO | P50 < 2s, P99 < 5s |
| Pod CPU/Mem 上限 | 2 CPU / 4 GiB(单 pod |
| Namespace 配额 | 16 CPU / 32 GiB / 20 pods(按 plan 调) |
| Egress 白名单数量 | <30 个域名 / 租户 |
| 审计保留期 | 90 天热 + 1 年冷 |
| 镜像签名 | cosign + Kyverno 强制 |

Some files were not shown because too many files have changed in this diff Show More