Adds a single script that walks the 5 verification layers in order
and emits pass/fail counts at the end:
static — boundary scans + full pytest (baseline ≥ 3250 passed,
≤ 18 fails matching known caplog flake set) + ruff lint
paths — .deer-flow/users/ should be empty (or absent);
workspaces/{wid}/threads/{tid}/... layout in place;
exercises `make migrate-paths DRY_RUN=1`
rds — PG reachable; alembic at 0003; service_accounts +
api_keys + external_users tables present;
idx_api_keys_active partial index has
"WHERE revoked_at IS NULL" predicate; workspace_id is
NOT NULL on thread_meta / runs / feedback / run_events;
UNIQUE(workspace_id, thread_id) on thread_meta
(requires DATABASE_URL; skipped if unset)
runtime — curl /health on the Gateway (requires `make dev`;
skips downstream e2e if unreachable)
e2e — register two users via /api/auth/register, capture each
session's csrf_token, create one thread per user,
cross-access GET + DELETE both return 404 (per PR6
"404 not 403" contract), same-workspace GET returns 200
Each layer is independently runnable: `./verify_stage0.sh paths rds`.
With no args runs all five.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/test_pr8_metadata_registration.py: opens a fresh SQLite
engine via init_engine() and asserts inspect(conn).get_table_names()
contains service_accounts, api_keys, and external_users. Guards
against an ORM row class being added under deerflow/persistence/* but
accidentally left out of deerflow/persistence/models/__init__.py —
which would leave the table un-provisioned at startup and surface as
a confusing "no such table" later in Stage 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/external_user/{__init__,model}.py with
ExternalUserRow:
- id: UUID36 PK
- workspace_id: FK workspaces ON DELETE CASCADE (redundant with SA's
workspace_id but stored directly to speed workspace-scoped queries
that span multiple SAs)
- service_account_id: FK service_accounts ON DELETE CASCADE
- external_id: String(128) — caller-supplied X-External-User-Id
- display_name: String(128) nullable (admin UI only, not auth-relevant)
- metadata_json: JSON nullable=False default {} — plan tier / region /
custom tags
- created_at / last_seen_at (UTC)
- UniqueConstraint (service_account_id, external_id)
name=uq_external_users_sa_external — the same external_id may be
reused under a different SA, but is upsert-unique under a single SA
T8.5 tests:
- test_unique_service_account_id_plus_external_id: second row with
same (SA, external_id) raises IntegrityError
- test_cascade_on_service_account_delete: deleting parent SA removes
all external_users rows under it
Registered in deerflow/persistence/models/__init__.py — all three PR8
tables (service_accounts / api_keys / external_users) are now wired
into Base.metadata.create_all().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/api_key/{__init__,model}.py with ApiKeyRow:
- id: UUID36 PK
- service_account_id: FK service_accounts ON DELETE CASCADE
- key_prefix: String(16), UNIQUE — global uniqueness preserved even
after revoke so audit logs never reference an ambiguous prefix
- key_hash: String(128) sha-256 hex (plaintext returned only once at
create time)
- name / scopes / rate_limit_rpm / expires_at / last_used_at /
revoked_at — all nullable or default-providing
- created_at: UTC
- Index idx_api_keys_sa (service_account_id) — list keys for an SA
- Index idx_api_keys_active (key_prefix) WHERE revoked_at IS NULL —
partial index, dual-dialect via sqlite_where + postgresql_where,
shrinks the hot-path lookup index by excluding revoked keys
T8.4 tests:
- test_unique_key_prefix_enforced: column-level UNIQUE blocks two
rows from sharing key_prefix (active or revoked alike)
- test_active_index_declares_both_dialect_where_clauses: schema
introspection confirms idx_api_keys_active has both
`dialect_options.sqlite.where` and `dialect_options.postgresql.where`
set to `revoked_at IS NULL` — guards against accidental loss of the
dual-driver hint when the Index is edited later
- test_cascade_on_service_account_delete: deleting the parent SA
removes all api_keys rows
Registered in deerflow/persistence/models/__init__.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
T8.2 test_cascade_on_workspace_delete: seed user → workspace → SA,
delete workspace, assert SA row is gone (ondelete CASCADE on
workspace_id FK).
T8.3 test_restrict_on_created_by_user_delete: seed user → workspace
→ SA, then attempt DELETE FROM users WHERE id = created_by, assert
IntegrityError raises and the SA row survives (ondelete RESTRICT on
created_by FK).
SQLite enforces FKs because the engine's connect-listener turns on
`PRAGMA foreign_keys = ON` for every new connection — see engine.py
init_engine().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/service_account/{__init__,model}.py with
ServiceAccountRow:
- id: UUID36 PK
- workspace_id: FK workspaces ON DELETE CASCADE
- name: String(64)
- role: String(16) default "member" (Stage 0 lone value; Stage 2 RBAC)
- identity_mode: String(16) default "collapsed" — three states
"collapsed" / "external_passthrough" / "both"; Stage 1 API key
auth layer branches on this to decide whether each call writes
an external_users row
- status: String(16) default "active"
- created_by: FK users ON DELETE RESTRICT (must hand off / delete
SAs before removing their creator)
- created_at / updated_at (UTC, onupdate)
- Index idx_service_accounts_workspace (workspace_id, status)
Registers in deerflow/persistence/models/__init__.py so the engine's
side-effect import path picks it up for Base.metadata.create_all().
T8.1 test: insert_smoke writes a row through a session, reads it back,
asserts each column round-trips. SQLite ephemeral DB per test via
tmp_path, no Postgres required at this layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the existing "Boundary check" section with two new entries:
- tests/test_workspace_boundary.py — AST static scan for direct imports of
langgraph.checkpoint.* (and the third-party postgres/sqlite packages)
outside tests/boundary_allowlist.toml. Notes the TYPE_CHECKING exemption
and points readers to `app.gateway.deps.get_checkpointer` as the
intended path; documents the "append to allowlist in the same PR"
contract for new legitimate importers.
- tests/test_workspace_boundary_self.py — self-tests guarding the scanner
from silent-empty regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds backend/tests/test_workspace_boundary.py: walks every .py under
backend/ (excluding tests/, docs/, build artefacts), parses with ast,
flags any ImportFrom/Import that targets langgraph.checkpoint.*,
langgraph_checkpoint_postgres or langgraph_checkpoint_sqlite outside the
allowlist loaded from tests/boundary_allowlist.toml.
Skip rules:
- Imports inside `if TYPE_CHECKING:` (or `if typing.TYPE_CHECKING:`)
blocks are exempt: they do not enter runtime so cannot bypass the
boundary. Verified against deerflow/agents/factory.py which only uses
BaseCheckpointSaver as a parameter annotation.
The scanner walks parent pointers up from each import node to detect
the enclosing TYPE_CHECKING guard, rather than checking only direct
parents — handles nested guards correctly.
RED proof (empty allowlist): 14 violations across the 4 ground-truth
runtime importers (threads / async_provider / provider / worker).
GREEN (with this PR's allowlist): single test passes in ~2s.
The collector function `collect_runtime_checkpoint_imports` and
`scan_violations` are kept at module scope so T7.3's self-test can
exercise them on a synthetic temp file in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds backend/tests/boundary_allowlist.toml that names the four current
legitimate runtime importers of langgraph.checkpoint.*:
- app/gateway/routers/threads.py (empty_checkpoint for thread init)
- packages/harness/deerflow/runtime/checkpointer/async_provider.py
- packages/harness/deerflow/runtime/checkpointer/provider.py
- packages/harness/deerflow/runtime/runs/worker.py (empty_checkpoint)
The allowlist replaces plan's draft (thread_runs.py / gateway/app.py)
with the ground-truth grep: both of those reach the checkpointer through
`app.gateway.deps.get_checkpointer` DI, so they need not be listed.
factory.py imports BaseCheckpointSaver only under `if TYPE_CHECKING:` —
the scanner (next commit) exempts type-only imports automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flip the 4 business ORM models (ThreadMetaRow, RunRow, FeedbackRow,
RunEventRow) to ``workspace_id: Mapped[str]`` with ``nullable=False``.
PR5's alembic 0003 already enforces NOT NULL at the DB layer; this
aligns the ORM-driven ``create_all()`` path (dev / tests) with the same
invariant so a new install ends up at the post-0003 schema without
running alembic.
Test fallout absorbed:
- `tests/conftest.py` autouse seed now produces a fully consistent
pair: user row (default_workspace_id = test-workspace-autouse) plus
the workspace itself. The PR5 backfill script's "users without
default_workspace_id" query no longer picks the fixture up. Insert
order is user → workspace → UPDATE user, walking around the chicken-
and-egg FK between `workspaces.owner_id` and `users.default_workspace_id`.
- `tests/test_backfill_workspace_id.py` adds a file-scoped autouse
fixture that temporarily flips `column.nullable = True` for the four
business tables (production correctness comes from alembic 0003;
the script's own job is exactly to fill rows between 0002 and 0003
so its tests need that transient state to be representable). Its
`_init_engine` also deletes the autouse seed rows to match the
"fresh DB" model the tests assume.
- `test_thread_meta_workspace_filter::test_create_workspace_none_bypasses`
renamed to `test_create_workspace_none_rejected_by_orm` and asserts
the new IntegrityError on explicit None — write paths can no longer
bypass workspace scope.
- 5 `test_workspace_context` tests + the auth-middleware reset test
get `@pytest.mark.no_auto_workspace` so they keep testing the
unset-contextvar path.
- `test_workspace_repo::test_list_by_user_bypass_returns_all` switches
to membership assertions instead of strict equality since the
autouse fixture surfaces under `user_id=None`.
3214 passed, 30 skipped; the remaining 17 are the documented
pre-existing caplog ordering flakes (all pass in isolation).
`app/gateway/app.py:_check_path_migration_pending` runs at lifespan
startup and emits a single WARNING log when `{base_dir}/users/` still
has content — the operator's cue to run the new migration script. The
warning is best-effort (silent on permission errors) so it can never
escalate into a gateway boot failure; reads from the legacy tree keep
working via the `user_id` branch of `Paths.thread_dir` until the
migration runs.
The root Makefile gains a `migrate-paths` target that wraps
`scripts/migrate_paths_to_workspace.py`. Two opt-in env knobs:
- `DRY_RUN=1` — pass `--dry-run` (preview only, no writes)
- `DEFAULT_WORKSPACE=…` — claim un-assigned users under this workspace id
(defaults to `legacy_workspace` to align with PR5's orphan-row bucket).
`help` documents the new target.
3 tests cover the warning's three states (legacy dir with content, dir
missing, dir empty).
`scripts/migrate_paths_to_workspace.py` walks the PR4 per-user filesystem
layout and rewrites it under the PR6 per-workspace dimension:
- `{base}/users/{uid}/threads/{tid}/` → `{base}/workspaces/{wid}/threads/{tid}/`
- `{base}/users/{uid}/memory.json` → `{base}/workspaces/{wid}/users/{uid}/memory.json`
- `{base}/users/{uid}/agents/{name}/` → `{base}/workspaces/{wid}/users/{uid}/agents/{name}/`
`{wid}` is read from `users.default_workspace_id` in the local sqlite
DB; users without one fall back to `--default-workspace`, defaulting to
`legacy_workspace` (the same bucket PR5's row backfill uses for orphan
rows so the file-system and database alignments stay consistent).
Pre-existing destinations route the legacy copy to
`{base}/migration-conflicts/workspace-migration/...` for manual review;
empty source dirs are cleaned up. The script is idempotent and supports
`--dry-run` (writes nothing, still logs the full report).
9 tests cover: thread / memory / agent rewrite, dry-run no-write, the
fallback workspace path, conflict diversion, post-migration cleanup of
empty `users/` dirs, and the missing-DB / missing-table degraded
modes.
`ThreadDataMiddleware.before_agent` now reads
`get_effective_workspace_id()` and routes the per-thread directory tree
through `Paths.sandbox_*_dir(thread_id, workspace_id=...)`, producing
`{base_dir}/workspaces/{wid}/threads/{tid}/user-data/...`. The legacy
user-id-only layout is no longer written by this middleware; the
migration script in T6.12 will lift any pre-existing `users/{uid}/...`
trees into the new shape.
In no-auth dev mode the contextvar is empty so
`get_effective_workspace_id` returns `"default"` and writes land at
`workspaces/default/...` — the layout invariant ("threads always live
inside a workspace") holds without a real auth setup. `thread_data` now
also exposes `user_id` and `workspace_id` so downstream middlewares
(sandbox, memory, etc.) can read them without re-resolving the
contextvar themselves.
4 new tests cover: contextvar workspace → expected path,
`no_auto_workspace` falls back to `default`, eager mode creates the
right dirs, and the `get_config` fallback still routes through
workspace. Existing 4 thread_data middleware tests stay green.
`Paths` learns a new top-level filesystem dimension. Precedence in
descending order:
- workspace_id given (PR6+):
`{base_dir}/workspaces/{wid}/threads/{tid}/...`
`{base_dir}/workspaces/{wid}/users/{uid}/memory.json`
`{base_dir}/workspaces/{wid}/users/{uid}/agents/{name}/...`
- user_id only (legacy user-isolation): unchanged
- neither (very legacy): unchanged
API: every thread/sandbox/host/ACP helper now takes
`workspace_id: str | None = None` alongside the existing `user_id`,
plus `Paths.workspace_dir()` and `_validate_workspace_id()` for safe
id substitution. `resolve_virtual_path` likewise routes through the
new precedence. The migration script in T6.12 will walk the old
`users/{uid}/threads/...` tree and rewrite to the new shape.
15 new tests (workspace shape, validation, sandbox dirs, ACP,
user-memory/agents under workspace, traversal defence) pass; 46
existing path tests stay green.
End-to-end coverage for the workspace boundary:
- GET /api/threads/{tid} from workspace B → 404
- DELETE /api/threads/{tid} from workspace B → 404
- PATCH /api/threads/{tid} from workspace B → 404
- Positive control: same workspace → 200
Wires `MemoryThreadMetaStore` (real impl, not a mock) behind a
stub-authed FastAPI app. The check_access call inside
`@require_permission` returns False on cross-workspace and the
decorator converts to 404 — proving the boundary holds at the HTTP
boundary, not just the unit level. Run stream/wait are skipped here
(they spin a background worker); their guard goes through the same
decorator path so unit coverage in `test_require_permission_workspace`
is sufficient.
`_StubAuthMiddleware` gains `override_user_contextvar=True` so
cross-user / cross-workspace tests can drive both contextvars from the
stub. Default stays off — the autouse user fixture continues to own
the contextvar for legacy tests whose routes resolve filesystem paths
via `get_effective_user_id()`.
86 router / boundary tests stay green.
`create_thread` in `app/gateway/routers/threads.py` already relies on
the AUTO sentinel default for ``ThreadMetaStore.create``, so once T6.1
landed the workspace_id was implicitly carried from the contextvar.
This commit adds the integration coverage: TestClient + stub auth
middleware + ``MemoryThreadMetaStore`` round-trip showing the persisted
row carries the right ``workspace_id`` under two distinct workspaces.
No production code change — the route wiring was correct, but the
guarantee was previously only proven at the unit level.
The decorator path was already updated in T6.4 to read
`get_effective_workspace_id()` and forward it as the third positional
to `check_access`. T6.6 closes the loop with dedicated coverage:
- `_StubAuthMiddleware` and `make_authed_test_app` now accept an
optional `workspace_factory` so router tests can drive the active
workspace contextvar end-to-end through the FastAPI middleware stack.
- 5 new probe tests assert: cross-workspace → 404 (not 403), same
workspace → 200, the third positional reaching `check_access`
carries the contextvar id, no-context tests (marked
`no_auto_workspace`) fall back to "default", and read-style routes
(`require_existing=False`) thread workspace_id too.
75 existing router tests (artifacts / runs / threads / uploads /
suggestions) stay green.
`ThreadMetaStore.check_access(thread_id, user_id, workspace_id, *,
require_existing)` is now a three-positional method. Cross-workspace is
denied unconditionally — even when the row exists and `user_id` matches
— so the decorator layer can convert the False into a **404** and never
leak the existence of a thread across tenants. Inside the workspace, the
existing legacy semantics still hold (NULL `row.user_id` stays
"shared in workspace", `require_existing` still gates the missing-row
path against ghost-row re-targeting).
`@require_permission(owner_check=True)` in `app/gateway/authz.py` now
reads the active workspace from `get_effective_workspace_id()` (set by
PR4 AuthMiddleware; falls back to "default" in no-auth dev) and passes
it through. The existing 404-not-403 mapping is unchanged.
Existing positional callers in `test_thread_meta_repo.py` and the
permissive mock in `test_threads_router.py` were updated for the new
arity. 58 thread_meta / router / memory tests stay green; 90 auth /
uploads / suggestions tests stay green.
Every remaining ThreadMetaStore method gains a `workspace_id` keyword
mirroring `user_id`:
- `search()` adds WHERE workspace_id and (for the memory impl) folds it
into the BaseStore filter dict.
- `update_display_name` / `update_status` / `update_metadata` / `delete`
no-op if the row lives in a different workspace. The SQL helper
`_check_ownership()` was widened to do both checks in one pass.
- `MemoryThreadMetaStore._get_owned_record()` likewise takes both ids.
5 new isolation tests prove writes from workspace B against workspace A's
thread are silently dropped (no row mutation when the caller re-reads
from workspace A). 38 existing thread_meta / owner / memory-store tests
still pass.
`get()` accepts `workspace_id: str | None | _AutoSentinel = AUTO` and
moves the workspace check into the SQL WHERE so cross-workspace lookups
short-circuit without loading the row. `MemoryThreadMetaStore._get_owned_record`
gets the same filter for parity.
The user_id check stays as a post-load comparison (preserves the existing
shared-row semantics where row.user_id IS NULL means "everyone in this
workspace"). Cross-workspace always returns None, never the row.
3 new tests cover the three states: in-workspace get returns the row,
out-of-workspace get returns None even when user_id matches, explicit
workspace_id=None bypasses (migration / CLI). Existing 22 thread_meta
tests stay green.
`create()` now accepts `workspace_id: str | None | _AutoSentinel = AUTO`
on both the SQL and in-memory implementations (and the abstract base).
AUTO resolves via `resolve_workspace_id()` from the workspace contextvar
that PR4 AuthMiddleware sets; explicit None bypasses for migration paths;
explicit str overrides the contextvar.
Test infrastructure:
- conftest gains an autouse `_auto_workspace_context` fixture mirroring
the existing user fixture. Opt-out via `@pytest.mark.no_auto_workspace`.
- A SQLAlchemy `after_create` listener on `Base.metadata` seeds the
matching `test-workspace-autouse` + `test-user-autouse` rows whenever
`init_engine` runs `create_all()`, so the FK from threads_meta to
workspaces resolves. Alembic migration tests bypass create_all and are
unaffected, keeping real FK constraints under test.
- `test_thread_meta_workspace_filter.py` covers the three AUTO / explicit
/ None paths.
3 new tests pass; 115 existing thread_meta/run/feedback/run_event/owner
tests stay green.
1445043649