Owner/admin self-service CRUD for service accounts: POST create,
GET list, PATCH status; workspace-scoped with 404 existence hiding
for cross-workspace targets. Gated by require_workspace_admin.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a single script that walks the 5 verification layers in order
and emits pass/fail counts at the end:
static — boundary scans + full pytest (baseline ≥ 3250 passed,
≤ 18 fails matching known caplog flake set) + ruff lint
paths — .deer-flow/users/ should be empty (or absent);
workspaces/{wid}/threads/{tid}/... layout in place;
exercises `make migrate-paths DRY_RUN=1`
rds — PG reachable; alembic at 0003; service_accounts +
api_keys + external_users tables present;
idx_api_keys_active partial index has
"WHERE revoked_at IS NULL" predicate; workspace_id is
NOT NULL on thread_meta / runs / feedback / run_events;
UNIQUE(workspace_id, thread_id) on thread_meta
(requires DATABASE_URL; skipped if unset)
runtime — curl /health on the Gateway (requires `make dev`;
skips downstream e2e if unreachable)
e2e — register two users via /api/auth/register, capture each
session's csrf_token, create one thread per user,
cross-access GET + DELETE both return 404 (per PR6
"404 not 403" contract), same-workspace GET returns 200
Each layer is independently runnable: `./verify_stage0.sh paths rds`.
With no args runs all five.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/test_pr8_metadata_registration.py: opens a fresh SQLite
engine via init_engine() and asserts inspect(conn).get_table_names()
contains service_accounts, api_keys, and external_users. Guards
against an ORM row class being added under deerflow/persistence/* but
accidentally left out of deerflow/persistence/models/__init__.py —
which would leave the table un-provisioned at startup and surface as
a confusing "no such table" later in Stage 1.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/external_user/{__init__,model}.py with
ExternalUserRow:
- id: UUID36 PK
- workspace_id: FK workspaces ON DELETE CASCADE (redundant with SA's
workspace_id but stored directly to speed workspace-scoped queries
that span multiple SAs)
- service_account_id: FK service_accounts ON DELETE CASCADE
- external_id: String(128) — caller-supplied X-External-User-Id
- display_name: String(128) nullable (admin UI only, not auth-relevant)
- metadata_json: JSON nullable=False default {} — plan tier / region /
custom tags
- created_at / last_seen_at (UTC)
- UniqueConstraint (service_account_id, external_id)
name=uq_external_users_sa_external — the same external_id may be
reused under a different SA, but is upsert-unique under a single SA
T8.5 tests:
- test_unique_service_account_id_plus_external_id: second row with
same (SA, external_id) raises IntegrityError
- test_cascade_on_service_account_delete: deleting parent SA removes
all external_users rows under it
Registered in deerflow/persistence/models/__init__.py — all three PR8
tables (service_accounts / api_keys / external_users) are now wired
into Base.metadata.create_all().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/api_key/{__init__,model}.py with ApiKeyRow:
- id: UUID36 PK
- service_account_id: FK service_accounts ON DELETE CASCADE
- key_prefix: String(16), UNIQUE — global uniqueness preserved even
after revoke so audit logs never reference an ambiguous prefix
- key_hash: String(128) sha-256 hex (plaintext returned only once at
create time)
- name / scopes / rate_limit_rpm / expires_at / last_used_at /
revoked_at — all nullable or default-providing
- created_at: UTC
- Index idx_api_keys_sa (service_account_id) — list keys for an SA
- Index idx_api_keys_active (key_prefix) WHERE revoked_at IS NULL —
partial index, dual-dialect via sqlite_where + postgresql_where,
shrinks the hot-path lookup index by excluding revoked keys
T8.4 tests:
- test_unique_key_prefix_enforced: column-level UNIQUE blocks two
rows from sharing key_prefix (active or revoked alike)
- test_active_index_declares_both_dialect_where_clauses: schema
introspection confirms idx_api_keys_active has both
`dialect_options.sqlite.where` and `dialect_options.postgresql.where`
set to `revoked_at IS NULL` — guards against accidental loss of the
dual-driver hint when the Index is edited later
- test_cascade_on_service_account_delete: deleting the parent SA
removes all api_keys rows
Registered in deerflow/persistence/models/__init__.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
T8.2 test_cascade_on_workspace_delete: seed user → workspace → SA,
delete workspace, assert SA row is gone (ondelete CASCADE on
workspace_id FK).
T8.3 test_restrict_on_created_by_user_delete: seed user → workspace
→ SA, then attempt DELETE FROM users WHERE id = created_by, assert
IntegrityError raises and the SA row survives (ondelete RESTRICT on
created_by FK).
SQLite enforces FKs because the engine's connect-listener turns on
`PRAGMA foreign_keys = ON` for every new connection — see engine.py
init_engine().
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds deerflow/persistence/service_account/{__init__,model}.py with
ServiceAccountRow:
- id: UUID36 PK
- workspace_id: FK workspaces ON DELETE CASCADE
- name: String(64)
- role: String(16) default "member" (Stage 0 lone value; Stage 2 RBAC)
- identity_mode: String(16) default "collapsed" — three states
"collapsed" / "external_passthrough" / "both"; Stage 1 API key
auth layer branches on this to decide whether each call writes
an external_users row
- status: String(16) default "active"
- created_by: FK users ON DELETE RESTRICT (must hand off / delete
SAs before removing their creator)
- created_at / updated_at (UTC, onupdate)
- Index idx_service_accounts_workspace (workspace_id, status)
Registers in deerflow/persistence/models/__init__.py so the engine's
side-effect import path picks it up for Base.metadata.create_all().
T8.1 test: insert_smoke writes a row through a session, reads it back,
asserts each column round-trips. SQLite ephemeral DB per test via
tmp_path, no Postgres required at this layer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends the existing "Boundary check" section with two new entries:
- tests/test_workspace_boundary.py — AST static scan for direct imports of
langgraph.checkpoint.* (and the third-party postgres/sqlite packages)
outside tests/boundary_allowlist.toml. Notes the TYPE_CHECKING exemption
and points readers to `app.gateway.deps.get_checkpointer` as the
intended path; documents the "append to allowlist in the same PR"
contract for new legitimate importers.
- tests/test_workspace_boundary_self.py — self-tests guarding the scanner
from silent-empty regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds backend/tests/test_workspace_boundary.py: walks every .py under
backend/ (excluding tests/, docs/, build artefacts), parses with ast,
flags any ImportFrom/Import that targets langgraph.checkpoint.*,
langgraph_checkpoint_postgres or langgraph_checkpoint_sqlite outside the
allowlist loaded from tests/boundary_allowlist.toml.
Skip rules:
- Imports inside `if TYPE_CHECKING:` (or `if typing.TYPE_CHECKING:`)
blocks are exempt: they do not enter runtime so cannot bypass the
boundary. Verified against deerflow/agents/factory.py which only uses
BaseCheckpointSaver as a parameter annotation.
The scanner walks parent pointers up from each import node to detect
the enclosing TYPE_CHECKING guard, rather than checking only direct
parents — handles nested guards correctly.
RED proof (empty allowlist): 14 violations across the 4 ground-truth
runtime importers (threads / async_provider / provider / worker).
GREEN (with this PR's allowlist): single test passes in ~2s.
The collector function `collect_runtime_checkpoint_imports` and
`scan_violations` are kept at module scope so T7.3's self-test can
exercise them on a synthetic temp file in the next commit.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds backend/tests/boundary_allowlist.toml that names the four current
legitimate runtime importers of langgraph.checkpoint.*:
- app/gateway/routers/threads.py (empty_checkpoint for thread init)
- packages/harness/deerflow/runtime/checkpointer/async_provider.py
- packages/harness/deerflow/runtime/checkpointer/provider.py
- packages/harness/deerflow/runtime/runs/worker.py (empty_checkpoint)
The allowlist replaces plan's draft (thread_runs.py / gateway/app.py)
with the ground-truth grep: both of those reach the checkpointer through
`app.gateway.deps.get_checkpointer` DI, so they need not be listed.
factory.py imports BaseCheckpointSaver only under `if TYPE_CHECKING:` —
the scanner (next commit) exempts type-only imports automatically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flip the 4 business ORM models (ThreadMetaRow, RunRow, FeedbackRow,
RunEventRow) to ``workspace_id: Mapped[str]`` with ``nullable=False``.
PR5's alembic 0003 already enforces NOT NULL at the DB layer; this
aligns the ORM-driven ``create_all()`` path (dev / tests) with the same
invariant so a new install ends up at the post-0003 schema without
running alembic.
Test fallout absorbed:
- `tests/conftest.py` autouse seed now produces a fully consistent
pair: user row (default_workspace_id = test-workspace-autouse) plus
the workspace itself. The PR5 backfill script's "users without
default_workspace_id" query no longer picks the fixture up. Insert
order is user → workspace → UPDATE user, walking around the chicken-
and-egg FK between `workspaces.owner_id` and `users.default_workspace_id`.
- `tests/test_backfill_workspace_id.py` adds a file-scoped autouse
fixture that temporarily flips `column.nullable = True` for the four
business tables (production correctness comes from alembic 0003;
the script's own job is exactly to fill rows between 0002 and 0003
so its tests need that transient state to be representable). Its
`_init_engine` also deletes the autouse seed rows to match the
"fresh DB" model the tests assume.
- `test_thread_meta_workspace_filter::test_create_workspace_none_bypasses`
renamed to `test_create_workspace_none_rejected_by_orm` and asserts
the new IntegrityError on explicit None — write paths can no longer
bypass workspace scope.
- 5 `test_workspace_context` tests + the auth-middleware reset test
get `@pytest.mark.no_auto_workspace` so they keep testing the
unset-contextvar path.
- `test_workspace_repo::test_list_by_user_bypass_returns_all` switches
to membership assertions instead of strict equality since the
autouse fixture surfaces under `user_id=None`.
3214 passed, 30 skipped; the remaining 17 are the documented
pre-existing caplog ordering flakes (all pass in isolation).
`app/gateway/app.py:_check_path_migration_pending` runs at lifespan
startup and emits a single WARNING log when `{base_dir}/users/` still
has content — the operator's cue to run the new migration script. The
warning is best-effort (silent on permission errors) so it can never
escalate into a gateway boot failure; reads from the legacy tree keep
working via the `user_id` branch of `Paths.thread_dir` until the
migration runs.
The root Makefile gains a `migrate-paths` target that wraps
`scripts/migrate_paths_to_workspace.py`. Two opt-in env knobs:
- `DRY_RUN=1` — pass `--dry-run` (preview only, no writes)
- `DEFAULT_WORKSPACE=…` — claim un-assigned users under this workspace id
(defaults to `legacy_workspace` to align with PR5's orphan-row bucket).
`help` documents the new target.
3 tests cover the warning's three states (legacy dir with content, dir
missing, dir empty).
`scripts/migrate_paths_to_workspace.py` walks the PR4 per-user filesystem
layout and rewrites it under the PR6 per-workspace dimension:
- `{base}/users/{uid}/threads/{tid}/` → `{base}/workspaces/{wid}/threads/{tid}/`
- `{base}/users/{uid}/memory.json` → `{base}/workspaces/{wid}/users/{uid}/memory.json`
- `{base}/users/{uid}/agents/{name}/` → `{base}/workspaces/{wid}/users/{uid}/agents/{name}/`
`{wid}` is read from `users.default_workspace_id` in the local sqlite
DB; users without one fall back to `--default-workspace`, defaulting to
`legacy_workspace` (the same bucket PR5's row backfill uses for orphan
rows so the file-system and database alignments stay consistent).
Pre-existing destinations route the legacy copy to
`{base}/migration-conflicts/workspace-migration/...` for manual review;
empty source dirs are cleaned up. The script is idempotent and supports
`--dry-run` (writes nothing, still logs the full report).
9 tests cover: thread / memory / agent rewrite, dry-run no-write, the
fallback workspace path, conflict diversion, post-migration cleanup of
empty `users/` dirs, and the missing-DB / missing-table degraded
modes.
`ThreadDataMiddleware.before_agent` now reads
`get_effective_workspace_id()` and routes the per-thread directory tree
through `Paths.sandbox_*_dir(thread_id, workspace_id=...)`, producing
`{base_dir}/workspaces/{wid}/threads/{tid}/user-data/...`. The legacy
user-id-only layout is no longer written by this middleware; the
migration script in T6.12 will lift any pre-existing `users/{uid}/...`
trees into the new shape.
In no-auth dev mode the contextvar is empty so
`get_effective_workspace_id` returns `"default"` and writes land at
`workspaces/default/...` — the layout invariant ("threads always live
inside a workspace") holds without a real auth setup. `thread_data` now
also exposes `user_id` and `workspace_id` so downstream middlewares
(sandbox, memory, etc.) can read them without re-resolving the
contextvar themselves.
4 new tests cover: contextvar workspace → expected path,
`no_auto_workspace` falls back to `default`, eager mode creates the
right dirs, and the `get_config` fallback still routes through
workspace. Existing 4 thread_data middleware tests stay green.
`Paths` learns a new top-level filesystem dimension. Precedence in
descending order:
- workspace_id given (PR6+):
`{base_dir}/workspaces/{wid}/threads/{tid}/...`
`{base_dir}/workspaces/{wid}/users/{uid}/memory.json`
`{base_dir}/workspaces/{wid}/users/{uid}/agents/{name}/...`
- user_id only (legacy user-isolation): unchanged
- neither (very legacy): unchanged
API: every thread/sandbox/host/ACP helper now takes
`workspace_id: str | None = None` alongside the existing `user_id`,
plus `Paths.workspace_dir()` and `_validate_workspace_id()` for safe
id substitution. `resolve_virtual_path` likewise routes through the
new precedence. The migration script in T6.12 will walk the old
`users/{uid}/threads/...` tree and rewrite to the new shape.
15 new tests (workspace shape, validation, sandbox dirs, ACP,
user-memory/agents under workspace, traversal defence) pass; 46
existing path tests stay green.
End-to-end coverage for the workspace boundary:
- GET /api/threads/{tid} from workspace B → 404
- DELETE /api/threads/{tid} from workspace B → 404
- PATCH /api/threads/{tid} from workspace B → 404
- Positive control: same workspace → 200
Wires `MemoryThreadMetaStore` (real impl, not a mock) behind a
stub-authed FastAPI app. The check_access call inside
`@require_permission` returns False on cross-workspace and the
decorator converts to 404 — proving the boundary holds at the HTTP
boundary, not just the unit level. Run stream/wait are skipped here
(they spin a background worker); their guard goes through the same
decorator path so unit coverage in `test_require_permission_workspace`
is sufficient.
`_StubAuthMiddleware` gains `override_user_contextvar=True` so
cross-user / cross-workspace tests can drive both contextvars from the
stub. Default stays off — the autouse user fixture continues to own
the contextvar for legacy tests whose routes resolve filesystem paths
via `get_effective_user_id()`.
86 router / boundary tests stay green.
1445043649