Commit Graph

302 Commits

Author SHA1 Message Date
1445043649 361e653d37 feat(persistence): PR6 T6.1 — ThreadMetaRepository.create workspace_id sentinel
`create()` now accepts `workspace_id: str | None | _AutoSentinel = AUTO`
on both the SQL and in-memory implementations (and the abstract base).
AUTO resolves via `resolve_workspace_id()` from the workspace contextvar
that PR4 AuthMiddleware sets; explicit None bypasses for migration paths;
explicit str overrides the contextvar.

Test infrastructure:
- conftest gains an autouse `_auto_workspace_context` fixture mirroring
  the existing user fixture. Opt-out via `@pytest.mark.no_auto_workspace`.
- A SQLAlchemy `after_create` listener on `Base.metadata` seeds the
  matching `test-workspace-autouse` + `test-user-autouse` rows whenever
  `init_engine` runs `create_all()`, so the FK from threads_meta to
  workspaces resolves. Alembic migration tests bypass create_all and are
  unaffected, keeping real FK constraints under test.
- `test_thread_meta_workspace_filter.py` covers the three AUTO / explicit
  / None paths.

3 new tests pass; 115 existing thread_meta/run/feedback/run_event/owner
tests stay green.
2026-05-13 17:17:56 +08:00
1445043649 30f2bd0084 test(persistence): alembic 0003 — NOT NULL + UNIQUE coverage
Three SQLite cases + Postgres twins: happy-path upgrade after
populating workspace_id, refuse-with-NULL-leftover raises RuntimeError,
and a UNIQUE(workspace_id, thread_id) IntegrityError reproducer. The
UNIQUE test uses a scratch table because the production schema's
PRIMARY KEY on thread_id would shadow the new UNIQUE index. Index
truthiness asserted instead of bool-equality so SQLite (int 1) and
Postgres (True) both pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:16:24 +08:00
1445043649 8a03abac75 test(persistence): backfill --dry-run end-to-end no-write proof
Snapshots workspace / membership / users.default_workspace_id row
counts before backfill(dry_run=True), runs the orchestrator, asserts
no counts changed. Pins the dry-run report's per-step semantics:
Step 1 counts candidates without populating defaults, so Step 2's
JOIN reports 0; Step 3 picks up all NULL business rows untouched.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:13:16 +08:00
1445043649 def45dd0c6 feat(persistence): backfill Step 3 — orphan rows -> legacy_workspace
_ensure_legacy_workspace creates the nil-UUID anchor (slug=legacy)
owned by the platform admin (or oldest user as fallback). Raises a
clear error if the DB has no users at all so we never silently create
an orphaned workspace. Step 3 UPDATEs each table's remaining
workspace_id IS NULL rows to LEGACY_WORKSPACE_ID. Orchestrator wires
ensure-then-loop between Step 2 and Step 3. 3 new tests: orphan
fan-out, no-users error, end-to-end orchestrator with mixed owned +
orphan rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:12:26 +08:00
1445043649 56f6572086 feat(persistence): backfill Step 2 — UPDATE 4 tables from users.default_workspace_id
Per-table correlated subquery UPDATE (portable across SQLite + Postgres).
Filters workspace_id IS NULL AND user_id IS NOT NULL so already-tagged
rows and orphan rows are skipped. Dry-run mode counts via a JOIN, never
writes. Returns rows-updated for the orchestrator report. 2 new tests:
single-user 4-table fan-out + multi-user isolation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:10:49 +08:00
1445043649 e6bb220979 feat(persistence): backfill Step 1 — workspace per user without default
For each user with NULL default_workspace_id, generate a base slug
from the email, walk past collisions/blacklist via next_available_slug,
create the workspace + owner membership, and set default_workspace_id.
Idempotent: candidates list is filtered by IS NULL, so re-running on a
populated DB is a no-op. 3 new unit tests (creation, idempotence,
blacklist-walker behaviour).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:09:16 +08:00
1445043649 d3361dba59 feat(persistence): 4 business ORM models — nullable workspace_id
ORM-side mirror of alembic 0002. threads_meta gets the matching
(workspace_id, user_id, updated_at) composite index so fresh
create_all() dev databases land on the same shape Alembic produces.
Pre-existing test_alembic_default_workspace_id pins to revision
0001 explicitly since 'head' now includes 0002.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:06:08 +08:00
1445043649 4e26ec9884 test(persistence): alembic 0002 round-trip on SQLite + Postgres
Mirrors test_alembic_default_workspace_id pattern: synchronous bodies
(env.py uses asyncio.run internally), pre-PR5 schema bootstrap with the
minimal columns the migration touches, upgrade/downgrade assertions for
the workspace_id column + FK + composite index on all 4 business tables.
Postgres cases auto-skip when Docker daemon is unreachable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 09:04:21 +08:00
1445043649 5c7753c0b8 feat(auth): lifespan _ensure_admin_user backfills missing workspace
Pre-PR4 admins have users.default_workspace_id NULL. Without this
backfill they would log in successfully but immediately bounce off the
workspace gate (T4.7) because their token cannot encode a wid. The
lifespan hook now resolves the admin user and calls
ensure_default_workspace (idempotent — no-op if the user already has
a workspace), so the post-upgrade boot makes the admin usable.

Renamed routers.auth._ensure_default_workspace → ensure_default_workspace
to make it importable across modules.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:39:09 +08:00
1445043649 91846a201e feat(auth): /auth/me returns workspaces[] with id/name/slug/role
GET /auth/me now responds with the new UserMeResponse: the existing
user fields plus default_workspace_id and a workspaces list that
joins WorkspaceRepository.list_by_user with the caller's role from
WorkspaceMembershipRepository.list_by_user. Stage 0 every user has
exactly one entry there, but the shape is forward-compatible for
Stage 2 multi-workspace memberships.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:37:29 +08:00
1445043649 634e5119e1 feat(auth): login + change_password re-issue JWTs with wid + role
After PR4 every protected request demands wid. Three JWT-issue paths
must encode it; T4.8/T4.9 covered /initialize and /register, and this
commit closes the remaining two:

- login_local calls _ensure_default_workspace (idempotent — returns
  the existing default_workspace_id when set) so users who pre-date
  PR4 are backfilled at login time, and signs the new cookie with
  wid + role='owner'.
- change_password does the same, ensuring a password change does not
  strip the workspace claim and lock the user out on their next
  request.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:35:06 +08:00
1445043649 a657d17995 feat(auth): /initialize auto-creates default workspace + owner membership
initialize_admin now seeds a 1-person workspace immediately after the
admin user is created: WorkspaceRepository.create(name, slug, owner_id)
+ WorkspaceMembershipRepository.add(role='owner') + writes the new
workspace id back to users.default_workspace_id. The session JWT is
re-issued with wid + role='owner' so subsequent requests pass the
T4.7 workspace gate.

Mechanical pieces:
- SQLiteUserRepository row<->user mapping now includes
  default_workspace_id (sql update_user too) so the column persists.
- workspace.sql.SLUG_BLACKLIST is now public (was _SLUG_BLACKLIST) and
  the registration helper treats blacklisted slugs as "taken" so the
  walker steps past reserved names like "admin" instead of crashing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:31:53 +08:00
1445043649 a06e88d58b feat(auth): workspace slug helpers — auto_slug_from_email + next_available_slug
auto_slug_from_email maps an email's local-part to a schema-valid base
slug (lowercase, [+_.]→'-', alphanumeric-and-hyphen, clamp 32 chars,
fallback to user-{token_hex(4)} for pathological inputs).
next_available_slug walks the {base, base-2, base-3, ...} sequence
against a caller-supplied async exists_check until it lands on a free
slot, truncating base when the suffix would push past 32 chars.

Pulled forward of T4.8/T4.9 because both /auth/initialize and
/auth/register need it. Lives in app.gateway.auth (not persistence)
since "email → slug" is a registration-time concern.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:27:42 +08:00
1445043649 2145d36744 feat(auth): AuthMiddleware injects workspace ContextVar from JWT wid/role
deps.get_current_user_from_request now stashes the decoded payload on
request.state.auth_payload so AuthMiddleware can populate the
workspace_context ContextVar without a second decode. Reset is paired
in the same try/finally as user_context to keep teardown atomic.

Also adds:
- auth.models.ActiveWorkspace — minimal proxy that satisfies the
  CurrentWorkspace protocol (id + role only).
- auth.models.User.default_workspace_id — surfaces the DB column added
  in T4.4 so the eventual /auth/me payload can reference it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:25:43 +08:00
1445043649 b4bef65079 feat(auth): decode_token rejects legacy 4-field JWTs as WORKSPACE_MISSING
After PR4 every JWT must carry wid (workspace_id). decode_token now
returns the new TokenError.WORKSPACE_MISSING when the signature is
valid but the payload lacks wid; expired tokens still report EXPIRED
first so /auth/refresh logic stays correct. AuthErrorCode gains a
matching WORKSPACE_REQUIRED for middleware to surface to clients.

Updates 13 existing test sites that issued tokens without wid to pass
workspace_id="ws-test" + role="owner", reflecting the new contract.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:22:10 +08:00
1445043649 54cb94c30f feat(auth): JWT TokenPayload accepts wid + role
TokenPayload gains optional wid (workspace_id) and role claims;
create_access_token accepts them as keyword-only args and only
encodes them when provided. Existing tokens and existing callers
keep working unchanged — the contract that protected requests
must carry wid is enforced by middleware (PR4 T4.6/T4.7), not by
the JWT type system.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:17:55 +08:00
1445043649 917d8fbeaf test(persistence): alembic 0001 round-trip on SQLite + Postgres
Verifies alembic upgrade 0001 adds users.default_workspace_id (with FK
to workspaces) and that downgrade cleanly removes it. Runs on both
dialects because the migration uses op.batch_alter_table for SQLite
ALTER compatibility.

Tests are intentionally sync — alembic's command layer is sync and
env.py calls asyncio.run(); running under pytest-anyio would deadlock
on the inner event loop.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 22:11:11 +08:00
1445043649 3313047ff1 test(workspace): partial unique on owner — Postgres twin test
新建 test_workspace_partial_unique_postgres.py,2 个 @pytest.mark.postgres
test,用 PR1 的 testcontainers postgres_url fixture 验:
  - test_partial_unique_on_owner_enforced_on_postgres:第 2 个 owner
    IntegrityError(SQLite twin 在 test_workspace_membership_repo 已覆盖)
  - test_multiple_admins_allowed_on_postgres:多个 admin/member 不触发约束

为什么 PG 单独写:sqlite_where vs postgresql_where 是两份 DDL;SQLAlchemy
能 emit 不代表 PG 真的执行。本测试 pin 两边语义一致。

本地无 docker daemon → SKIPPED;CI workflow backend-postgres-tests 会实跑。

Stage 0 PR3 T3.8。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:15:25 +08:00
1445043649 36ffe2713f feat(persistence): WorkspaceMembershipRepository + 8 unit tests
新建 backend/packages/harness/deerflow/persistence/workspace_membership/sql.py,方法:
  - add(workspace_id, user_id, role, invited_by=None) 校验 role ∈
    {owner, admin, member}(Stage 0 仅写 owner,schema 已支持其余)
  - remove(workspace_id, user_id) → bool(rowcount > 0)
  - list_by_user(user_id) 按 joined_at desc
  - list_by_workspace(workspace_id) 按 joined_at asc
  - get_role(workspace_id, user_id) → str | None
  - change_role(workspace_id, user_id, new_role) → bool

MembershipValidationError 自定义异常(role 非法)。

8 test 覆盖:
  - add → get_role round-trip
  - remove 命中/未命中返回 True/False
  - 同 workspace 第 2 个 owner 触发 IntegrityError(partial unique)
  - 同 workspace 多个 admin 不触发(Stage 2 forward compat)
  - CASCADE: 删 user 自动清理 memberships
  - list_by_user 按 joined_at desc 排序(多 workspace)
  - role 校验拒绝 'viewer'
  - change_role 命中改值 + 未命中返 False

注:owner 转让需要事务内两行原子 swap(先把现 owner 改 admin,再把新 owner
插 owner),放 PR4+ auth router 内做带权限检查的版本;本仓储不暴露
transfer_ownership 方法以保持单一职责。

Stage 0 PR3 T3.7。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:13:52 +08:00
1445043649 a40df03521 feat(persistence): WorkspaceRepository + 23 unit tests
新建 backend/packages/harness/deerflow/persistence/workspace/sql.py,方法:
  - create(name, slug, owner_id, *, workspace_id=None, status='active')
    UUID v4 自动生成;强校验 slug 格式(regex ^[a-z0-9](-?[a-z0-9])*\$ + 3-32
    长度)+ slug 黑名单(25 个保留字)+ status 枚举
  - get(workspace_id, *, user_id=AUTO) JOIN workspace_memberships 做成员校验;
    user_id=None 显式 bypass(迁移/admin)
  - get_by_slug(slug) 不带成员校验(path-based routing 用:先 slug→workspace_id
    再到 route handler 里查成员)
  - list_by_user(*, user_id=AUTO) 列 user 所属所有 workspace
  - update_status / delete platform-admin 操作,不带成员校验

WorkspaceValidationError 自定义异常(slug 格式 / 黑名单 / status)。

23 test 覆盖:
  - CRUD smoke + get_by_slug missing
  - 重复 slug → IntegrityError
  - 8 个 invalid slug pattern(短/长/大写/空格/破折号位置/连续破折号/下划线)
  - 6 个 blacklisted slug
  - status 状态机 + 非法值拒绝
  - delete CASCADE 到 memberships(SQLite FK PRAGMA 已开启)
  - get/list 成员过滤(user-A 看不见 user-B 的 workspace)
  - list user_id=None 显式 bypass

全部在 SQLite ephemeral DB 上跑(< 1s)。partial-unique 双驱动验证留给 T3.8。

Stage 0 PR3 T3.4 + T3.5。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:11:28 +08:00
1445043649 f63089aea8 feat(runtime): add workspace_context module + tests
新建 backend/packages/harness/deerflow/runtime/workspace_context.py,仿
user_context.py 的 API 形态:

  - CurrentWorkspace Protocol(要求 .id: str + .role: str)
  - _current_workspace ContextVar + set/reset/get/require
  - DEFAULT_WORKSPACE_ID = "default" + get_effective_workspace_id(fallback
    友好,不抛错;用于文件系统路径)
  - AUTO 哨兵 + resolve_workspace_id 三态(AUTO/str/None)

与 user_context 的区别:CurrentWorkspace 额外要求 .role 字段,让 Protocol 同时
约束"workspace 是哪个"和"caller 在该 workspace 内的角色"(Stage 0 只见 'owner',
Stage 2 RBAC 打开 admin/member)。

backend/tests/test_workspace_context.py 16 个 test,覆盖:
  - 4 个 set/reset/require 行为
  - 3 个 Protocol structural check(接受 .id+.role / 拒少 .role / 拒少 .id)
  - 4 个 get_effective_workspace_id(含 UUID → str 强转)
  - 5 个 resolve_workspace_id 三态(AUTO/AUTO 无 ctx 抛错/explicit str/explicit
    None/AUTO 强转 str)

Stage 0 PR3 T3.1 + T3.2。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-12 21:04:49 +08:00
1445043649 7f17cb8fee chore(test): bump testcontainers Postgres to 17-alpine
\`make doctor\` 实测远程 Aliyun RDS = PostgreSQL 17.9。把
backend/tests/fixtures/postgres.py 的 testcontainers 镜像从
postgres:16-alpine 调到 postgres:17-alpine,与生产对齐。

同步把 STATUS.md "RDS 大版本对齐" 项标 done。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 17:29:16 +08:00
1445043649 d312bdf968 test(config): pin default postgres backend + sqlite regression
新建 backend/tests/test_default_database_backend.py 两个测试:

  - test_explicit_sqlite_backend_still_works (T2.3)
    显式 database.backend=sqlite 仍生效;防 PR2 改默认后 SQLite 用户
    悄无声息 regression
  - test_config_example_default_backend_is_postgres (T2.4)
    直接读 on-disk config.example.yaml,断言 database.backend == 'postgres'
    且 postgres_url == '\$DATABASE_URL'(不允许硬编码凭据)

Stage 0 PR2 T2.3 + T2.4.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 09:12:28 +08:00
1445043649 8f480cd76f test(persistence): postgres smoke tests for init_engine + ThreadMetaRepo
新建 backend/tests/test_postgres_smoke.py,4 个 test 用 postgres_url fixture:

  - test_postgres_url_creates_isolated_database (T1.4)
    asyncpg 直连验证 fixture 落到 test_<hex> DB
  - test_postgres_url_isolates_between_tests (T1.4)
    手动起第 2 个 DB 比对,证明每 test 隔离
  - test_init_engine_postgres_creates_tables (T1.5)
    init_engine('postgres', postgres_url) 跑完后 information_schema
    出现 5 张现有表(users/threads_meta/runs/feedback/run_events)
  - test_thread_meta_repo_postgres_round_trip (T1.6)
    ThreadMetaRepository.create + get 完整 round-trip,验证 PG 上仓储行为
    与 SQLite 对齐

全部 @pytest.mark.postgres 门控;Docker 不可用时由 fixture 跳过。本次本地
docker daemon 未起,4 测试 SKIP;CI workflow(T1.9)将带 docker service
触发实跑。

Regression:跑全套 `pytest tests/` 3085 passed + 23 skipped + 0 failed
(基线 3086 passed + 18 skipped;diff = 4 新 postgres skip + 1 env 相关
live test 偶发 skip)。

Stage 0 PR1 T1.4 + T1.5 + T1.6.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:50:10 +08:00
1445043649 31361e2dcf test(fixtures): add testcontainers Postgres fixture for Stage 0
新建 backend/tests/fixtures/postgres.py,提供两层 fixture:
  - postgres_container (session 级):用 PostgresContainer 起 postgres:16-alpine
  - postgres_url (function 级):每 test 一个 ephemeral DB,teardown 时
    pg_terminate_backend 清掉残连后 DROP DATABASE

为什么 per-DB 而不是 per-schema:asyncpg + SQLAlchemy 不通过 URL 传 search_path,
per-DB 一次 ~50ms 开销可接受,让 test 代码不感知 schema。

Docker 不可用时 fixture 自动 pytest.skip 而非 error,dev 环境无 Docker
仍能跑其余 3086 个测试。

注册 pytest mark `postgres`、把 tests/ 加 sys.path 让 pytest_plugins
按 `fixtures.postgres` 路径解析(不加 tests/__init__.py 避免干扰
现有 pytest 发现行为)。

Stage 0 PR1 T1.3.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 22:44:02 +08:00
DanielWalnut 2b1fcb3e43 fix(task): remove max_turns parameter from task tool interface (#2783)
Unit Tests / backend-unit-tests (push) Has been cancelled
Frontend Unit Tests / frontend-unit-tests (push) Has been cancelled
Lint Check / lint (push) Has been cancelled
Lint Check / lint-frontend (push) Has been cancelled
* fix(task): remove max_turns parameter from task tool interface

Subagents should always use their configured max_turns value. Exposing
this parameter allowed callers to override the admin-configured limit,
which is undesirable. The value is now exclusively driven by subagent
config (per-agent overrides and global defaults in config.yaml).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-08 15:05:24 +08:00
He Wang 7de9b5828b fix(tools): introduce Runtime type alias to eliminate Pydantic serialization warning (#2774)
* fix(tools): introduce Runtime type alias to eliminate Pydantic serialization warning

Add deerflow/tools/types.py with:

    Runtime = ToolRuntime[dict[str, Any], ThreadState]

Replace every runtime: ToolRuntime[ContextT, ThreadState] and
runtime: ToolRuntime[dict[str, Any], ThreadState] annotation in
sandbox/tools.py, present_file_tool.py, task_tool.py, view_image_tool.py,
and skill_manage_tool.py with the new Runtime alias.

The unbound ContextT TypeVar (default None) caused
PydanticSerializationUnexpectedValue warnings on every tool call because
LangChain's BaseTool._parse_input calls model_dump() on the auto-generated
args_schema while DeerFlow passes a dict as runtime context.
Binding the context to dict[str, Any] aligns Pydantic's serialization
expectations with reality and removes the noise from all run modes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(tools): extend Runtime alias to setup_agent and update_agent tools

Replace bare ToolRuntime annotations in setup_agent_tool.py and
update_agent_tool.py with the shared Runtime alias introduced in the
previous commit, and add both tools to the Pydantic serialization
warning regression test (13 cases total).

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(tools): loosen Pydantic warning filter to avoid version-specific format

Replace the brittle "field_name='context'" substring check with a looser
"context" match so the assertion stays valid if Pydantic changes its
internal warning format across versions.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(tools): simplify warning filter and clean up docstring

Remove the "context" substring condition from the Pydantic warning
filter — asserting that no PydanticSerializationUnexpectedValue fires
at all is both simpler and more comprehensive, since the test payload
contains only the tool's own args plus runtime.

Also update the module docstring to remove the version-specific warning
format example that was inconsistent with the looser filter.

Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-08 14:50:33 +08:00
Eilen Shin 37db689349 fix(events): serialize structured db event content (#2762) 2026-05-08 10:17:17 +08:00
Eilen Shin bd45cb2846 fix(sandbox): disable msys path conversion (#2766) 2026-05-08 10:13:11 +08:00
Eilen Shin 5fd0e6ac89 fix(middleware): sync raw tool call metadata (#2757) 2026-05-08 10:08:53 +08:00
Tao Liu daa3ffc29b feat(loop-detection): make loop detection configurable with per-tool frequency overrides (#2711)
* Make loop detection configurable

Expose LoopDetectionMiddleware thresholds through config.yaml while preserving existing defaults and allowing the middleware to be disabled.

Refs bytedance/deer-flow#2517

* feat(loop-detection): add per-tool tool_freq_overrides to Phase 1

Adds ToolFreqOverride model and tool_freq_overrides field to
LoopDetectionConfig, wires it through LoopDetectionMiddleware, and
documents the option in config.example.yaml.

Resolves the gap flagged in the #2586 review: without per-tool overrides,
users hit by #2510/#2511 (RNA-seq workflows exceeding the bash hard limit)
had no way to raise thresholds for one tool without loosening the global
limit for every tool.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* docs(loop-detection): document tool_freq_overrides in LoopDetectionMiddleware docstring

Add the missing Args entry for tool_freq_overrides, explaining the
(warn, hard_limit) tuple structure and how per-tool thresholds supersede
the global tool_freq_warn / tool_freq_hard_limit for named tools.
Also run ruff format on the three files flagged by the lint check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(loop-detection): validate LoopDetectionMiddleware __init__ params eagerly

Raise clear ValueError at construction time instead of crashing at
unpack-time inside _track_and_check when bad values are passed:
- tool_freq_overrides: must be 2-tuples of positive ints with hard_limit >= warn
- scalar thresholds: warn_threshold, hard_limit, tool_freq_warn,
  tool_freq_hard_limit must be >= 1 and hard limits must >= their warn pairs
- window_size, max_tracked_threads must be >= 1

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(test): isolate credential loader directory-path test from real ~/.claude

The test didn't monkeypatch HOME, so on any machine with real Claude Code
credentials at ~/.claude/.credentials.json the function fell through to
those credentials and the assertion failed. Adding HOME redirect ensures
the default credential path doesn't exist during the test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style(test): add blank lines after import pytest in TestInitValidation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(loop-detection): collapse dual validation to LoopDetectionConfig

Modifications
  - LoopDetectionMiddleware.__init__: stripped of all ValueError raises;
    becomes a plain field-assignment constructor.
  - LoopDetectionMiddleware.from_config: classmethod that builds the
    middleware from a Pydantic-validated LoopDetectionConfig and handles
    the ToolFreqOverride -> tuple[int, int] conversion.
  - agents/factory.py: SDK construction routed through
    LoopDetectionMiddleware.from_config(LoopDetectionConfig()) so the
    defaults path is Pydantic-validated too.
  - agents/lead_agent/agent.py: uses from_config instead of unpacking
    config fields by hand.
  - tests/test_loop_detection_middleware.py: deleted TestInitValidation
    (16 methods exercising the removed __init__ checks); added
    TestFromConfig (4 tests: scalar field mapping, override tuple
    conversion, empty overrides, behavioral smoke test).

Result: one validation layer (Pydantic), zero duplication, no __new__
hacks. Both production construction sites flow through LoopDetectionConfig.

Test results
  make test   -> 2977 passed, 18 skipped, 0 failed (137s)
  make format -> All checks passed; 411 files left unchanged

* feat(agents): make loop_detection configurable in create_deerflow_agent

Adds a `loop_detection: bool | AgentMiddleware = True` field to
RuntimeFeatures, mirroring the existing pattern used by `sandbox`,
`memory`, and `vision`. SDK users can now disable LoopDetectionMiddleware
or replace it with a custom instance built from their own
LoopDetectionConfig — e.g.
`LoopDetectionMiddleware.from_config(my_cfg)` — instead of being stuck
with the hardcoded defaults previously installed by the SDK factory.

The lead-agent path (which already reads AppConfig.loop_detection) is
unchanged, and the default `True` preserves prior always-on behavior for
all existing callers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

---------

Co-authored-by: knight0940 <631532668@qq.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Amorend <142649913+knight0940@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-07 16:15:15 +08:00
AochenShen99 cef4224381 fix(skills): enforce allowed-tools metadata (#2626)
* fix(skills): parse allowed-tools frontmatter

* fix(skills): validate allowed-tools metadata

* fix(skills): add shared allowed-tools policy

* fix(subagents): enforce skill allowed-tools

* fix(agent): enforce skill allowed-tools

* refactor(skills): dedupe TypeVar and reuse cached enabled skills

- Drop redundant module-level TypeVar in tool_policy; rely on PEP 695 syntax.
- Expose get_cached_enabled_skills() and have the lead agent reuse it
  instead of synchronously rescanning skills on every request.

* fix(agent): expose config-scoped skill cache

* fix(subagents): pass filtered tools explicitly

* fix(skills): clean allowed-tools policy feedback
2026-05-07 08:34:43 +08:00
Hinotobi 2b0e62f679 [security] fix(auth): reject cross-site auth POSTs (#2740)
* fix(security): reject cross-site auth posts

* fix(auth): align secure cookie proxy scheme handling

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-07 07:58:06 +08:00
Eilen Shin 1336872b15 fix(channels): authenticate gateway command requests (#2742) 2026-05-06 15:27:34 +08:00
KiteEater 4ead2c6b19 fix(config): reset config-backed singletons on hot reload (#2588)
* Fix stale config singletons on reload

* fix(config): update checkpointer imports after runtime move

* Fix config reload singleton mutation on validation failure

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-06 10:17:55 +08:00
yangzheli 59c4a3f0a4 feat(agent): add custom-agent self-updates with user isolation (#2713)
* feat(agent): add update_agent tool for in-chat custom-agent self-updates (#2616)

Custom agents had no built-in way to persist updates to their own SOUL.md /
config.yaml from a normal chat — `setup_agent` was only bound during the
bootstrap flow, so when the user asked the agent to refine its description
or personality, the agent would shell out via bash/write_file and the edits
landed in a temporary sandbox/tool workspace instead of
`{base_dir}/agents/{agent_name}/`.

Changes:
- New `update_agent` builtin tool with partial-update semantics (only the
  fields you pass are written) and atomic temp-file + os.replace writes so
  a failed update never corrupts existing SOUL.md / config.yaml.
- Lead agent now binds `update_agent` in the non-bootstrap path whenever
  `agent_name` is set in the runtime context. Default agent (no
  agent_name) and bootstrap flow are unchanged.
- New `<self_update>` system-prompt section is injected for custom agents,
  instructing them to use `update_agent` — and explicitly NOT bash /
  write_file — to persist self-updates.
- Tests: 11 new cases in `tests/test_update_agent_tool.py` covering
  validation (missing/invalid agent_name, unknown agent, no fields),
  partial updates (soul-only, description-only, skills=[] vs omitted),
  no-op detection, atomic-write safety, and AgentConfig round-tripping;
  plus 2 new cases in `tests/test_lead_agent_prompt.py` covering the
  self-update prompt section.
- Docs: updated backend/CLAUDE.md builtin tools list and tools.mdx
  (en/zh) with the new tool description.

* feat(agent): isolate custom agents per user

Store custom agent definitions under the effective user, keep legacy agents readable until migration, and cover API/tool/migration behavior with tests.

Co-authored-by: Cursor <cursoragent@cursor.com>

* feat: consistent write/delete targets & add --user-id to migration

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-05 23:17:42 +08:00
Nan Gao e8675f266d fix(loop-detection): keep tool-call pairing on warn injection (#2724) (#2725)
* fix(loop-detection): keep tool-call pairing on warn injection (#2724)

* make format

* fix(loop-detection): avoid IMMessage leak to downstream consumer

* fix(channels): filter loop warning text from IM replies
2026-05-05 18:53:49 +08:00
Xun 680187ddc2 fix: Supplement list_running in RemoteSandboxBackend (#2716)
* fix: Supplement list_running in RemoteSandboxBackend

* fix

* except requests.RequestException as exc:

* fix
2026-05-05 18:53:10 +08:00
Willem Jiang 028493bfd8 fix(docker):force ngix to resolve upstream names at request time (#2717)
* fix(docker):force ngix to resolve upstream names at request time

* fix(docker): set resolver valid=0s to eliminate DNS cache window for request-time re-resolution

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/07bdb872-022f-4fd2-9fa8-d800a4ce34a7

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

* Update DNS resolver valid time and add upstreams

* fix the unit test error

* Remove upstream server configurations from nginx.conf

Removed upstream server configurations for gateway and frontend.

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-05-05 14:35:55 +08:00
Willem Jiang 8e48b7e85c fix(channels): preserve clarification conversation history across follow-up turns (#2444)
* fix(channels): preserve clarification conversation history across follow-up turns

Pin channel-triggered runs to the root checkpoint namespace and ensure thread_id is always present in configurable run config so follow-up replies resume the same conversation state.

Add regression coverage to channel tests:

assert checkpoint_ns/thread_id are passed in wait and stream paths
add an integration-style clarification flow test that verifies the second user reply continues prior context instead of starting a new session
This addresses history loss after ask_clarification interruptions (issue #2425).

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix(channels): copy configurable dict before injecting run-scoped fields

  When configurable was already a plain dict, _resolve_run_params mutated
  it in place, leaking checkpoint_ns and thread_id back into the shared
  session config. Always copy via dict() before mutating to prevent
  cross-user or cross-channel config pollution.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-05-04 16:14:07 +08:00
YuJitang d02f762ab0 feat: refine token usage display modes (#2329)
* feat: refine token usage display modes

* docs: clarify token usage accounting semantics

* fix: avoid duplicate subtask debug keys

* style: format token usage tests

* chore: address token attribution review feedback

* Update test_token_usage_middleware.py

* Update test_token_usage_middleware.py

* chore: simplify token attribution fallback

* fix token usage metadata follow-up handling

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-04 09:56:16 +08:00
Nan Gao f80ac961ec fix(harness): restore legacy skills path fallback (#2694) (#2696)
* fix(harness): restore legacy skills path fallback (#2694)

* fix(format): make format

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-03 23:40:59 +08:00
wanxsb 44ab21fc44 feat(community): add Serper web search provider (#2630)
* feat(community): add Serper web search provider

Add a new community search provider backed by the Serper Google Search
API (https://serper.dev). Serper returns real-time Google results via a
simple JSON API and requires only an API key — no extra Python package.

Changes:
- backend/packages/harness/deerflow/community/serper/__init__.py
- backend/packages/harness/deerflow/community/serper/tools.py
  Implements web_search_tool using httpx (already a project dependency).
  API key is read from config.yaml `api_key` field or SERPER_API_KEY env var.
  Follows the same interface / output shape as the existing ddg_search provider.
  Exposes max_results parameter (default 5) with config override logic.
- backend/tests/test_serper_tools.py
  Unit tests covering API key resolution, config overrides, HTTP errors,
  empty results, and parameter passing.
- config.example.yaml: add commented-out Serper example alongside other providers
- .env.example: add SERPER_API_KEY placeholder

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix the lint error

* Fix the lint error

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-02 16:22:35 +08:00
Hinotobi e543bbf5d6 [security] fix(upload): reject symlinked upload destinations (#2623)
* fix: reject symlinked upload destinations

* test: harden upload destination checks

* fix: address PR feedback for #2623

* test: cover safe upload re-uploads

* fix: preserve upload limit checks after rebase

* fix(upload): stream safe HTTP upload writes
2026-05-02 15:19:28 +08:00
Xinmin Zeng ca3332f8bf fix(gateway): return ISO 8601 timestamps from threads endpoints (#2599)
* fix(gateway): return ISO 8601 timestamps from threads endpoints (#2594)

ThreadResponse documents created_at / updated_at as ISO timestamps,
matching the LangGraph Platform schema (langgraph_sdk.schema.Thread
exposes them as datetime, JSON-encoded as ISO 8601). The gateway
threads router was instead emitting str(time.time()) — unix-second
floats — breaking frontend new Date() parsing and producing a mixed
ISO/unix wire format that also corrupted the search sort order.

Centralize timestamp generation in deerflow.utils.time:
- now_iso()       — datetime.now(UTC).isoformat()
- coerce_iso(x)   — heals legacy unix-timestamp strings on read so the
                    store converges to ISO without a one-shot migration

threads.py: replace 6 time.time() call sites with now_iso(); wrap all
read paths and Phase-2 checkpoint metadata with coerce_iso(); _store_upsert
opportunistically heals legacy created_at on update; drop unused time import.

thread_runs.py: reuse now_iso() instead of a private duplicate _now_iso(),
preventing future drift between the two timestamp call sites.

Tests: 9 unit tests for the helper; 5 integration tests pinning the ISO
contract for create/get/patch/search and the legacy-healing path on the
internal store upsert. Full suite: 2144 passed, 15 skipped, 0 failed.

Closes #2594

* fix(gateway): coerce checkpoint metadata timestamps to ISO on read

After the merge with main, three additional read paths in ``threads.py``
were still emitting raw ``str(metadata.get("created_at", ""))`` —
``get_thread_state``, ``update_thread_state``, and ``get_thread_history``.

Same root cause as #2594: when the checkpoint metadata's ``created_at``
is a unix-second float (legacy data, or a checkpoint written by an older
Gateway version), ``str(float)`` produces ``"1777252410.411327"`` and the
frontend's ``new Date(...)`` returns ``Invalid Date``. The fix on the
``/threads/{id}`` GET path was already in place; these three sibling
endpoints needed the same treatment.

All four call sites now flow through ``coerce_iso``, so:
- legacy float metadata heals to ISO on the way out,
- ISO metadata passes through unchanged,
- ``datetime`` instances (which the new ``coerce_iso`` branch handles
  explicitly) emit with the ``T`` separator instead of falling through
  to the space-separated ``str(datetime)`` form.

Coverage added for the two endpoints not already pinned by the merge:
- ``test_get_thread_state_returns_iso_for_legacy_checkpoint_metadata``
- ``test_get_thread_history_returns_iso_for_legacy_checkpoint_metadata``

Both pre-seed a checkpoint whose metadata carries the literal float
from the issue body and assert the wire format is ISO.
2026-05-02 15:16:16 +08:00
Willem Jiang bb8b234d85 chroe(2585): keep polishing the code of codex token usage (#2689) 2026-05-02 15:04:11 +08:00
KiteEater 17447fccbe fix(runtime): make rollback restore checkpoint supersede newer checkpoints (#2582)
* Restore rollback checkpoints with fresh ids

* Tighten rollback checkpoint tests and imports

* Update test_run_worker_rollback.py

---------

Co-authored-by: Willem Jiang <willem.jiang@gmail.com>
2026-05-02 11:25:45 +08:00
KiteEater 866d1ca409 Populate Codex usage metadata for token accounting (#2585) 2026-05-02 11:16:03 +08:00
greatmengqi 8ba01dfd83 refactor: thread app_config through lead and subagent task path (#2666)
* refactor: thread app config through lead prompt

* fix: honor explicit app config across runtime paths

* style: format subagent executor tests

* fix: thread resolved app config and guard subagents-only fallback

Address two PR review findings:

1. _create_summarization_middleware passed the original (possibly None)
   app_config into create_chat_model, forcing the model factory back to
   ambient get_app_config() and risking config drift between the
   middleware's resolved view and the model's view. Pass the resolved
   AppConfig instance through end-to-end.

2. get_available_subagent_names accepted Any-typed config and forwarded
   it to is_host_bash_allowed, which reads ``.sandbox``. A
   SubagentsAppConfig (also accepted upstream as a sum-type input) has
   no ``.sandbox`` attribute and would be silently treated as "no
   sandbox configured", incorrectly disabling the bash subagent. Guard
   on hasattr and fall back to ambient lookup otherwise.

Adds regression tests for both paths.

* chore: simplify hasattr guard and tighten regression tests

- Collapse if/else into ternary in get_available_subagent_names; hasattr(None, ...) is False so the explicit None check was redundant.
- Drop comments that narrate the change rather than explain non-obvious WHY (test names already convey intent).
- Replace stringly-typed sentinel "no-arg" in regression test with direct args tuple comparison.

---------

Co-authored-by: greatmengqi <chenmengqi.0376@bytedance.com>
2026-05-02 06:37:49 +08:00
Willem Jiang 189b82405c fix(sandbox): pass no_change_timeout to exec_command to prevent 120s premature termination (#2685)
* fix(sandbox): pass no_change_timeout to exec_command to prevent 120s premature termination

  The agent_sandbox library's shell API defaults no_change_timeout to 120
  seconds. When AioSandbox.execute_command() called exec_command() without
  this parameter, commands producing no output for 120s would return with
  NO_CHANGE_TIMEOUT status even though the script was still running.

  Pass no_change_timeout=600 to all exec_command calls (matching the
  client-level HTTP timeout) so long-running commands are not cut short.

  Fixes #2668

* test(sandbox): add assertions for no_change_timeout in execute_command and list_dir

Agent-Logs-Url: https://github.com/bytedance/deer-flow/sessions/2f37bc72-0826-4443-a6ba-e5b78c22fb5a

Co-authored-by: WillemJiang <219644+WillemJiang@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-05-01 22:27:02 +08:00