430f4a113202a16356b278091c05c099272e4a41
26 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 1445043649
|
430f4a1132 |
docs(impl): PR5 implementation note + STATUS update
11 PR5 commits: alembic 0002 (nullable workspace_id + FK + composite index on 4 business tables) -> backfill_workspace_id.py (3-step idempotent) -> alembic 0003 (pre-flight refuse + NOT NULL + UNIQUE). 18 new tests (9 alembic 0002+0003 + 9 backfill, including dry-run and no-users error). 3150 passed + 30 skipped + 16 pre-existing caplog flake (unchanged from PR4). T5.11 (ORM nullable=False) deferred to PR6 — flipping the ORM-side NOT NULL would break 6 INSERT sites whose repository signatures get workspace_id wiring in PR6 anyway. DB-level invariant is already enforced by alembic 0003 along the production path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
44f84800e9 |
docs(impl): PR4 implementation note + STATUS update
Records the 14-task PR4 delivery (T4.1-T4.14), final test baseline (3136 passed / 26 skipped), key design decisions that diverged from the plan draft (TokenPayload optional fields, ensure_default_workspace helper shape, slug walker treating blacklist as taken), follow-up items, and a copy-paste live smoke command list for the user to verify make-dev end-to-end. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
d98498b705 |
docs(impl): T4.1 — confirm alembic baseline is not needed
alembic heads/history both empty, versions/ dir empty: 0001 can be the first revision with down_revision=None. alembic_version table will be auto-created on first upgrade head. doctor.py needs no new check. Also records T4 preparation grep findings on _ensure_admin_user current behavior (open issue #3 resolved). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
c70c6594de |
docs(impl): STATUS — record LOCK sign-off and origin push
Two unblocking events on 2026-05-12: team signed off all 7 irreversible schema decisions, and docs branch was finally pushed to origin (38 commits, via SSH-over-443 to bypass local proxy). PR4 is now unblocked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
a592319e3c |
docs(impl): STATUS update after PR3 merge
PR3 ✅ merged 进 docs branch（7 commits, T3.1-T3.10）+ RDS live 验证通过 （11 张表含 workspaces/workspace_memberships，partial unique on owner 索引 建出来）。测试基线 3134 passed + 25 skipped + 0 failed。 下一个 PR4（auth 改造）plan 推荐 Inline 模式（跨 jwt.py/auth_middleware.py/ routers/auth.py 多文件耦合紧）。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
dda8264057 |
docs(impl): PR3 implementation note + acceptance checklist
记录 Stage 0 PR3 的实施落点、LOCK 决策、跟进项与 6 commit 列表。 验收：3134 passed + 25 skipped + 0 failed（PR2 末 3087 + 47 新 ws_context/ ws_repo/membership_repo 测试 + 2 PG-only skipped）。RDS live 验证通过： workspaces / workspace_memberships 表 + partial unique on owner 都建出来。 T3.6 (membership ORM) 提前到 T3.4-T3.5 (workspace repo) 之前，因为 WorkspaceRepository.get/list_by_user 会 JOIN memberships。 Stage 0 PR3 T3.10. PR3 完成。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
cbbb83a706 |
docs(impl): STATUS update after PR2 live verification on RDS
PR1 + PR2 已 live 验证：远程 Aliyun RDS (PostgreSQL 17.9) 上 9 张表 (DeerFlow 5 + LangGraph 4) 全部 create_all 成功。 记 3 个 PR2 follow-up commits（不在原 plan task list 但 live 必需）： - testcontainers image 16→17 对齐 RDS 大版本 - serve.sh auto-add --extra postgres 防 uv sync 卸 asyncpg - async_provider 剥 +asyncpg dialect 前缀让 LangGraph saver 能解析 URL 更新"一句话状态" + 标 PG live 验证（区分 testcontainers 路径仍未实跑）。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
7f17cb8fee |
chore(test): bump testcontainers Postgres to 17-alpine
\`make doctor\` 实测远程 Aliyun RDS = PostgreSQL 17.9。把 backend/tests/fixtures/postgres.py 的 testcontainers 镜像从 postgres:16-alpine 调到 postgres:17-alpine，与生产对齐。 同步把 STATUS.md "RDS 大版本对齐" 项标 done。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
ad22242ecc |
docs(impl): add Stage 0 progress dashboard (STATUS.md)
新增 docs/multi-tenant-redesign/03-impl/STATUS.md 作为 Stage 0 唯一的"现在 到哪了"权威来源： - 8 PR 状态表（PR0-PR2 ✅ merged，PR3-PR8 🟡 pending） - 用户必须跟进的事（live PG 实跑 / RDS 大版本对齐 / push origin / LOCK review / 远程 RDS 密码轮换） - 跳过/推迟的子任务（T1.10 docker-pending、T2.7 acked、T2.8 optional skipped、T2.9 backend/CLAUDE.md 待补） - 即将遇到的开放问题（PR4 alembic baseline / _ensure_admin_user 现状 / PG 大版本） - PR3 模式选择 trade-off（Inline vs Subagent-Driven） - 维护规则：完成 PR 后必更新；新 session 首先读本文件 填补 plan/impl notes 之外的空白：plan 是静态的，impl notes 是 per-PR 快照；STATUS 是跨 PR 的执行状态视图。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
1112a1971b |
docs(impl): PR2 implementation note + acceptance checklist
记录 Stage 0 PR2 的实施落点、关键 LOCK 决策、跟进项与 7 commit 列表。 验收：3087 passed + 23 skipped + 0 failed（vs PR1 基线 3085 + 2 新 default-backend 测试）。 Stage 0 PR2 T2.10. PR2 完成。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
85a14f4c05 |
docs(impl): PR1 implementation note + acceptance checklist
记录 Stage 0 PR1 的实施落点、关键 LOCK 决策、fixture 用法示例、跟进项与 10 commit 列表。验收：3085 passed + 23 skipped + 0 failed（无 regression）， PG smoke 在 docker daemon 起来后实跑（CI workflow 已配）。 Stage 0 PR1 T1.10. PR1 完成。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
a74b88a45c |
docs(superpowers): add Stage 0 multi-tenant foundation master plan
8 PR 串成的 stage-level 拆解（Postgres 切换 + workspaces schema + auth 扩字段 + 入口路由强校验 + CI boundary + headless schema 预留）。每个 PR 含 commit-sized TDD task list 和验收清单。执行时各 PR 独立 review checkpoint。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
89fe54cc07 |
docs(multi-tenant): 加入汇总索引、跨文档一致性修订、Postgres 切换前移到 Stage 0
* 新增 README.zh-CN.md 汇总索引：ADR 状态表 + Stage 0-4 业务目标 / 技术路径 / 验证方式 + Stage↔ADR 对照矩阵 + 不可逆决策一览 + 用语映射 + FAQ + 阅读路径 * 新增 workspace-schema-design.zh-CN.md（Stage 0 schema 锁定版）一并入库 * 7 份 ADR 顶部加"代码命名"映射行（tenant_id ↔ workspace_id） * ADR-002 §1 加分期落地提示，明确 K8s 推迟到 Stage 3 * ADR-005 §5"第 1/2 阶段"补出与 rollout Stage 2/3 的映射 * ADR-007 §4 加 /api/v1/ 反向链接；§8 加 tid → wid 字段名映射 * headless-api §0/§7 把"SaaS + on-prem 双主线"改为"SaaS 主线、schema 兼容 on-prem" * phased-rollout 去除重复的"Go/No-Go 进入 Stage 2"段 Postgres 切换从 Stage 1 提前到 Stage 0：Stage 0 已要 ALTER 4 张表加 workspace_id，先 SQLite 再 PG 是纯返工；Stage 0 没有生产数据，迁移阻力最小。 同步调整 phased-rollout / headless-api / phase-0-plan / workspace-schema-design / README 中的时间盒（Stage 0: 3-4→4-5 周；Stage 1: 10-15→8-13 周）、不可逆决策 清单、PR 顺序、轨道前置依赖。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
ecc9339ede |
docs(multi-tenant): 加入 Headless API 改造轨道（Pattern A/B）+ Stage 1 双轨修订
新增 headless-api-track 文档，把 DeerFlow 作为后端服务的两种集成 pattern 全部纳入设计： - Pattern A（业务 backend 代理）：业务系统 backend 用 API key 调， 浏览器走他们的 backend。覆盖 IM channels + server-to-server 集成。 - Pattern B（自研 web 浏览器直连）：业务 backend 调 /api/v1/auth/exchange-token 换 5-15 min 短期 JWT → 浏览器拿 JWT 直连 含 SSE。核心 4 件事：exchange-token endpoint、ServiceTokenAuthBackend （AuthMiddleware 第三条路径）、workspaces.allowed_origins + CORS 中间件、SSE 跨域验证。 - 不做 widget / iframe（纯 API）。 数据模型：service_accounts / api_keys / external_users 表（schema 在 Stage 0 末加上不阻塞）；身份模式三态（collapsed / external_passthrough / both）按 endpoint 分支。 Stage 1 改为三轨并行：付费 SaaS / Pattern A / Pattern B；时间盒从 6-10 周延到 10-15 周。Pattern B 依赖 Pattern A 完成，建议 Stage 1 末 1-2 周做。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
fd8d0d637d |
docs(multi-tenant): 加入 Stage 0 代码地图
把 Stage 0 必做项落到具体文件 + 行号，覆盖 7 个子系统： auth、user 数据模型、thread 入口路由、ThreadDataMiddleware + 路径系统、 仓储访问模式、setup / 注册流程、CSRF + ContextVar 注入。 文档结构：每节有【关键文件 + 行号】、【关键函数 / 类】、【当前数据流】、 【Stage 0 改动锚点】；外加改动影响面总览（每条 Stage 0 必做项映射到 具体代码位置）、不可逆决策落点、推荐阅读顺序（11 个文件，1.5-2.5h 粗读）。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
35ae97d0a6 |
docs(multi-tenant): Stage 2 加入 per-user skill 覆盖与 config 设计
补充团队 workspace 内不同成员的 skill 区分模型： - 启用状态：workspace 级默认 + per-user 覆盖 (final_enabled = user_override ?? workspace_default) - skill 私有配置（API key 等）：per-user 强制隔离 + KMS 加密 - 上传权限：仅 owner / admin，成员只能 enable/disable + 填自己 config PR 顺序新增第 8 步，依赖 RBAC + KMS 已就位。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
e78ed687a6 |
docs(multi-tenant): 加入按规模分期的落地方案（02-rollout）
基于已收敛的目标客户画像（个人用户为主 + 小团队 / 中心化 SaaS / freemium）把 7 份 ADR 拆成 5 期落地： - Stage 0：workspace 模型立起来 + auth 收紧（不可逆决策集中在此） - Stage 1：Postgres + Quota 必落（freemium 不上 quota = 信用卡递给攻击者） - Stage 2：DeerFlow 表 RLS、KMS、ObjectStorage S3、付费分层 - Stage 3：K8s sandbox + BYO key + audit DB 拆分 - Stage 4：SSO / custom domain / per-tenant DB（按 enterprise 客户合同驱动） 关键取舍：① Stage 0 末就把 workspace_id 列加到 SQLite，避免 Stage 1 切 Postgres 时再补；② Quota 比 ADR-003 原稿提前一档到 Stage 1；③ K8s sandbox 推到 Stage 3，AioSandbox + 出网白名单 + 资源限额撑到几千用户。 时间盒：最小可付费 4 个月（Stage 0+1）/ 风险可控增长 8 个月（+Stage 2） / 全功能 14 个月（+Stage 3）。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
8bdd308ea0 |
docs(multi-tenant): 据审计对齐 ADR-002/003/004/005 与 phase-0 计划
非 spike 驱动的对齐改动： - ADR-002/004/005：头部加现状提示 + 链接审计报告（沙箱出网/资源缺位、 token_version 已存在 MembershipCache 全新建、ObjectStorage/7 表/KMS 全部从 0 起） - ADR-003 LLM 计费：修正 TokenUsageMiddleware 当前只 log 不持久化的描述； 补充 create_chat_model sync→async 改造的连带影响说明 - phase-0 计划：新增 §3.5 底座先行（Postgres 测试夹具 / ObjectStorage Protocol / KMS 抽象 3 件并行做），时间盒 2 → 3 周；ADR-006/007 摘要 对齐；DoD 加底座骨架检查项 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
27c4f14233 |
docs(multi-tenant): 加入 ADR 审计 + spike，并据其修订 ADR-001/006/007
新增两份评审产出物： - adr-vs-code-audit：7 份 ADR 与现状代码的差异核对，标注每条假设是否成立 - adr-spike-langgraph-postgres：实测 langgraph-checkpoint-postgres==3.0.5 注入能力，确认不存在 connection_factory 参数，且 psycopg_pool 自带的 configure callback 不是 per-acquire hook 据 spike 与审计修订三份 ADR： - ADR-001 数据隔离：LangGraph 表改为应用层强校验 + threads_meta unique 约束兜底（不再挂 RLS、不 ALTER 表）；hook 点从 AssistantsCompat 修正 为 threads.py + thread_runs.py - ADR-006 运行时与渠道：§2.1 完全重写为应用层强校验；MCP OAuth token 从"无持久化进程内存"直接做加密 DB；channel store binding 改为新建 channel_bindings 表 - ADR-007 路由与前端：删除 Better Auth 假设（前端实际无此依赖），改为 扩展现有 auth/jwt.py TokenPayload 加 tid/role 字段 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
1445043649
|
dce5e9598b |
docs(multi-tenant): add ADRs and phase-0 plan for multi-tenant redesign
Add 7 ADRs and a phase-0 plan covering the multi-tenant redesign of DeerFlow, plus an architecture-overview snapshot of the current state. ADRs: - 001 data isolation: row-level tenant_id + Postgres RLS, including LangGraph-owned checkpoint tables (subquery RLS or column upgrade path). - 002 sandbox isolation: K8s namespace + gVisor + NetworkPolicy default- deny, threat model and pod spec defaults. - 003 LLM key & billing: hybrid platform/BYO with pessimistic reservation to handle the "ghost token" overflow on the last call, plus a usage category split for memory/title/summarization charges. - 004 RBAC: two-level (owner/admin/member), JWT-with-role + 30s LRU cache for reads, strict DB lookup for sensitive writes, token_version bump as the single revocation path. - 005 storage topology: Postgres (structured) + S3-compatible object store (large objects) + emptyDir (ephemeral); explicit treatment of the extensions_config.json migration's downstream effects. - 006 runtime & channel tenancy: per-tenant MCP cache, dual-track skills loader, sandbox provider routing by namespace, internal LLM call billing, IM channel-to-tenant binding model. - 007 routing & frontend: path-slug URL form, JWT-only API auth, TenantProvider, hard-reload tenant switch, Better Auth integration. These docs are decision records; no code changes are included. |
||
|
rayhpeng
|
56d5fa3337 |
feat(persistence):Unified persistence layer with event store, feedback, and rebase cleanup (#2134)
* feat(persistence): add unified persistence layer with event store, token tracking, and feedback (#1930) * feat(persistence): add SQLAlchemy 2.0 async ORM scaffold Introduce a unified database configuration (DatabaseConfig) that controls both the LangGraph checkpointer and the DeerFlow application persistence layer from a single `database:` config section. New modules: - deerflow.config.database_config — Pydantic config with memory/sqlite/postgres backends - deerflow.persistence — async engine lifecycle, DeclarativeBase with to_dict mixin, Alembic skeleton - deerflow.runtime.runs.store — RunStore ABC + MemoryRunStore implementation Gateway integration initializes/tears down the persistence engine in the existing langgraph_runtime() context manager. Legacy checkpointer config is preserved for backward compatibility. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(persistence): add RunEventStore ABC + MemoryRunEventStore Phase 2-A prerequisite for event storage: adds the unified run event stream interface (RunEventStore) with an in-memory implementation, RunEventsConfig, gateway integration, and comprehensive tests (27 cases). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(persistence): add ORM models, repositories, DB/JSONL event stores, RunJournal, and API endpoints Phase 2-B: run persistence + event storage + token tracking. - ORM models: RunRow (with token fields), ThreadMetaRow, RunEventRow - RunRepository implements RunStore ABC via SQLAlchemy ORM - ThreadMetaRepository with owner access control - DbRunEventStore with trace content truncation and cursor pagination - JsonlRunEventStore with per-run files and seq recovery from disk - RunJournal (BaseCallbackHandler) captures LLM/tool/lifecycle events, accumulates token usage by caller type, buffers and flushes to store - RunManager now accepts optional RunStore for persistent backing - Worker creates RunJournal, writes human_message, injects callbacks - Gateway deps use factory functions (RunRepository when DB available) - New endpoints: messages, run messages, run events, token-usage - ThreadCreateRequest gains assistant_id field - 92 tests pass (33 new), zero regressions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(persistence): add user feedback + follow-up run association Phase 2-C: feedback and follow-up tracking. - FeedbackRow ORM model (rating +1/-1, optional message_id, comment) - FeedbackRepository with CRUD, list_by_run/thread, aggregate stats - Feedback API endpoints: create, list, stats, delete - follow_up_to_run_id in RunCreateRequest (explicit or auto-detected from latest successful run on the thread) - Worker writes follow_up_to_run_id into human_message event metadata - Gateway deps: feedback_repo factory + getter - 17 new tests (14 FeedbackRepository + 3 follow-up association) - 109 total tests pass, zero regressions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test+config: comprehensive Phase 2 test coverage + deprecate checkpointer config - config.example.yaml: deprecate standalone checkpointer section, activate unified database:sqlite as default (drives both checkpointer + app data) - New: test_thread_meta_repo.py (14 tests) — full ThreadMetaRepository coverage including check_access owner logic, list_by_owner pagination - Extended test_run_repository.py (+4 tests) — completion preserves fields, list ordering desc, limit, owner_none returns all - Extended test_run_journal.py (+8 tests) — on_chain_error, track_tokens=false, middleware no ai_message, unknown caller tokens, convenience fields, tool_error, non-summarization custom event - Extended test_run_event_store.py (+7 tests) — DB batch seq continuity, make_run_event_store factory (memory/db/jsonl/fallback/unknown) - Extended test_phase2b_integration.py (+4 tests) — create_or_reject persists, follow-up metadata, summarization in history, full DB-backed lifecycle - Fixed DB integration test to use proper fake objects (not MagicMock) for JSON-serializable metadata - 157 total Phase 2 tests pass, zero regressions Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * config: move default sqlite_dir to .deer-flow/data Keep SQLite databases alongside other DeerFlow-managed data (threads, memory) under the .deer-flow/ directory instead of a top-level ./data folder. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(persistence): remove UTFJSON, use engine-level json_serializer + datetime.now() - Replace custom UTFJSON type with standard sqlalchemy.JSON in all ORM models. Add json_serializer=json.dumps(ensure_ascii=False) to all create_async_engine calls so non-ASCII text (Chinese etc.) is stored as-is in both SQLite and Postgres. - Change ORM datetime defaults from datetime.now(UTC) to datetime.now(), remove UTC imports. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(gateway): simplify deps.py with getter factory + inline repos - Replace 6 identical getter functions with _require() factory. - Inline 3 _make_*_repo() factories into langgraph_runtime(), call get_session_factory() once instead of 3 times. - Add thread_meta upsert in start_run (services.py). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(docker): add UV_EXTRAS build arg for optional dependencies Support installing optional dependency groups (e.g. postgres) at Docker build time via UV_EXTRAS build arg: UV_EXTRAS=postgres docker compose build Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(journal): fix flush, token tracking, and consolidate tests RunJournal fixes: - _flush_sync: retain events in buffer when no event loop instead of dropping them; worker's finally block flushes via async flush(). - on_llm_end: add tool_calls filter and caller=="lead_agent" guard for ai_message events; mark message IDs for dedup with record_llm_usage. - worker.py: persist completion data (tokens, message count) to RunStore in finally block. Model factory: - Auto-inject stream_usage=True for BaseChatOpenAI subclasses with custom api_base, so usage_metadata is populated in streaming responses. Test consolidation: - Delete test_phase2b_integration.py (redundant with existing tests). - Move DB-backed lifecycle test into test_run_journal.py. - Add tests for stream_usage injection in test_model_factory.py. - Clean up executor/task_tool dead journal references. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(events): widen content type to str|dict in all store backends Allow event content to be a dict (for structured OpenAI-format messages) in addition to plain strings. Dict values are JSON-serialized for the DB backend and deserialized on read; memory and JSONL backends handle dicts natively. Trace truncation now serializes dicts to JSON before measuring. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(events): use metadata flag instead of heuristic for dict content detection Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(converters): add LangChain-to-OpenAI message format converters Pure functions langchain_to_openai_message, langchain_to_openai_completion, langchain_messages_to_openai, and _infer_finish_reason for converting LangChain BaseMessage objects to OpenAI Chat Completions format, used by RunJournal for event storage. 15 unit tests added. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(converters): handle empty list content as null, clean up test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(events): human_message content uses OpenAI user message format Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(events): ai_message uses OpenAI format, add ai_tool_call message event - ai_message content now uses {"role": "assistant", "content": "..."} format - New ai_tool_call message event emitted when lead_agent LLM responds with tool_calls - ai_tool_call uses langchain_to_openai_message converter for consistent format - Both events include finish_reason in metadata ("stop" or "tool_calls") Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(events): add tool_result message event with OpenAI tool message format Cache tool_call_id from on_tool_start keyed by run_id as fallback for on_tool_end, then emit a tool_result message event (role=tool, tool_call_id, content) after each successful tool completion. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(events): summary content uses OpenAI system message format Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(events): replace llm_start/llm_end with llm_request/llm_response in OpenAI format Add on_chat_model_start to capture structured prompt messages as llm_request events. Replace llm_end trace events with llm_response using OpenAI Chat Completions format. Track llm_call_index to pair request/response events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(events): add record_middleware method for middleware trace events Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(events): add full run sequence integration test for OpenAI content format Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(events): align message events with checkpoint format and add middleware tag injection - Message events (ai_message, ai_tool_call, tool_result, human_message) now use BaseMessage.model_dump() format, matching LangGraph checkpoint values.messages - on_tool_end extracts tool_call_id/name/status from ToolMessage objects - on_tool_error now emits tool_result message events with error status - record_middleware uses middleware:{tag} event_type and middleware category - Summarization custom events use middleware:summarize category - TitleMiddleware injects middleware:title tag via get_config() inheritance - SummarizationMiddleware model bound with middleware:summarize tag - Worker writes human_message using HumanMessage.model_dump() Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(threads): switch search endpoint to threads_meta table and sync title - POST /api/threads/search now queries threads_meta table directly, removing the two-phase Store + Checkpointer scan approach - Add ThreadMetaRepository.search() with metadata/status filters - Add ThreadMetaRepository.update_display_name() for title sync - Worker syncs checkpoint title to threads_meta.display_name on run completion - Map display_name to values.title in search response for API compatibility Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(threads): history endpoint reads messages from event store - POST /api/threads/{thread_id}/history now combines two data sources: checkpointer for checkpoint_id, metadata, title, thread_data; event store for messages (complete history, not truncated by summarization) - Strip internal LangGraph metadata keys from response - Remove full channel_values serialization in favor of selective fields Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove duplicate optional-dependencies header in pyproject.toml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(middleware): pass tagged config to TitleMiddleware ainvoke call Without the config, the middleware:title tag was not injected, causing the LLM response to be recorded as a lead_agent ai_message in run_events. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve merge conflict in .env.example Keep both DATABASE_URL (from persistence-scaffold) and WECOM credentials (from main) after the merge. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(persistence): address review feedback on PR #1851 - Fix naive datetime.now() → datetime.now(UTC) in all ORM models - Fix seq race condition in DbRunEventStore.put() with FOR UPDATE and UNIQUE(thread_id, seq) constraint - Encapsulate _store access in RunManager.update_run_completion() - Deduplicate _store.put() logic in RunManager via _persist_to_store() - Add update_run_completion to RunStore ABC + MemoryRunStore - Wire follow_up_to_run_id through the full create path - Add error recovery to RunJournal._flush_sync() lost-event scenario - Add migration note for search_threads breaking change - Fix test_checkpointer_none_fix mock to set database=None Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(persistence): address 22 review comments from CodeQL, Copilot, and Code Quality Bug fixes: - Sanitize log params to prevent log injection (CodeQL) - Reset threads_meta.status to idle/error when run completes - Attach messages only to latest checkpoint in /history response - Write threads_meta on POST /threads so new threads appear in search Lint fixes: - Remove unused imports (journal.py, migrations/env.py, test_converters.py) - Convert lambda to named function (engine.py, Ruff E731) - Remove unused logger definitions in repos (Ruff F841) - Add logging to JSONL decode errors and empty except blocks - Separate assert side-effects in tests (CodeQL) - Remove unused local variables in tests (Ruff F841) - Fix max_trace_content truncation to use byte length, not char length Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * style: apply ruff format to persistence and runtime files Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Potential fix for pull request finding 'Statement has no effect' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com> * refactor(runtime): introduce RunContext to reduce run_agent parameter bloat Extract checkpointer, store, event_store, run_events_config, thread_meta_repo, and follow_up_to_run_id into a frozen RunContext dataclass. Add get_run_context() in deps.py to build the base context from app.state singletons. start_run() uses dataclasses.replace() to enrich per-run fields before passing ctx to run_agent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(gateway): move sanitize_log_param to app/gateway/utils.py Extract the log-injection sanitizer from routers/threads.py into a shared utils module and rename to sanitize_log_param (public API). Eliminates the reverse service → router import in services.py. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: use SQL aggregation for feedback stats and thread token usage Replace Python-side counting in FeedbackRepository.aggregate_by_run with a single SELECT COUNT/SUM query. Add RunStore.aggregate_tokens_by_thread abstract method with SQL GROUP BY implementation in RunRepository and Python fallback in MemoryRunStore. Simplify the thread_token_usage endpoint to delegate to the new method, eliminating the limit=10000 truncation risk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: annotate DbRunEventStore.put() as low-frequency path Add docstring clarifying that put() opens a per-call transaction with FOR UPDATE and should only be used for infrequent writes (currently just the initial human_message event). High-throughput callers should use put_batch() instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(threads): fall back to Store search when ThreadMetaRepository is unavailable When database.backend=memory (default) or no SQL session factory is configured, search_threads now queries the LangGraph Store instead of returning 503. Returns empty list if neither Store nor repo is available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(persistence): introduce ThreadMetaStore ABC for backend-agnostic thread metadata Add ThreadMetaStore abstract base class with create/get/search/update/delete interface. ThreadMetaRepository (SQL) now inherits from it. New MemoryThreadMetaStore wraps LangGraph BaseStore for memory-mode deployments. deps.py now always provides a non-None thread_meta_repo, eliminating all `if thread_meta_repo is not None` guards in services.py, worker.py, and routers/threads.py. search_threads no longer needs a Store fallback branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor(history): read messages from checkpointer instead of RunEventStore The /history endpoint now reads messages directly from the checkpointer's channel_values (the authoritative source) instead of querying RunEventStore.list_messages(). The RunEventStore API is preserved for other consumers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(persistence): address new Copilot review comments - feedback.py: validate thread_id/run_id before deleting feedback - jsonl.py: add path traversal protection with ID validation - run_repo.py: parse `before` to datetime for PostgreSQL compat - thread_meta_repo.py: fix pagination when metadata filter is active - database_config.py: use resolve_path for sqlite_dir consistency Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Implement skill self-evolution and skill_manage flow (#1874) * chore: ignore .worktrees directory * Add skill_manage self-evolution flow * Fix CI regressions for skill_manage * Address PR review feedback for skill evolution * fix(skill-evolution): preserve history on delete * fix(skill-evolution): tighten scanner fallbacks * docs: add skill_manage e2e evidence screenshot * fix(skill-manage): avoid blocking fs ops in session runtime --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> * fix(config): resolve sqlite_dir relative to CWD, not Paths.base_dir resolve_path() resolves relative to Paths.base_dir (.deer-flow), which double-nested the path to .deer-flow/.deer-flow/data/app.db. Use Path.resolve() (CWD-relative) instead. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Feature/feishu receive file (#1608) * feat(feishu): add channel file materialization hook for inbound messages - Introduce Channel.receive_file(msg, thread_id) as a base method for file materialization; default is no-op. - Implement FeishuChannel.receive_file to download files/images from Feishu messages, save to sandbox, and inject virtual paths into msg.text. - Update ChannelManager to call receive_file for any channel if msg.files is present, enabling downstream model access to user-uploaded files. - No impact on Slack/Telegram or other channels (they inherit the default no-op). * style(backend): format code with ruff for lint compliance - Auto-formatted packages/harness/deerflow/agents/factory.py and tests/test_create_deerflow_agent.py using `ruff format` - Ensured both files conform to project linting standards - Fixes CI lint check failures caused by code style issues * fix(feishu): handle file write operation asynchronously to prevent blocking * fix(feishu): rename GetMessageResourceRequest to _GetMessageResourceRequest and remove redundant code * test(feishu): add tests for receive_file method and placeholder replacement * fix(manager): remove unnecessary type casting for channel retrieval * fix(feishu): update logging messages to reflect resource handling instead of image * fix(feishu): sanitize filename by replacing invalid characters in file uploads * fix(feishu): improve filename sanitization and reorder image key handling in message processing * fix(feishu): add thread lock to prevent filename conflicts during file downloads * fix(test): correct bad merge in test_feishu_parser.py * chore: run ruff and apply formatting cleanup fix(feishu): preserve rich-text attachment order and improve fallback filename handling * fix(docker): restore gateway env vars and fix langgraph empty arg issue (#1915) Two production docker-compose.yaml bugs prevent `make up` from working: 1. Gateway missing DEER_FLOW_CONFIG_PATH and DEER_FLOW_EXTENSIONS_CONFIG_PATH environment overrides. Added in |
||
|
DanielWalnut
|
888f7bfb9d |
Implement skill self-evolution and skill_manage flow (#1874)
* chore: ignore .worktrees directory * Add skill_manage self-evolution flow * Fix CI regressions for skill_manage * Address PR review feedback for skill evolution * fix(skill-evolution): preserve history on delete * fix(skill-evolution): tighten scanner fallbacks * docs: add skill_manage e2e evidence screenshot * fix(skill-manage): avoid blocking fs ops in session runtime --------- Co-authored-by: Willem Jiang <willem.jiang@gmail.com> |
||
|
totoyang
|
2d1f90d5dc |
feat(tracing): add optional Langfuse support (#1717)
* feat(tracing): add optional Langfuse support * Fix tracing fail-fast behavior for explicitly enabled providers * fix(lint) |
||
|
DanielWalnut
|
76803b826f |
refactor: split backend into harness (deerflow.*) and app (app.*) (#1131)
* refactor: extract shared utils to break harness→app cross-layer imports Move _validate_skill_frontmatter to src/skills/validation.py and CONVERTIBLE_EXTENSIONS + convert_file_to_markdown to src/utils/file_conversion.py. This eliminates the two reverse dependencies from client.py (harness layer) into gateway/routers/ (app layer), preparing for the harness/app package split. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: split backend/src into harness (deerflow.*) and app (app.*) Physically split the monolithic backend/src/ package into two layers: - **Harness** (`packages/harness/deerflow/`): publishable agent framework package with import prefix `deerflow.*`. Contains agents, sandbox, tools, models, MCP, skills, config, and all core infrastructure. - **App** (`app/`): unpublished application code with import prefix `app.*`. Contains gateway (FastAPI REST API) and channels (IM integrations). Key changes: - Move 13 harness modules to packages/harness/deerflow/ via git mv - Move gateway + channels to app/ via git mv - Rename all imports: src.* → deerflow.* (harness) / app.* (app layer) - Set up uv workspace with deerflow-harness as workspace member - Update langgraph.json, config.example.yaml, all scripts, Docker files - Add build-system (hatchling) to harness pyproject.toml - Add PYTHONPATH=. to gateway startup commands for app.* resolution - Update ruff.toml with known-first-party for import sorting - Update all documentation to reflect new directory structure Boundary rule enforced: harness code never imports from app. All 429 tests pass. Lint clean. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: add harness→app boundary check test and update docs Add test_harness_boundary.py that scans all Python files in packages/harness/deerflow/ and fails if any `from app.*` or `import app.*` statement is found. This enforces the architectural rule that the harness layer never depends on the app layer. Update CLAUDE.md to document the harness/app split architecture, import conventions, and the boundary enforcement test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add config versioning with auto-upgrade on startup When config.example.yaml schema changes, developers' local config.yaml files can silently become outdated. This adds a config_version field and auto-upgrade mechanism so breaking changes (like src.* → deerflow.* renames) are applied automatically before services start. - Add config_version: 1 to config.example.yaml - Add startup version check warning in AppConfig.from_file() - Add scripts/config-upgrade.sh with migration registry for value replacements - Add `make config-upgrade` target - Auto-run config-upgrade in serve.sh and start-daemon.sh before starting services - Add config error hints in service failure messages Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix comments * fix: update src.* import in test_sandbox_tools_security to deerflow.* Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: handle empty config and search parent dirs for config.example.yaml Address Copilot review comments on PR #1131: - Guard against yaml.safe_load() returning None for empty config files - Search parent directories for config.example.yaml instead of only looking next to config.yaml, fixing detection in common setups Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: correct skills root path depth and config_version type coercion - loader.py: fix get_skills_root_path() to use 5 parent levels (was 3) after harness split, file lives at packages/harness/deerflow/skills/ so parent×3 resolved to backend/packages/harness/ instead of backend/ - app_config.py: coerce config_version to int() before comparison in _check_config_version() to prevent TypeError when YAML stores value as string (e.g. config_version: "1") - tests: add regression tests for both fixes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update test imports from src.* to deerflow.*/app.* after harness refactor Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
LofiSu
|
c8f7bc28e1 |
docs: 添加技能名称冲突修复的详细文档
- 记录 public 和 custom 技能同名冲突问题的解决方案
- 详细说明所有代码改动（后端配置、API、前端）
- 包含配置格式变更、API 变更说明
- 标注已知问题暂时保留，待后续版本修复
- 提供测试建议和回滚方案
相关改动：
- 使用组合键 {category}:{name} 存储配置
- API 支持可选的 category 查询参数
- 添加类别内重复技能名称检查
- 前端传递 category 参数确保唯一性
Co-authored-by: Cursor <cursoragent@cursor.com>
|
||
|
LofiSu
|
46048c76ce |
chore: 移除所有 Citations 相关逻辑，为后续重构做准备
- Backend: 删除 lead_agent / general_purpose 中的 citations_format 与引用相关 reminder；artifacts 下载不再对 markdown 做 citation 清洗，统一走 FileResponse，保留 Response 用于二进制 inline - Frontend: 删除 core/citations 模块、inline-citation、safe-citation-content；新增 MarkdownContent 仅做 Markdown 渲染；消息/artifact 预览与复制均使用原始 content - i18n: 移除 citations 命名空间（loadingCitations、loadingCitationsWithCount） - 技能与 demo: 措辞改为 references，demo 数据去掉 <citations> 块 - 文档: 更新 CLAUDE/AGENTS/README 描述，新增按文件 diff 的代码变更总结 Co-authored-by: Cursor <cursoragent@cursor.com> |