feat(eval): add report quality evaluation module and UI integration (#776)

* feat(eval): add report quality evaluation module

Addresses issue #773 - How to evaluate generated report quality objectively.

This module provides two evaluation approaches:
1. Automated metrics (no LLM required):
   - Citation count and source diversity
   - Word count compliance per report style
   - Section structure validation
   - Image inclusion tracking

2. LLM-as-Judge evaluation:
   - Factual accuracy scoring
   - Completeness assessment
   - Coherence evaluation
   - Relevance and citation quality checks

The combined evaluator provides a final score (1-10) and letter grade (A+ to F).

Files added:
- src/eval/__init__.py
- src/eval/metrics.py
- src/eval/llm_judge.py
- src/eval/evaluator.py
- tests/unit/eval/test_metrics.py
- tests/unit/eval/test_evaluator.py

* feat(eval): integrate report evaluation with web UI

This commit adds the web UI integration for the evaluation module:

Backend:
- Add EvaluateReportRequest/Response models in src/server/eval_request.py
- Add /api/report/evaluate endpoint to src/server/app.py

Frontend:
- Add evaluateReport API function in web/src/core/api/evaluate.ts
- Create EvaluationDialog component with grade badge, metrics display,
  and optional LLM deep evaluation
- Add evaluation button (graduation cap icon) to research-block.tsx toolbar
- Add i18n translations for English and Chinese

The evaluation UI allows users to:
1. View quick metrics-only evaluation (instant)
2. Optionally run deep LLM-based evaluation for detailed analysis
3. See grade (A+ to F), score (1-10), and metric breakdown

* feat(eval): improve evaluation reliability and add LLM judge tests

- Extract MAX_REPORT_LENGTH constant in llm_judge.py for maintainability
- Add comprehensive unit tests for LLMJudge class (parse_response,
  calculate_weighted_score, evaluate with mocked LLM)
- Pass reportStyle prop to EvaluationDialog for accurate evaluation criteria
- Add researchQueries store map to reliably associate queries with research
- Add getResearchQuery helper to retrieve query by researchId
- Remove unused imports in test_metrics.py

* fix(eval): use resolveServiceURL for evaluate API endpoint

The evaluateReport function was using a relative URL '/api/report/evaluate'
which sent requests to the Next.js server instead of the FastAPI backend.
Changed to use resolveServiceURL() consistent with other API functions.

* fix: improve type accuracy and React hooks in evaluation components

- Fix get_word_count_target return type from Optional[Dict] to Dict since it always returns a value via default fallback
- Fix useEffect dependency issue in EvaluationDialog using useRef to prevent unwanted re-evaluations
- Add aria-label to GradeBadge for screen reader accessibility
This commit is contained in:
Willem Jiang
2025-12-25 21:55:48 +08:00
committed by GitHub
parent 84a7f7815c
commit 8d9d767051
17 changed files with 2103 additions and 2 deletions
+27 -1
View File
@@ -16,6 +16,7 @@ import { jsPDF } from "jspdf";
import {
Check,
Copy,
GraduationCap,
Headphones,
Pencil,
Undo2,
@@ -43,9 +44,10 @@ import {
} from "~/components/ui/dropdown-menu";
import { Tabs, TabsContent, TabsList, TabsTrigger } from "~/components/ui/tabs";
import { useReplay } from "~/core/replay";
import { closeResearch, listenToPodcast, useStore } from "~/core/store";
import { closeResearch, getResearchQuery, listenToPodcast, useStore, useSettingsStore } from "~/core/store";
import { cn } from "~/lib/utils";
import { EvaluationDialog } from "./evaluation-dialog";
import { ResearchActivitiesBlock } from "./research-activities-block";
import { ResearchReportBlock } from "./research-report-block";
@@ -84,6 +86,7 @@ export function ResearchBlock({
const [editing, setEditing] = useState(false);
const [isDownloading, setIsDownloading] = useState(false);
const [copied, setCopied] = useState(false);
const [showEvaluation, setShowEvaluation] = useState(false);
const handleCopy = useCallback(() => {
if (!reportId) {
return;
@@ -676,6 +679,16 @@ ${htmlContent}
{copied ? <Check /> : <Copy />}
</Button>
</Tooltip>
<Tooltip title={t("evaluateReport")}>
<Button
className="text-gray-400"
size="icon"
variant="ghost"
onClick={() => setShowEvaluation(true)}
>
<GraduationCap />
</Button>
</Tooltip>
<DropdownMenu>
<Tooltip title={t("downloadReport")}>
<DropdownMenuTrigger asChild>
@@ -796,6 +809,19 @@ ${htmlContent}
</TabsContent>
</Tabs>
</Card>
{/* Evaluation Dialog */}
{reportId && researchId && (
<EvaluationDialog
open={showEvaluation}
onOpenChange={setShowEvaluation}
reportContent={
useStore.getState().messages.get(reportId)?.content ?? ""
}
query={getResearchQuery(researchId)}
reportStyle={useSettingsStore.getState().general.reportStyle.toLowerCase()}
/>
)}
</div>
);
}