feat(eval): add report quality evaluation module and UI integration (#776)
* feat(eval): add report quality evaluation module Addresses issue #773 - How to evaluate generated report quality objectively. This module provides two evaluation approaches: 1. Automated metrics (no LLM required): - Citation count and source diversity - Word count compliance per report style - Section structure validation - Image inclusion tracking 2. LLM-as-Judge evaluation: - Factual accuracy scoring - Completeness assessment - Coherence evaluation - Relevance and citation quality checks The combined evaluator provides a final score (1-10) and letter grade (A+ to F). Files added: - src/eval/__init__.py - src/eval/metrics.py - src/eval/llm_judge.py - src/eval/evaluator.py - tests/unit/eval/test_metrics.py - tests/unit/eval/test_evaluator.py * feat(eval): integrate report evaluation with web UI This commit adds the web UI integration for the evaluation module: Backend: - Add EvaluateReportRequest/Response models in src/server/eval_request.py - Add /api/report/evaluate endpoint to src/server/app.py Frontend: - Add evaluateReport API function in web/src/core/api/evaluate.ts - Create EvaluationDialog component with grade badge, metrics display, and optional LLM deep evaluation - Add evaluation button (graduation cap icon) to research-block.tsx toolbar - Add i18n translations for English and Chinese The evaluation UI allows users to: 1. View quick metrics-only evaluation (instant) 2. Optionally run deep LLM-based evaluation for detailed analysis 3. See grade (A+ to F), score (1-10), and metric breakdown * feat(eval): improve evaluation reliability and add LLM judge tests - Extract MAX_REPORT_LENGTH constant in llm_judge.py for maintainability - Add comprehensive unit tests for LLMJudge class (parse_response, calculate_weighted_score, evaluate with mocked LLM) - Pass reportStyle prop to EvaluationDialog for accurate evaluation criteria - Add researchQueries store map to reliably associate queries with research - Add getResearchQuery helper to retrieve query by researchId - Remove unused imports in test_metrics.py * fix(eval): use resolveServiceURL for evaluate API endpoint The evaluateReport function was using a relative URL '/api/report/evaluate' which sent requests to the Next.js server instead of the FastAPI backend. Changed to use resolveServiceURL() consistent with other API functions. * fix: improve type accuracy and React hooks in evaluation components - Fix get_word_count_target return type from Optional[Dict] to Dict since it always returns a value via default fallback - Fix useEffect dependency issue in EvaluationDialog using useRef to prevent unwanted re-evaluations - Add aria-label to GradeBadge for screen reader accessibility
This commit is contained in:
@@ -16,6 +16,7 @@ import { jsPDF } from "jspdf";
|
||||
import {
|
||||
Check,
|
||||
Copy,
|
||||
GraduationCap,
|
||||
Headphones,
|
||||
Pencil,
|
||||
Undo2,
|
||||
@@ -43,9 +44,10 @@ import {
|
||||
} from "~/components/ui/dropdown-menu";
|
||||
import { Tabs, TabsContent, TabsList, TabsTrigger } from "~/components/ui/tabs";
|
||||
import { useReplay } from "~/core/replay";
|
||||
import { closeResearch, listenToPodcast, useStore } from "~/core/store";
|
||||
import { closeResearch, getResearchQuery, listenToPodcast, useStore, useSettingsStore } from "~/core/store";
|
||||
import { cn } from "~/lib/utils";
|
||||
|
||||
import { EvaluationDialog } from "./evaluation-dialog";
|
||||
import { ResearchActivitiesBlock } from "./research-activities-block";
|
||||
import { ResearchReportBlock } from "./research-report-block";
|
||||
|
||||
@@ -84,6 +86,7 @@ export function ResearchBlock({
|
||||
const [editing, setEditing] = useState(false);
|
||||
const [isDownloading, setIsDownloading] = useState(false);
|
||||
const [copied, setCopied] = useState(false);
|
||||
const [showEvaluation, setShowEvaluation] = useState(false);
|
||||
const handleCopy = useCallback(() => {
|
||||
if (!reportId) {
|
||||
return;
|
||||
@@ -676,6 +679,16 @@ ${htmlContent}
|
||||
{copied ? <Check /> : <Copy />}
|
||||
</Button>
|
||||
</Tooltip>
|
||||
<Tooltip title={t("evaluateReport")}>
|
||||
<Button
|
||||
className="text-gray-400"
|
||||
size="icon"
|
||||
variant="ghost"
|
||||
onClick={() => setShowEvaluation(true)}
|
||||
>
|
||||
<GraduationCap />
|
||||
</Button>
|
||||
</Tooltip>
|
||||
<DropdownMenu>
|
||||
<Tooltip title={t("downloadReport")}>
|
||||
<DropdownMenuTrigger asChild>
|
||||
@@ -796,6 +809,19 @@ ${htmlContent}
|
||||
</TabsContent>
|
||||
</Tabs>
|
||||
</Card>
|
||||
|
||||
{/* Evaluation Dialog */}
|
||||
{reportId && researchId && (
|
||||
<EvaluationDialog
|
||||
open={showEvaluation}
|
||||
onOpenChange={setShowEvaluation}
|
||||
reportContent={
|
||||
useStore.getState().messages.get(reportId)?.content ?? ""
|
||||
}
|
||||
query={getResearchQuery(researchId)}
|
||||
reportStyle={useSettingsStore.getState().general.reportStyle.toLowerCase()}
|
||||
/>
|
||||
)}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user