docs(evals): S01–S10 标注集 v1、评测工具和评测报告

- evals/gold:1,887 个标注单元(AI 标注、逐条裁定,未经人工复核),规范见 GUIDELINES.md
- prepare / run_all / score / gold:导出文字层、批量抽取、打分、标注定位
- docs/architecture/extract-eval.md:修复前后的结果(三项目标全部达标)和剩余问题
- decisions.md 记 M1(只用 Flash)和 Q1–Q4;架构 §13 SK13;AGENTS.md 进度

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
YANG JIANKUAN
2026-11-03 23:19:30 +08:00
parent f98fd51128
commit 602703c47c
21 changed files with 29427 additions and 18 deletions

18
evals/run_all.sh Executable file
View File

@@ -0,0 +1,18 @@
#!/usr/bin/env bash
# 对 S01–S10 跑全部抽取任务(输入是 prepare.py 生成的规范化 PDF),结果在 evals/out/runs/<名字>/<ID>/。
# 用法(仓库根目录):evals/run_all.sh <名字> [run_extract.py 的其他参数]
# 换模型:BD_AI_MODELS='{"reasoning": "deepseek-v4-pro"}' evals/run_all.sh pro
set -euo pipefail
name=$1; shift
root=$(cd "$(dirname "$0")/.." && pwd)
cd "$root/server"
for id in S01 S02 S03 S04 S05 S06 S07 S08 S09 S10; do
echo "$id"
../.venv/bin/python ../evals/run_extract.py "../evals/out/docs/$id/normalized.pdf" "../evals/out/runs/$name/$id" "$@" \
> /dev/null 2> "../evals/out/runs/$name.$id.log" &
# 同时最多 3 份(每份内部 8 路并发调用模型)
# macOS 的 bash 3.2 在 $(…) 子 shell 里看不到后台任务,先写到文件再数
while jobs -r > "$root/evals/out/runs/.jobs" && [ "$(wc -l < "$root/evals/out/runs/.jobs")" -ge 3 ]; do sleep 2; done
done
wait
echo done