fix(llm): forbid year hallucination in comic identify prompt (bookshelf-3493) #985

Merged

zombor merged 1 commit from bd-bookshelf-3493 into main 2026-07-06 20:31:17 +00:00

Conversation 2 Commits 1 Files changed 2 +60 -2

zombor commented 2026-07-06 20:14:22 +00:00

Owner

Copy link

Summary

• Strengthens comicIdentifyPrompt in internal/metadata/llm/llm.go to FORBID the vision model from guessing or inferring the cover year from prior knowledge
• Old phrasing: "Read the cover date if visible ... Omit the parenthetical year entirely if the cover date is not visible" — too weak; model hallucinated year=1990 for a 1988 Millennium #4 comic whose cover has no legible date, driving a wrong ComicVine search
• New phrasing: "ONLY include a year if a cover date is CLEARLY and LEGIBLY PRINTED on the cover image. Do NOT guess, infer, or use any prior knowledge. It is better to omit the year than to guess."
• Adds black-box httptest tests (package llm_test) that capture the outgoing request body and assert: (a) new emphatic instructions are present, (b) old weak phrasing is absent
• No changes to parsing, the book prompt, or the output contract (comicIssueRe already handles absent year → empty string)

Test plan

• make test passes
• New Describe("Identify — comic prompt body") specs assert the request body contains "Do NOT guess", "LEGIBLY PRINTED", "better to omit", and does NOT contain the old weak sentences
• All existing Identify specs continue to pass (output contract unchanged)
• golangci-lint run ./internal/metadata/llm/... — 0 issues

Closes bead bookshelf-3493 on merge.

## Summary - Strengthens `comicIdentifyPrompt` in `internal/metadata/llm/llm.go` to FORBID the vision model from guessing or inferring the cover year from prior knowledge - Old phrasing: "Read the cover date if visible ... Omit the parenthetical year entirely if the cover date is not visible" — too weak; model hallucinated year=1990 for a 1988 Millennium #4 comic whose cover has no legible date, driving a wrong ComicVine search - New phrasing: "ONLY include a year if a cover date is CLEARLY and LEGIBLY PRINTED on the cover image. Do NOT guess, infer, or use any prior knowledge. It is better to omit the year than to guess." - Adds black-box httptest tests (`package llm_test`) that capture the outgoing request body and assert: (a) new emphatic instructions are present, (b) old weak phrasing is absent - No changes to parsing, the book prompt, or the output contract (`comicIssueRe` already handles absent year → empty string) ## Test plan - [x] `make test` passes - [x] New `Describe("Identify — comic prompt body")` specs assert the request body contains "Do NOT guess", "LEGIBLY PRINTED", "better to omit", and does NOT contain the old weak sentences - [x] All existing `Identify` specs continue to pass (output contract unchanged) - [x] `golangci-lint run ./internal/metadata/llm/...` — 0 issues Closes bead bookshelf-3493 on merge.

zombor added 1 commit 2026-07-06 20:14:22 +00:00

fix(llm): forbid year hallucination in comic identify prompt (bookshelf-3493) ... 93c3254008

Strengthen comicIdentifyPrompt to FORBID guessing or inferring the cover
year from prior knowledge. The old prompt said only "Omit the parenthetical
year if not visible" — too weak; the vision model hallucinated year=1990 for
a 1988 comic whose cover has no legible date.

New wording: "ONLY include a year if a cover date is CLEARLY and LEGIBLY
PRINTED on the cover image. Do NOT guess, infer, or use any prior knowledge.
It is better to omit the year than to guess."

Add black-box httptest tests that capture the outgoing request body and
assert the strengthened instruction is present and the old weak phrasing
is absent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

zombor commented 2026-07-06 20:20:32 +00:00

Author

Owner

Copy link

Security Review — bookshelf-3493

Diff scope: prompt-string edit in internal/metadata/llm/llm.go + accompanying test in llm_test.go. No new endpoint, HTTP surface, external call, or dependency introduced.

Checks performed:

1. Secrets / credentials / API keys — none introduced. The prompt is a plain instruction string; the test uses llm.Config{BaseURL: server.URL, Model: "test-model"} (synthetic, httptest-local values only).
2. httptest fixtures — httptest.NewServer serves a hard-coded synthetic JSON blob; no real endpoint URL, no real API key, no PII.
3. Package / black-box hygiene — test file declares package llm_test; all referenced symbols use the llm. prefix. Black-box convention satisfied.
4. No other files changed — diff is exactly two files: the prompt constant and its test.

No findings.

REVIEW VERDICT: 0 blocker, 0 major, 0 minor

## Security Review — bookshelf-3493 Diff scope: prompt-string edit in `internal/metadata/llm/llm.go` + accompanying test in `llm_test.go`. No new endpoint, HTTP surface, external call, or dependency introduced. **Checks performed:** 1. **Secrets / credentials / API keys** — none introduced. The prompt is a plain instruction string; the test uses `llm.Config{BaseURL: server.URL, Model: "test-model"}` (synthetic, httptest-local values only). 2. **httptest fixtures** — `httptest.NewServer` serves a hard-coded synthetic JSON blob; no real endpoint URL, no real API key, no PII. 3. **Package / black-box hygiene** — test file declares `package llm_test`; all referenced symbols use the `llm.` prefix. Black-box convention satisfied. 4. **No other files changed** — diff is exactly two files: the prompt constant and its test. No findings. REVIEW VERDICT: 0 blocker, 0 major, 0 minor

zombor commented 2026-07-06 20:22:04 +00:00

Author

Owner

Copy link

CODE REVIEW: APPROVED

Phase 0: DEMO Verification

No DEMO block - this PR is a prompt-text + test change with no runnable CLI command. CI is green and the diff is self-verifying via the new test. Proceeding.

Phase 1: Spec Compliance

Diff: internal/metadata/llm/llm.go and internal/metadata/llm/llm_test.go only.

Prompt strengthening (llm.go:245-252):

• Old weak phrasing removed: "Read the cover date if visible" and "Omit the parenthetical year entirely if the cover date is not visible".
• New phrasing: "ONLY include a year if a cover date is CLEARLY and LEGIBLY PRINTED", "Do NOT guess, infer, or use any prior knowledge", "It is better to omit the year than to guess".

Output contract unchanged:

• comicIssueRe at llm.go:93 is untouched. Optional year group still makes year omittable - absent year still parses to empty string. No downstream behavior change.
• bookIdentifyPrompt (llm.go:261) is untouched.

Domain boundary clean: No go-workflows/wfengine import in internal/metadata/llm.

Coverage exclusions: git diff origin/main -- scripts/check-coverage.sh is empty.

Phase 2: Code Quality

Test is black-box (llm_test.go:1): package llm_test confirmed. makeTestJPEG() comes from image_test.go (also package llm_test).

LLM mocked at httptest boundary: httptest.NewServer at llm_test.go:602 captures the outgoing request body. Assertions inspect capturedBody (actual serialized JSON sent over the wire), not the local const directly - would catch a bug where const is updated but not wired into the request.

Prompt assertions are non-vacuous:

• Positive: "Do NOT guess" (llm_test.go:640), "LEGIBLY PRINTED" (llm_test.go:644), "better to omit" (llm_test.go:648).
• Negative: old "Read the cover date if visible" absent (llm_test.go:652), old "Omit the parenthetical year entirely if the cover date is not visible" absent (llm_test.go:656).
• Five It blocks, each with exactly one Expect.

Ordered + BeforeAll pattern: appropriate for run-once/assert-multiple - JustBeforeEach would re-fire per It and is not applicable here.

No bugs, no security issues, no convention violations found.

REVIEW VERDICT: 0 blocker, 0 major, 0 minor

CODE REVIEW: APPROVED ## Phase 0: DEMO Verification No DEMO block - this PR is a prompt-text + test change with no runnable CLI command. CI is green and the diff is self-verifying via the new test. Proceeding. ## Phase 1: Spec Compliance Diff: internal/metadata/llm/llm.go and internal/metadata/llm/llm_test.go only. **Prompt strengthening (llm.go:245-252):** - Old weak phrasing removed: "Read the cover date if visible" and "Omit the parenthetical year entirely if the cover date is not visible". - New phrasing: "ONLY include a year if a cover date is CLEARLY and LEGIBLY PRINTED", "Do NOT guess, infer, or use any prior knowledge", "It is better to omit the year than to guess". **Output contract unchanged:** - comicIssueRe at llm.go:93 is untouched. Optional year group still makes year omittable - absent year still parses to empty string. No downstream behavior change. - bookIdentifyPrompt (llm.go:261) is untouched. **Domain boundary clean:** No go-workflows/wfengine import in internal/metadata/llm. **Coverage exclusions:** git diff origin/main -- scripts/check-coverage.sh is empty. ## Phase 2: Code Quality **Test is black-box (llm_test.go:1):** package llm_test confirmed. makeTestJPEG() comes from image_test.go (also package llm_test). **LLM mocked at httptest boundary:** httptest.NewServer at llm_test.go:602 captures the outgoing request body. Assertions inspect capturedBody (actual serialized JSON sent over the wire), not the local const directly - would catch a bug where const is updated but not wired into the request. **Prompt assertions are non-vacuous:** - Positive: "Do NOT guess" (llm_test.go:640), "LEGIBLY PRINTED" (llm_test.go:644), "better to omit" (llm_test.go:648). - Negative: old "Read the cover date if visible" absent (llm_test.go:652), old "Omit the parenthetical year entirely if the cover date is not visible" absent (llm_test.go:656). - Five It blocks, each with exactly one Expect. **Ordered + BeforeAll pattern:** appropriate for run-once/assert-multiple - JustBeforeEach would re-fire per It and is not applicable here. No bugs, no security issues, no convention violations found. REVIEW VERDICT: 0 blocker, 0 major, 0 minor

zombor merged commit fdd61f7132 into main 2026-07-06 20:31:17 +00:00

zombor referenced this pull request from a commit 2026-07-06 20:31:21 +00:00

Merge pull request 'fix(llm): forbid year hallucination in comic identify prompt (bookshelf-3493)' (#985) from bd-bookshelf-3493 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

zombor/pergamum!985

No description provided.