feat(wfengine): bulk-ops by-IDs parent workflows (bookshelf-htpc.1) #828
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "bd-bookshelf-htpc.1"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
BulkEnrichHandler/BulkGenerateCoversHandler/BulkCustomFetchHandlerwere spawning one top-level workflow per selected book (N workflows for N books). At 4k selected books this floods the go-workflows queue.BulkEnrichByIDsWorkflowandBulkCoversGenerateByIDsWorkflow: single parent workflows thatContinueAsNew-paginate the ID list (batch 50) and fan out per-book sub-workflows under a bounded single-digit concurrency window.internal/books/trigger_dispatch.gowith three new single-workflow dispatchers wired throughWFTriggerDeps.Test plan
detail.Result == "null",Refetchcalled exactly 10 timesmake testpassesmake coveragepasses (100% oninternal/wfengineandinternal/books)Closes bead bookshelf-htpc.1 on merge.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Security Review — bookshelf-htpc.1 (PR #828)
Reviewer: security-reviewer subagent (claude-sonnet-4-6)
Diff reviewed:
origin/main...origin/bd-bookshelf-htpc.1CI: green (not re-run — CI is source of behavioral truth)
Multi-user scoping
All three bulk handlers (
BulkEnrichHandler,BulkCustomFetchHandler,BulkGenerateCoversHandler) delegate tobulkBookSetHandlerwhich:userIDviauserIDFromRequest(r)— sourced fromd.ExtractUser(r).IDwhich reads JWT claims set byAuthMiddleware, never from the request body.getUserLibraryIDs(ctx, userID)scoped to that session user.filterOwnedIDs(ctx, req.BookIDs, userLibraryIDs)before invokingenqueue.Only the filtered, ownership-verified IDs are forwarded into the new single-parent workflow. The workflow itself never has access to unfiltered IDs. Scoping is correct and unchanged by this refactor.
MaxBulkBookIDs cap
MaxBulkBookIDs = 1000is enforced in every handler beforeenqueueis called:internal/books/bulk_handler.go:77(enrich + covers)internal/books/bulk_custom_fetch_handler.go:49(custom-fetch)The workflow boundary adds no second cap (none needed — the HTTP boundary is the validated entry point). DoS risk from unbounded input is mitigated.
optsJSON deserialization
MetadataRefreshOptionscontains onlystring,bool, and composite struct fields. The round-tripjson.Marshal(opts) → optsJSONintrigger_dispatch.goandjson.Unmarshal(optsJSON, &o)inbuild_extended_deps.go:204is typed and safe. Nointerface{}/any, no URL fields susceptible to SSRF, no callable fields. No injection or unsafe deserialization risk.Architecture boundary
internal/wfengine/bulk_by_ids_workflow.goimportsinternal/books(for*books.MetadataRefreshOptions). The domain packageinternal/booksdoes not importgo-workflowsor any workflow-engine symbol. Boundary rule is satisfied.Secrets / PII logging
Both
trigger_dispatch.goandengine.golog onlyinstance_idandbook_count. No book IDs, no options content, no tokens are emitted to the log. Clean.SSRF
No new outbound URL fetching is introduced. The cover/enrich pipeline is unchanged; only the fan-out dispatch path is refactored. No new SSRF surface.
Concurrency cap
clampCovers()enforces 1–9 range; falls back todefaultFanOutConcurrency(4) for zero/out-of-range values.metadataWindow()falls back todefaultFanOutConcurrency(4) for zero. (Note:metadataWindowhas no upper-bound clamp, but this is pre-existing behaviour shared withBulkEnrichWorkflowand is not introduced by this PR.)Concurrency < 1also falls back todefaultFanOutConcurrency. Fan-out HARD RULE is satisfied for this PR's additions.No findings.
REVIEW VERDICT: 0 blocker, 0 major, 0 minor
Code Review — bookshelf-htpc.1 (PR #828)
Reviewer: code-reviewer subagent (claude-sonnet-4-6)
Diff reviewed:
origin/main...origin/bd-bookshelf-htpc.1CI: green (not re-run — CI is source of behavioral truth)
Phase 0: DEMO Verification
No explicit DEMO block in the bead or its comments. This is a pure internal workflow refactor with no user-visible endpoint output to demo. CI is green with comprehensive integration coverage. Proceeding on that basis.
Phase 1: Spec Compliance
All spec requirements met:
BulkEnrichHandler→BulkEnrichByIDsWorkflow(one parent instance) ✓BulkCustomFetchHandler→BulkEnrichByIDsWorkflowwithoptsJSON✓BulkGenerateCoversHandler→BulkCoversGenerateByIDsWorkflow(one parent instance) ✓bulkByIDsBatchSize = 50, separate from concurrency ✓defaultFanOutConcurrency = 4, covers clamped ≤9, metadata configurable) ✓filterOwnedBookIDs+getUserLibraryIDs+userIDFromRequestall preserved unchanged ✓MaxBulkBookIDscap: unchanged handler code, enforced before dispatch ✓optsJSON []bytecrosses the domain→adapter boundary (no wfengine type ininternal/books) ✓Phase 2: Code Quality
[MAJOR]
internal/wfengine/engine_integration_test.go—BulkCoversGenerateByIDsWorkflowhas no multi-epoch real-engine integration testThe
workflow-integration-test-must-assert-successrule requires ≥3 real ContinueAsNew epochs on the real engine, asserting the terminalResultvalue (not just.state == "completed").BulkCoversGenerateByIDsWorkflowhas only:BulkEnrichByIDsWorkflow(same PR) correctly has a full multi-epoch real-engine test (10 IDs / batch 3 = 4 epochs, assertsdetail.Result == "null"ANDrefetchCalls == numIDs). The same rapid-ContinueAsNewWorkflowExecutionStartedrace (#583) that motivated that test applies equally toBulkCoversGenerateByIDsWorkflow. The fix is small: add a test mirroring the enrich multi-epoch test withWithBulkByIDsBatchSize(3), agenerateCovercall counter viasync/atomic, and assertions ondetail.Result == "null"+coverCalls == numIDs.Positive observations
boundedFanOutseeds up toconcurrencychildren, then awaits the oldest (FIFO) before scheduling the next — genuine schedule-N/await-one-before-next, not 50-at-once.bulkByIDsBatchSize = 50(epoch page size) vsdefaultFanOutConcurrency = 4/coversWindow/metadataWindow(per-epoch concurrent children). Clean separation.startBulkEnrichByIDsWorkflowclosure setsinput.Concurrency = metadataWindowbefore creating the workflow;startBulkCoversGenerateByIDsWorkflowsetsinput.Concurrency = coversWindow. Carried throughContinueAsNew.clampCovers()enforces ≤9 hard cap at the engine boundary.internal/wfengineimportsinternal/books(adapter importing domain — allowed).internal/books/trigger_dispatch.gocarriesoptsJSON []byte(not a wfengine type). Confirmedinternal/bookshas nogo-workflowsimport.map-range over ID slice, notime.Nowinside workflow coroutines.detail.Result == "null"(the SUCCESS value — catches errored-but-marked-completed), assertsatomic.LoadInt64(&refetchCalls) == numIDs(all IDs processed across all epochs). Usessync/atomicfor the shared counter (race-safe). This is exactly what the rule requires.wfengine.New()+StartWorker(): the production constructor + worker startup, catching queue/activity-registry mismatches invisible to the in-memory tester.filterOwnedBookIDs+getUserLibraryIDs+userIDFromRequest. The workflow only ever receives ownership-verified IDs.REVIEW VERDICT: 0 blocker, 1 major, 0 minor
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
MAJOR addressed: Added a real-engine multi-epoch ContinueAsNew integration test for
BulkCoversGenerateByIDsWorkflow(commitee19244e).The new test mirrors the existing
BulkEnrichByIDsWorkflowmulti-epoch test exactly:wfengine.New()+StartWorker()on the production constructor path (real MySQL engine)WithBulkByIDsBatchSize(3)with 10 IDs → 4 ContinueAsNew epochssync/atomiccounter on thegenerateCoverstub; asserts== numIDsat the enddetail.Result == "null"(JSON null = true SUCCESS), not merely absence of an error stringCI green. PR is mergeable.
ee19244e73fba1cda8dcWorkflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)