fix(bookdrop): recursive fsnotify watcher for nested drops (bookshelf-v3ks) #913
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "bd-bookshelf-v3ks"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Watch()ininternal/bookdrop/watcher.go:94calledwatcher.Add(dir)for the root only — fsnotify is non-recursive so files dropped into any subdirectory fired no real-time events (only the periodic WalkDir scan recovered them).Add()every existing subdirectory to the watcher.Createevents for directories; callAdd()on the new dir and enqueue files already present (race-window coverage).Add()on any subdir logsWarnand continues — non-fatal per spec.handleEvent()andaddSubdirs()helpers to keep every function under 30 lines and CC < 10.Files changed
internal/bookdrop/watcher.go— bug fixinternal/bookdrop/export_test.go—AddSubdirs+ updatedWatchLoopsignatureinternal/bookdrop/watcher_test.go— 5 new specs + updated existingWatchLoopcallsTest plan
AddSubdirstest: failing add logs warn, does not panic/crashWatchLooptest: Create event for a directory calls addWatch + enqueues existing filesWatchLooptest: failing addWatch on new dir does not stop the loopWatch (recursive)test: detects epub in pre-existing subdir (startup walk)Watch (recursive)test: detects epub in newly-created subdir (runtime add)make coveragegate greenCloses bead bookshelf-v3ks on merge.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Security Review — bd-bookshelf-v3ks (recursive bookdrop watcher)
[MAJOR] internal/bookdrop/watcher.go:228 — Symlink escape:
os.Statfollows symlinks inhandleEvent, permitting ingest-scope escapeos.Stat(name)follows symlinks. If any actor with write access to the bookdrop root creates a symlink-to-directory (e.g.ln -s /mnt/private-books /bookdrop/escape), fsnotify fires a Create event for/bookdrop/escape;os.Statfollows the link and returnsIsDir()=true, causinghandleNewDirto (a) register the symlink target with inotify viawatcher.Addand (b) walk and enqueue its contents. Book-format files in the target directory are ingested into the pipeline as if they were under the bookdrop root. The existingWalkDirfunction (line 35) explicitly guards against this withd.Type()&os.ModeSymlink != 0; the new runtime path has no equivalent guard. Under DISK_TYPE=NETWORK (NFS/SMB), a remote actor with share-write access can exploit this without local shell access. Fix: replaceos.Stat(name)withos.Lstat(name)at line 228 so a symlink is never mistaken for a plain directory.[MAJOR] internal/bookdrop/watcher.go:111,129 — Unbounded inotify watch-descriptor consumption, no cap or operator signal
addSubdirs(line 111) andhandleNewDir(line 129) register every subdirectory recursively with no upper bound. Linux's per-user inotify watch limit defaults to 8,192 (fs.inotify.max_user_watches). A deep or wide directory tree — whether legitimate (large author/series/volume hierarchy) or adversarially planted (via NFS/SMB write access when DISK_TYPE=NETWORK) — can exhaust all descriptors system-wide, silently starving other inotify consumers (editors, container runtimes, other services). Failures fromwatcher.Add()are only logged at Warn individually; there is no aggregate count, no Prometheus counter, and no escalation to Error — the operator has no visibility that watch coverage is partially degraded. Fix: impose a configurable cap (e.g. 1,000 watched dirs) and log at Error + emit abookdrop_watch_dirs_totalPrometheus counter when the cap is reached; at minimum, emit a counter on every failedAdd()call so an operator can alert on descriptor exhaustion.[MINOR] internal/bookdrop/watcher.go:130 —
handleNewDirWalkDir enqueues symlinked files without a symlink checkThe WalkDir callback inside
handleNewDir(line 130) skips directory entries (d.IsDir()) but does not skip symlink entries (d.Type()&os.ModeSymlink). A file symlink with a book-format extension inside a new subdirectory (e.g.novel.epub -> /home/user/private.epub) is enqueued and follows the full ingest pipeline. The existing productionWalkDir(line 35) explicitly skips symlinks. Fix: addif d.Type()&os.ModeSymlink != 0 { return nil }in thehandleNewDirWalkDir callback, mirroring the policy inWalkDir.Architecture boundary: No workflow-engine imports introduced. Clean.
DISK_TYPE guard: The guard lives in
RejectProposal(destructive file ops). The watcher itself is unguarded on both main and this branch — no regression introduced here.Newly-created dir ingest path: Files in new subdirs go through the same
isBookFile→onEvent→IngestFilepipeline; no existing validation is bypassed.REVIEW VERDICT: 0 blocker, 2 major, 1 minor
CODE REVIEW: NOT APPROVED
Phase 0: DEMO Verification
No explicit DEMO block with a re-runnable command. The PR test-plan has a checked checkbox claiming a temporary probe was used. I ran independent probes on the worktree to verify gate behavior directly.
Probe results (all run against
.worktrees/bd-bookshelf-scev):NewBigHandlerappended tointernal/books/handler.gofires correctly (per-function exclusion works; new function names are gated)Findings
[MAJOR] .golangci.yml line ~24 — gocyclo threshold=30 allows genuinely complex new functions to ship silently
CLAUDE.md specifies CC<10. Gate at CC>30 leaves the entire CC 11-30 range completely ungated. Probe confirmed: a function with CC=20 (twice the style guide) produces
0 issues. The bead description itself suggested 10-15 as a pragmatic range; 30 is twice that suggestion. A new god-function with CC=25 will pass lint without warning. This is the permanent quality bar being set. At CC>30 the gate only catches the most catastrophic outliers.Fix: Tighten to 15 (the bead's own upper bound). If existing code violates CC 15-30, grandfather those per-function as done for the other linters. At 15, CC 11-14 still slips through but the range is narrow and defensible. At 30, the gate is cosmetic for the CC 11-29 range that actually needs governance.
[MAJOR] .golangci.yml (nestif per-file section) — 25 whole-file exclusions permanently blind nestif to NEW code in god-files
The 25 per-file nestif exclusions (
path: internal/books/handler\.go + linters: [nestif]) exempt EVERY nestif violation in those files — past AND future. Probe confirmed: a new function with 4 levels of nesting (nestif complexity=10) appended tointernal/books/handler.goproduces0 issues. The config comment says "New files with deep nesting are still gated" — true for new FILES, but new FUNCTIONS added to existing grandfathered files (the exact god-files that need governance most) are permanently invisible to nestif.The technical constraint is real: nestif embeds raw condition text in its message (not the function name), making per-function text patterns brittle. But the consequence is a gate that does not gate the worst offenders.
Fix options (in order of preference):
// LINT-EXEMPT: nestif whole-filecomment inside each grandfathered source file so developers adding new deeply-nested code to that file see the warning at the point of edit.exclude-rulessupports matching nestif violations by surrounding function context in newer versions.[MINOR] .golangci.yml comment/setting mismatch — comment says "nesting complexity >= 4" but min-complexity: 5
The inline comment reads
# gate: nesting complexity >= 4but the actual setting isnestif: min-complexity: 5. The gate fires at complexity >=5, not >=4. Fix: change comment to>= 5.[MINOR] funlen=60 is 2x the CLAUDE.md style guide (func<30 lines)
Functions up to 60 lines pass silently. The PR is explicit about pragmatism here, and 60 is at least a real gate. Noting it because this is the permanent bar — CLAUDE.md's 30-line aspiration is effectively unenforced by CI. Consider tightening to 45 at the next opportunity.
REVIEW VERDICT: 0 blocker, 2 major, 2 minor
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Code Review — focused re-review of fix commits (inotify cap + config/metrics wiring)
Phase 0 — DEMO verification
No traditional shell-command DEMO block exists in this PR. The implementation is verified by the test suite. CI reports
successon SHAa64a2a7fd7e6dcc3d8bbc6d5c65c212c452417ea. Proceeding on CI-green as the behavioral ground truth per project conventions.Phase 1 — cappedAdd correctness
(a) Off-by-one:
watchedDirsstarts at 1 (root added directly at line 108). cappedAdd guard is*count >= maxDirs. WithmaxDirs=1, first subdir attempt:1 >= 1→ true → skip. Exactly MaxWatchedDirs dirs are registered. Correct.(b) Count only on success:
*count++executes only afteradd(dir)returns nil (lines 136–140). Count is never incremented on a failed Add. Correct.(c) Concurrency:
addSubdirsruns synchronously beforewatchLoopstarts (lines 115–119). InsidewatchLoop, only the main select goroutine callsadd(viahandleEvent → handleNewDir). Theflushclosure spawned bytime.AfterFunctouches onlypendingPaths(mutex-protected) andonEvent— it never callsadd. No concurrent mutation ofcount. Correct.(d) MaxWatchedDirs<=0 → no cap: Guard is
maxDirs > 0 && *count >= maxDirs. WhenmaxDirs == 0the first condition is false and the body is never entered. Correct.Phase 2 — OnDirDropped
Fires exactly once per dropped dir (one call per cappedAdd invocation that hits the cap). nil-safe:
if onDrop != nil { onDrop() }at line 131. Logs atlogger.Error(not Warn). All correct per original finding.Phase 3 — Config wiring
Default 1000 set in
config.Defaults()(config.go:317). Flag registered with that default.app.Run()passescfg.BookdropMaxWatchedDirsdirectly toWatchOptions.MaxWatchedDirs(app.go:951). No nil/zero-value trap in the production path —Defaults()always sets 1000, and 0 is documented as "no cap" (a safe fallback if someone bypasses defaults).Phase 4 — Symlink completeness
grep os.Stat internal/bookdrop/watcher.goreturns only line 288 (os.Lstat). No remainingos.Staton event paths.handleNewDirusesd.Type()&os.ModeSymlink != 0in the WalkDir callback.addSubdirsrelies onfilepath.WalkDirnot following symlinks, so symlinked dirs never satisfyd.IsDir(). Correct.Phase 5 — Tests
dropped > 0. Valid.added. Valid.addeddoes not contain symlinkPath. Valid.handleNewDirWalkFunccovers thewalkErr != nilpath without filesystem permission tricks. Valid.Phase 6 — No new golangci.yml exclusions: Confirmed — diff of
.golangci.ymlis empty.Findings
[MINOR] internal/appwire/appwire.go:226 —
BookdropMaxWatchedDirsfield inappwire.Depsis stored but never consumedThe field is populated in
app.New()at app.go:424 (BookdropMaxWatchedDirs: cfg.BookdropMaxWatchedDirs) but no code in the codebase readsdeps.BookdropMaxWatchedDirs.app.Run()reads the value directly from itscfg *config.Configparameter (app.go:951). The stored Deps field is dead code. No correctness impact — the correct value flows throughcfg— but the field adds noise to the Deps struct. Fix: remove the field fromappwire.Depsand the assignment inapp.New(), sinceRun()already sources it fromcfgdirectly.REVIEW VERDICT: 0 blocker, 0 major, 1 minor
Workflow Detail page screenshot (wf-detail-older-execution)
Older completed ContinueAsNew epoch detail — execution ID and state visible, Cancel absent.
Recompute Match Score — kebab open screenshot (recompute-match-score-kebab-open)
371d1198831971d07d7d