zombor/pergamum
1
0
Fork 0
Code Issues Pull requests 2 Projects Releases Packages 2 Wiki Activity Actions 1

fix(integration): raise go test -timeout 20m to prevent spurious TestDB panic (bookshelf-0xjkz) #1338

Closed
zombor wants to merge 1 commit from bd-bookshelf-0xjkz into main
pull from: bd-bookshelf-0xjkz
merge into: zombor:main
Conversation 1 Commits 1 Files changed 1 +5 -1
zombor commented 2026-08-05 02:18:46 +00:00
Owner
Copy link

Root cause

internal/db has 260+ sequential DB specs. Locally with a warm testcontainer
the suite completes in ~3–4 minutes. On CI with the untuned MySQL :1 image
(limited redo-log capacity, disk-backed tmpfs), each spec runs slower and the
full suite approaches or exceeds the default 10-minute go test timeout —
producing the misleading panic: test timed out after 10m0s / running tests: TestDB (10m0s) seen on main.

Key findings (investigation):

  • CI Integration uses make integration = go test -p 8, NOT -p 12 as
    the bead prompt assumed. Only 5 packages run concurrently (max).
  • The clone-storm / thundering-herd (bookshelf-7akw) is already fixed; the
    ensureTemplateDB advisory lock path is intact in internal/dbtest/dbtest.go.
  • The issue is NOT concurrency-related. It is a single slow test suite (TestDB
    in internal/db, 260+ specs) running against a cold/untuned MySQL image.
  • Locally: 210–250s (~3.5–4m). CI with untuned :1 image: 600s+ (>10m).

Fix

Add -timeout 20m to make integration. This gives 10 minutes of headroom
over the observed local maximum, while still bounding genuine hangs at 20m
(vs the previous silent hang-until-10m-panic).

Relationship to PR #1335 (bookshelf-7w1mn)

This fix is independent of the tuned MySQL :2 image PR (#1335). Once
#1335 merges, per-spec execution will be faster and the 20m budget will be
consumed even less. Both fixes are complementary.

Test plan

  • make integration passes locally in ~3.5m with -timeout 20m explicit.
  • CI Integration job will now have 20m instead of 10m; the suite that was
    timing out at 10m should complete successfully within 20m on the CI MySQL :1
    image.

Closes bead bookshelf-0xjkz on merge.

## Root cause `internal/db` has 260+ sequential DB specs. Locally with a warm testcontainer the suite completes in ~3–4 minutes. On CI with the untuned MySQL `:1` image (limited redo-log capacity, disk-backed tmpfs), each spec runs slower and the full suite approaches or exceeds the default 10-minute `go test` timeout — producing the misleading `panic: test timed out after 10m0s / running tests: TestDB (10m0s)` seen on main. Key findings (investigation): - CI Integration uses `make integration` = `go test -p 8`, NOT `-p 12` as the bead prompt assumed. Only 5 packages run concurrently (max). - The clone-storm / thundering-herd (bookshelf-7akw) is already fixed; the `ensureTemplateDB` advisory lock path is intact in `internal/dbtest/dbtest.go`. - The issue is NOT concurrency-related. It is a single slow test suite (TestDB in `internal/db`, 260+ specs) running against a cold/untuned MySQL image. - Locally: 210–250s (~3.5–4m). CI with untuned `:1` image: 600s+ (>10m). ## Fix Add `-timeout 20m` to `make integration`. This gives 10 minutes of headroom over the observed local maximum, while still bounding genuine hangs at 20m (vs the previous silent hang-until-10m-panic). ## Relationship to PR #1335 (bookshelf-7w1mn) This fix is **independent** of the tuned MySQL `:2` image PR (#1335). Once #1335 merges, per-spec execution will be faster and the 20m budget will be consumed even less. Both fixes are complementary. ## Test plan - `make integration` passes locally in ~3.5m with `-timeout 20m` explicit. - CI Integration job will now have 20m instead of 10m; the suite that was timing out at 10m should complete successfully within 20m on the CI MySQL `:1` image. Closes bead bookshelf-0xjkz on merge.
fix(integration): raise go test -timeout to 20m to prevent spurious 10m TestDB panic
All checks were successful
/ JS Unit Tests (pull_request) Successful in 3m19s
Details
/ Coverage (pull_request) Successful in 3m23s
Details
/ Lint (pull_request) Successful in 3m59s
Details
/ Test Race (pull_request) Successful in 4m3s
Details
/ E2E API (pull_request) Successful in 4m9s
Details
/ Integration (pull_request) Successful in 6m48s
Details
/ E2E Browser (pull_request) Successful in 8m26s
Details
29b8a11ae2
 
internal/db has 260+ sequential DB specs. Locally with a warm testcontainer
the suite completes in 3-4 minutes, but on CI with the untuned MySQL :1 image
(no redo-log tuning, disk-backed tmpfs) each spec runs slower, and the full
suite approaches or exceeds the default 10-minute go test timeout — producing
the 'panic: test timed out after 10m0s / running tests: TestDB (10m0s)' failure
seen on main (bookshelf-0xjkz).

Root cause: go test defaults to 10m; the integration target had no explicit
-timeout, so any CI slowdown past 10m caused a misleading panic instead of a
useful spec failure. Raising to 20m gives headroom for slow CI without masking
genuine hangs (a real hang would still be caught at 20m).

This fix is independent of bookshelf-7w1mn (PR #1335, tuned MySQL :2 image):
once #1335 lands, per-spec execution will be faster and the 20m budget will be
used even less.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zombor commented 2026-08-05 12:09:04 +00:00
Author
Owner
Copy link

Closing without merging. The -timeout 20m band-aid is unnecessary — the intermittent Integration stall is the thundering-herd class, addressed by the 7w1mn MySQL tuning (PR #1335: Integration green in ~5m44s on the tuned :2 image), not by extending the test timeout. Per the project's don't-mask-with-bigger-timeouts guideline. The remaining Integration flake is a separate go-workflows ContinueAsNew race tracked in bookshelf-7pobp.

Closing without merging. The `-timeout 20m` band-aid is unnecessary — the intermittent Integration stall is the thundering-herd class, addressed by the 7w1mn MySQL tuning (PR #1335: Integration green in ~5m44s on the tuned :2 image), not by extending the test timeout. Per the project's don't-mask-with-bigger-timeouts guideline. The remaining Integration flake is a separate go-workflows ContinueAsNew race tracked in bookshelf-7pobp.
zombor closed this pull request 2026-08-05 12:09:09 +00:00
All checks were successful
/ JS Unit Tests (pull_request) Successful in 3m19s
Details
/ Coverage (pull_request) Successful in 3m23s
Required
Details
/ Lint (pull_request) Successful in 3m59s
Required
Details
/ Test Race (pull_request) Successful in 4m3s
Required
Details
/ E2E API (pull_request) Successful in 4m9s
Required
Details
/ Integration (pull_request) Successful in 6m48s
Required
Details
/ E2E Browser (pull_request) Successful in 8m26s
Required
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
zombor/pergamum!1338
No description provided.