• webv4 forum: unread counts and thread views look up every message numb

    From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Wed Nov 11 21:37:46 2026
    open https://gitlab.synchro.net/main/sbbs/-/work_items/1339

    ## Summary

    webv4's forum code has the same problem c5f0ca87d9 (reliable-11-fear, 2026-11-11) fixed for the mail page: it looks up messages one number at a time, or reads every header in a sub, where an index read would do. Some of it runs on a timer for every logged-in viewer of a forum page.

    Code references are as of master 36867a9bbf (farm-3-array, 2026-11-11); `webv4/lib/forum.js` is unchanged since c5f0ca87d9.

    Measurements are read-only benchmarks against a production install's message bases (147 subs) using one real user's scan pointers. A header lookup there costs about 0.1 ms (Linux, loopback SMB mount). c5f0ca87d9 measured 8.6 ms per lookup for webv4 on Windows reading the message bases over an SMB share; the "at 8.6 ms" column uses that figure and is an estimate, not a measurement.

    ## 1. Unread counts: per-number loop, re-run every 60 s per viewer (high)

    `getSubUnreadCount()` (`lib/forum.js:101`) calls `get_msg_header()` for every number from `scan_ptr + 1` to `last_msg`. `getSubUnreadCounts()` and `getGroupUnreadCounts()` call it for every sub in a group, or in every group.

    The forum's group list and sub list (`pages/001-forum.xjs:382`, `:420`) subscribe to `groups_unread` / `subs_unread` on `api/events.ssjs`, and `lib/events/forum.js` re-runs those scans every `refresh_interval` (default 60000 ms), once per open page.

    | | Count |
    |---|---|
    | Header lookups for one pass over all subs | 256,940 |
    | Messages actually above the user's scan pointers | 58,681 |
    | Lookups for numbers that don't exist | 77% |
    | Worst sub: never read (`scan_ptr` 0), numbers 108531-113539 | 113,539 lookups for 5,009 messages |

    | | Measured (0.1 ms/lookup) | At 8.6 ms/lookup |
    |---|---|---|
    | Current loop, one pass | about 26 s | about 37 min |
    | `MsgBase.get_index()` on every sub | 0.9 s | |

    `cycle()` in `lib/events/forum.js` sets `last_run` before it scans. A pass that takes longer than `refresh_interval` therefore starts the next one as soon as it ends, so a slow host keeps the loop running continuously for as long as the page is open (up to `event_stream_duration`, default one hour).

    Suggested fix:

    - Count from `get_index()`: records with `number > scan_ptr`, excluding `MSG_DELETE`/`MSG_NODISP`.
    - For `SCAN_CFG_YONLY`, compare the index `to` field (`smb_name_crc()`, a CRC-16 of the lower-cased name) against `crc16_calc()` of the lower-cased alias and name. Read the header only on a match, to rule out CRC collisions.
    - In `lib/events/forum.js`, skip a sub whose `last_msg` and the user's `scan_ptr` are unchanged since the previous pass, and set `last_run` after the scan rather than before.

    ## 2. Thread views read every header in the sub (high)

    With `max_messages=0` (the stock `modopts.ini` value), `getMessageThreads()` (`lib/forum.js:917`) calls `get_all_msg_headers()` on the whole sub. It runs on each:

    - thread list page (`pages/001-forum.ssjs:602`)
    - thread view (`:387`, `pages/001-forum.xjs:24`)
    - `list-threads` API page, i.e. each "load more" (`listThreads()`)
    - vote refresh (`getVotesInThread()`, `getVotesInThreads()`)
    - 60 s `scan_threads` cycle per viewer of a sub (`getThreadStats()`)

    | Sub | Messages | `get_all_msg_headers()` |
    |---|---|---|
    | largest | 159,537 | 1.9 s |
    | second largest | 67,247 | 0.9 s |
    | third largest | 23,202 | 0.3 s |

    The 60 s cycle can skip the work when the sub's `last_msg`/`total_msgs` are unchanged (votes are messages, so they change both). Removing the per-page-view cost needs the computed threads cached across requests, keyed on those values.

    ## 3. Correctness bugs in getMessageThreads()

    - With `max_messages > 0`, `getSomeMessageHeaders()` (`lib/forum.js:973`) reads numbers `last_msg - max` through `last_msg`. When numbering is sparse that window holds fewer than `max` messages, so threads are silently dropped.
    - `lib/forum.js:1006` calls `is_spam(header)`, but no `header` is defined in that scope (the record is `headers[h]`). With `forum_no_spam` set and `max_messages=0`, this throws a ReferenceError.

    ## 4. Poll data reads every header in the sub (medium)

    `getUserPollData()` (`lib/forum.js:287`) calls `get_all_msg_headers()` to find one poll's `tally`; its comment calls this temporary. The tally is only assembled by `get_all_msg_headers()`, so fixing this needs a way to get one message's tally natively.

    ## 5. Lower priority

    - `getNewestMessageInSub()` (`lib/forum.js:56`) walks down from `last_msg` past deleted messages and opens every sub in the group. Its only page caller is commented out; it is still reachable through the `get-newest-message-per-sub` API call.
    - The user list page (`pages/More/001-userlist.xjs:143`) reads every user record on every view (1,457 records, 0.1 s here).
    - The `node-list` call in `root/api/system.ssjs` reads every user record on every call. No webv4 page uses it.

    Possibly related: #1129 (intermittent long pauses navigating webv4).

    *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Wed Nov 11 22:55:56 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1339#note_10920

    dbc2439f2d (describe-27-ours, 2026-11-11) addresses item 1 and item 3, and part of item 2.

    **Done**
    - Item 1: `getSubUnreadCount()` counts from one `get_index()` read per sub. A sub whose `last_msg`, `total_msgs`, scan pointer and scan config are unchanged is not recounted within the same event stream. `cycle()` sets `last_run` after the scan.
    - Item 2 (the 60 s cycle only): `scan_threads` skips the rebuild when the sub is unchanged.
    - Item 3: both `getMessageThreads()` bugs.

    **Measured on vert.synchro.net** (message bases over SMB, one user with 28,900 unread messages):

    | Call | Before | After |
    |---|---|---|
    | `get-group-unread-counts` (all groups) | 355 s | 9.4 s |
    | `get-sub-unread-counts`, one 23-sub group | 69 s | 3.0 s |

    The per-sub output was byte-identical before and after.

    **Still open**
    - Item 2: every thread list, thread view and "load more" still calls `get_all_msg_headers()` on the whole sub.
    - Item 4: poll tally.
    - Item 5.

    **Found along the way:** `setScanCfg()` always sets `SCAN_CFG_NEW` together with `SCAN_CFG_YONLY`, so with the web UI's "to you only" setting, `scanned` equals `total`. The to-you count only applies to a scan config set elsewhere. Behaviour unchanged by this commit.

    — *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Nov 13 17:35:58 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1339#note_10933

    Item 2 is addressed by the commit "webv4: cache a sub's forum threads instead of rebuilding them per request" (on master, not yet pushed; it will cross-link here when it is).

    `getMessageThreads()` now caches each sub's threads in `data/webv4/threads/<code>.json`, keyed on the `.shd`/`.sid` file dates and sizes plus `max_messages` and `forum_no_spam`. Any post, vote, deletion or header update rebuilds it. Unread counts are computed per user after loading, so nothing per-user is cached. Thread lists, thread views, "load more" and vote refreshes all go through it.

    **Measured on Linux** (cached result byte-identical to a rebuild):

    | Sub | Messages | Before | Cache hit | Rebuild + write |
    |---|---|---|---|---|
    | syncprog | 9,103 | 392 ms | 181 ms | 630 ms |
    | usenet_altcomposwin11 | 23,248 | 925 ms | 433 ms | 1.35 s |
    | usenet_altcomposwin10 | 159,560 | 7.1 s | 2.8 s | 10.1 s |

    Reading headers was only about a third of the old cost: each field read from a `get_all_msg_headers()` header costs about 1 us, so copying 13 fields per message took as long again. On the largest sub a cache hit is still mostly parsing 36 MB of JSON; shrinking that would mean changing what the pages consume.

    **Still open**
    - Item 4: poll tally still reads every header in the sub (needs a native way to get one message's tally).
    - Item 5 (lower priority).

    *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Sun Nov 15 00:35:31 2026
    close https://gitlab.synchro.net/main/sbbs/-/work_items/1339
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Sun Nov 15 00:35:31 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1339#note_11024

    Item 4 is addressed by the commit "webv4: take a poll's tally from the sub's threads, not every header" (on master, not yet pushed; it will cross-link here when it is). Item 5 has moved to #1349, so this issue is closed.

    `getUserPollData()` now takes the poll's tally from `getMessageThreads()`, which caches it with the sub's threads, and the poll's other fields from `get_msg_header()`. A thread view has already loaded the threads, so its polls reuse them. A poll that isn't in the threads (for example, one older than `max_messages`) still falls back to `get_all_msg_headers()`. It now also requires that the user can read the sub.

    **Measured on Linux**, poll lookups on a thread page whose threads are already loaded:

    | Sub | Messages | Polls | Before | After |
    |---|---|---|---|---|
    | sync_sys | 5,145 | 8 | 393 ms | 14 ms |
    | sync-js | 4,374 | 3 | 131 ms | 5 ms |
    | dove-sys | 5,037 | 2 | 82 ms | 4 ms |

    Output was identical to the previous code for all 25 polls on the install.

    Two follow-ups to the thread cache are also on master, unpushed: its temporary files are now named correctly and abandoned ones are removed, and the unused `get-thread-list` API call is gone. That call has always failed with a ReferenceError; correcting the reference would have let any logged-in user list the headers of any sub, including other users' mail.

    *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)