• NNTP, IMAP, Gopher, msgutils.js: per-number message loops and per-reco

    From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Wed Nov 11 21:54:28 2026
    open https://gitlab.synchro.net/main/sbbs/-/work_items/1340

    ## Summary

    Several stock JavaScript services read a message base one record at a time, or look up every message number in a range, where one `MsgBase.get_index()` call (or clamping the range) would do. This is the same pattern c5f0ca87d9 (reliable-11-fear, 2026-11-11) fixed in webv4's mail page and #1339 describes in webv4's forum. Message numbers are sparse on most real bases (one production sub numbers 1,044 messages from 317,056 to 318,099), so a per-number loop mostly looks up numbers that don't exist, and every lookup is a separate locked index search.

    The cost scales with per-call latency. On Linux, with the data on an SMB share served by the same host, a header lookup measured about 0.1 ms. Where the data directory is on an SMB share, every call is a network round trip: c5f0ca87d9 measured 8.6 ms per lookup on Windows.

    Code references are as of master 36867a9bbf (farm-3-array, 2026-11-11).

    ## 1. NNTP: XOVER/XHDR look up every number in the client's range (high)

    `exec/nntpservice.js:564` (`OVER`/`XOVER`) and `:618` (`HDR`/`XHDR`) call `get_msg_header(false, i)` for every `i` from `first` to `last`, where both come from the client and are not clamped to the group's `first_msg`/`last_msg`.

    - `XOVER 1-` (an open range is `1` through `last_msg`) on the sub above makes about 318,000 lookups to return 1,044 messages.
    - In the mail newsgroup (`include_mail`), only the user's own mail is listed, so even an exact range is mostly other users' messages: the case c5f0ca87d9 fixed in webv4 (24,870 lookups for 926 messages).
    - A client can repeat this at will, and on a host whose data is on an SMB share each lookup is a round trip.

    Suggested fix: clamp `first`/`last` to `first_msg`/`last_msg`, then walk the `get_index()` records inside the range (skipping `MSG_DELETE`/`MSG_VOTE`, and other users' mail by the index `to`/`from`), reading only those headers.

    ## 2. IMAP: the folder index is rebuilt one read per message (high)

    `read_index()` (`exec/imapservice.js:1388`) calls `get_msg_index(true, i)` for every offset (`:1402`). `update_status()` rebuilds it whenever the base's `last_msg` changes (`:1345`). For INBOX that is the mail base, whose `last_msg` changes whenever anyone on the system receives mail, so every connected IMAP client re-reads the whole mail index on its next status update.

    Measured on a 9,364-message mail base (Linux, same-host SMB share): 40 ms per rebuild with per-offset reads, 17 ms with one `get_index()`. With the data on an SMB share each of the 9,364 reads is a round trip.

    Suggested fix: build the index from one `get_index()` call.

    Related, minor: a `UID` sequence set such as `1:*` makes `:671` iterate every number from 1 to `last_msg` (752,280 on that mail base) in memory to find the user's messages. Walking `index.offsets` would avoid that.

    ## 3. NNTP: LISTGROUP reads the index one record per message (medium)

    `exec/nntpservice.js:519` calls `get_msg_index(true, i)` per offset. `count_msgs()` in the same file (`:130`) already does this with one `get_index()` call; LISTGROUP can do the same.

    ## 4. Gopher: a sub listing reads and returns every message (medium)

    `exec/gopherservice.js:298` calls `get_all_msg_headers()` and lists the entire sub with no paging. On a 159,537-message sub, `get_all_msg_headers()` alone took 1.9 s, and the response is one line per message. Each crawler request repeats it.

    Suggested fix: list only the newest N messages (from `get_index()`, reading those headers), with a link to older ones.

    ## 5. msgutils.js getMessageThreads(): the max branch throws and mis-keys results (bug)

    `exec/load/msgutils.js:360` `getMessageThreads(sub, max)`: when `max` is given and `last_msg - max` is above `first_msg`, it loops `m` from `last_msg - max` to `last_msg` (`:382`):

    - `get_msg_header(m)` returns `null` for any missing or deleted number, and `header.num` then throws a TypeError.
    - `header.num` is not a header property (the number is `header.number`), so every header is stored under the key `"undefined"`, and only the last one survives.
    - On sparse numbering the window holds fewer than `max` messages.

    Callers: `xtrn/ecreader/lib.js:376` (passes `settings.max`, normally unset, which takes the `get_all_msg_headers()` path) and the old web UI's `web/lib/forum.ssjs:119`, `:303` (`webIni.maxMessages`).

    Suggested fix: take the newest `max` records from `get_index()` and read those headers by offset.

    ## 6. Lower priority: loops over every user record

    `exec/load/birthdays.js:22`, `exec/load/consensus_vote.js:201`, `exec/hotline.js:970` and the old web UI's `web/root/ecwebv3/sidebar/002-whosOnline.ssjs:27` read every user record (1,457 records took 0.1 s on Linux). None runs per request in a default setup.

    *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab issue in main/sbbs on Fri Nov 13 16:43:49 2026
    close https://gitlab.synchro.net/main/sbbs/-/work_items/1340
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)
  • From Rob Swindell@1:103/705 to GitLab note in main/sbbs on Fri Nov 13 16:43:49 2026
    https://gitlab.synchro.net/main/sbbs/-/work_items/1340#note_10928

    Items 1-5 are fixed on master:

    - 1 and 3 (NNTP XOVER/XHDR range lookups, LISTGROUP per-record index reads): 8edb4bf421 (unlock-12-taught, 2026-11-11)
    - 2 (IMAP folder index rebuilt one read per message, `UID 1:*` iterating every number): 79c2da3198 (exceed-14-mint, 2026-11-11)
    - 4 (Gopher sub listing returns every message): 6bb3b32af0 (rough-30-idea, 2026-11-11)
    - 5 (`getMessageThreads()` max branch throws and mis-keys results): 1a255ae945 (oral-2-dump, 2026-11-11)

    Item 6 (loops over every user record in `birthdays.js`, `consensus_vote.js`, `hotline.js` and ecwebv3's `002-whosOnline.ssjs`) is left as is: none of them runs per request in a default setup.

    *Authored by Claude (Claude Code), on behalf of @rswindell*
    --- SBBSecho 3.38-Linux
    * Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)