Incidents

Stefan Coetzeecreated 2026-10-09updated 2026-10-091 min read

Vault note, not reviewed against the source. Written in the knowledge vault on 2026-10-09 by models working with Stefan Coetzee and published as it stands, with private addresses, e-mail addresses and an employer name redacted. Check claims against the primary source before relying on them.

Incident, cause and mitigation records from the vault, written up from trap files and session logs. Each record names its source.

Child pages

Authenticated Cookie Jar for Gated PostsAn exported cookies.txt lets yt-dlp list and fetch gated posts.
Back Up SQLite With the Backup CommandUse sqlite3 .backup, or checkpoint the WAL first, to get a restore point that holds every write.
Bracket Pattern Process CheckUse pgrep -f "[b]atchtranscribe.py" or the ps | awk form.
Compare Videos Found Against the In-App CountThe owner's in-app count minus videosfound measures the gap.
Compare Videos Found Run to RunFirst fix for the short scrape: treat any drop in videosfound between runs as suspect.
Differential Test in the Same MinutesRun the failing path and its nearest working variants side by side, within the same few minutes, changing one factor at a time.
Eliminate Concurrent Saves Before Blaming the ScraperAny 'scrape found N, collection holds M' gap must first rule out that the owner saved something mid-run.
Enumeration Retry With Unauthenticated FallbackEnumeration tries the cookie jar twice, then unauthenticated mode three times with backoff, and logs which path succeeded.
Expired Bot-Manager Cookie Returns Empty Bodyyt-dlp sends expired entries from a Netscape cookie jar as they are.
File Copy of a WAL-Mode Database Misses Recent WritesA SQLite database in WAL mode keeps recent writes in the -wal file.
Filter Expired Cookies Before Each Runcookieargs() writes a filtered copy of the cookie jar with expired entries dropped, logs each drop, and passes the filtered file to yt-dlp.
Hand-Check Output Against the Primary SourceCompare a sample of generated summaries or extracted figures with the source they came from.
Handle-Based Ad Filter Matches Artist AccountsThe creator rule \.(de|official)$ was written for brand and shop accounts.
Incident 2026-02-27 Dead Collection Short LinksTwo collections stopped scraping because their stored short links died.
Incident 2026-07 Summaries Inverted IronyOne sarcastic recommendation was summarised as a sincere warning, one summary named the wrong person, one was selectively pessimistic against its source.
Incident 2026-07-14 Backup Missed WAL WritesA pre-change backup copied the main database file only, while the database was in WAL mode with a multi-megabyte WAL file.
Incident 2026-07-14 Short Scrape Looked Like a Quiet FeedSix runs in a row reported 19 videos found; a later run reported 32, and the owner's own count was closer than the database's.
Incident 2026-07-19 Pending Status on Finished VideosTwelve videos sat at statuspending while holding full transcripts and summaries.
Incident 2026-07-21 Ad Filter Removed Artist VideosTen videos from artists with .official handles were rejected by the ad filter before transcription and looked like unprocessed backlog.
Incident 2026-08-08 Empty-Transcript Videos Invisible to ReviewA reel made of on-screen text returned no transcript.
Incident 2026-08-08 False Enumeration Bug FindingA scrape found 46 videos and a re-enumeration minutes later found 48.
Incident 2026-09-11 Three-Morning Cron CrashThe 06:15 cron died three mornings running with CalledProcessError.
Incident 2026-09-13 Ghost Still-Running JobA process check over SSH reported the batch job as still running after it had exited.
Incident 2026-09-16 Server Cannot List the CollectionFrom 2026-09-16 the server could not enumerate the collection from any client or egress, while the laptop could with the same yt-dlp version through the same egress.
No-Transcript Bucket in the Review ToolPlanned: have the review tool surface rows with a NULL review and an empty transcript as their own 'needs a manual look' bucket, so they stop disappearing whether or not OCR ever lands.
Probe Pending Rows Before Calling Them BacklogA pending count is unclassified until the rows are looked at.
Probe the Artifact Before Reasoning From CountsWhen a count looks wrong, fetch one real item and read the error.
Process Check Matches Its Own Shellpgrep -f <pattern> run over SSH matches the remote shell whose command line contains the pattern.
Publish Date Treated As Time In CollectionA video's publish date says nothing about when it was saved to a collection.
Pull the Full Row List After Any ScrapeDo not take the review tool's count as the batch size.
Re-Scrape Relinks Processed Rows As PendingRe-scraping a collection re-links videos that were already processed and sets them back to statuspending without flipping them to complete again.
Review Tool Lists Only Rows With a Transcriptreviewstatus.py reports on rows where length(transcript) > 0.
Sensitivity-Gated Posts Vanish From Unauthenticated ListingTikTok refuses to serve age-restricted or sensitivity-flagged posts to an unauthenticated client.
Short Link Redirects to the HomepageStored vm.tiktok.com short links for two collections now redirect to the site homepage.
Single Video URL As the Collection ArgumentA single video URL works as the --collection-url argument, so individual videos can be reprocessed without a working collection link.
Small Local Model Misreads Ironic RegisterA 7B local summarizer renders sarcasm as sincere statement, swaps subjects, or turns selectively pessimistic against its own source.
Summaries Are a Triage IndexUse summaries to decide what to read, never as evidence.
Swallowed Stderr Hides the Causesubprocess.run(..., checkTrue, captureoutputTrue) captures the child's stderr and raises CalledProcessError.
Transcriber Incident RegisterEvery failure of the video transcriber pipeline since February 2026, as typed atoms.
Verify Transcript Length Never StatusTreat status as a report. Check length(transcript) and length(summary) per row.