From 9f5cc59af7cbb9be7f351168592cf0e78ed7f10b Mon Sep 17 00:00:00 2001 From: DonVoo Date: Thu, 16 Jul 2026 13:22:27 +0200 Subject: [PATCH] Stabilize mail identity: canonical hash, index drift, HTML panic Behebt drei Fehler derselben Klasse: an jeder Stelle wurde derselbe Wert zweimal berechnet, statt einmal berechnet und weitergereicht - und die beiden Berechnungen liefen auseinander. 1. Instabiler Ersatzschluessel (bug-hashkey-instabil.md) Mails ohne Message-ID bekamen sha256 ueber die IMAP-Rohbytes, der Reindex hashte dieselbe Mail ueber die mbox-gespeicherten Bytes (>From-Quoting, andere Zeilenenden) -> zwei Schluessel, 4.024 Doppel-Eintraege in copied. Fix: canonicalMessageBytes() bringt beide Seiten auf eine Form (>From zurueckdrehen, CRLF->LF, Trailing-Newlines weg). Alle Pfade (Index, Migration, Viewer, Dedup) nutzen dieselbe Funktion. body_sha256 in copied + mbox_index, UNIQUE erweitert. 2. Index-Drift (bug-index-drift.md) Der plain-mbox-Reader las nach Datei-Position statt nach dem gespeicherten file_offset. Weil SQLITE_BUSY (busy_timeout=0) je Ordner einen Index-Eintrag verschluckt hatte, war ab dieser Luecke alles um 1 verschoben: Klick auf Mail X zeigte Mail X+1. Betroffen genau die 6 Ordner mit Busy-Fehler beim Rettungslauf. Fix: busy_timeout=10000, Lesen ueber file_offset, --reindex --rebuild. 3. HTML-Vorschau-Panic (bug-htmltotext-panic.md) replaceCaseInsensitive/stripHTMLBlock indizierten mit Offsets aus strings.ToLower(s) in s - ToLower ist nicht byte-laengen-erhaltend (z.B. U+0130). ~0,1% der Mails brachten die Vorschau zum Absturz. Fix: asciiFoldIndex() sucht direkt auf den Original-Bytes; zusaetzlich Rohtext-Fallback, damit keine archivierte Mail unsichtbar wird. Weiter: Archiv-zuerst-Reihenfolge (mbox_done/target_done) - ein sterbendes Ziel kostet keine Archiv-Kopie mehr; Dedup vergleicht zusaetzlich den Inhalt und schuetzt byte-verschiedene Varianten. WICHTIG: Die 62.073 alten Alias-Zeilen in copied bleiben bewusst erhalten. Sie sehen wie Muell aus, sind aber die Zeilen, auf die der alte Schluessel matcht - ein Loeschen wuerde Mails erneut in fremde Postfaecher kopieren. Verifiziert: go test ./... gruen; 58.051 Archivmails, 0 ohne copied-Zeile; Drift 0 ueber alle 69 Ordner; 2.000 HTTP-Stichproben ueber 8 Ordner: 0 falsche Zuordnung, 0 Panics; Watcher 220 + 60 Mails archived=0 target=0 errors=0. Live als Image fix-identity2-20260716. Co-Authored-By: Claude Opus 4.8 --- .gitignore | 2 + backend/00-router.go | 6 +- backend/02-database.go | 478 ++++++++++++++++++++++++++++++-- backend/04-imap-source.go | 49 ++++ backend/04-imap-source_test.go | 31 +++ backend/06-mbox.go | 181 ++++++++---- backend/06-mbox_test.go | 286 +++++++++++++++++++ backend/07-migrate.go | 223 +++++++++++---- backend/08-viewer.go | 19 +- backend/13-target-dedup.go | 71 +++-- backend/13-target-dedup_test.go | 70 +++++ bug-hashkey-instabil.md | 84 ++++++ bug-htmltotext-panic.md | 106 +++++++ bug-index-drift.md | 88 ++++++ rettungslauf-fixes.md | 137 +++++++++ 15 files changed, 1650 insertions(+), 181 deletions(-) create mode 100644 backend/13-target-dedup_test.go create mode 100644 bug-hashkey-instabil.md create mode 100644 bug-htmltotext-panic.md create mode 100644 bug-index-drift.md create mode 100644 rettungslauf-fixes.md diff --git a/.gitignore b/.gitignore index 38f80bb..57c1735 100644 --- a/.gitignore +++ b/.gitignore @@ -6,8 +6,10 @@ config.json # Lokales Cold Backup (mbox) nicht versionieren backup/ +# Gebaute Binaries (auch datierte Build-Varianten wie mail-graveyard-fix-identity-20260716) /mail-graveyard /mail-graveyard.exe +/mail-graveyard-* # Bun / Frontend frontend-js/node_modules/ diff --git a/backend/00-router.go b/backend/00-router.go index 2ab5beb..821fc3a 100644 --- a/backend/00-router.go +++ b/backend/00-router.go @@ -1305,7 +1305,7 @@ func renderTransferPreview(kind, value, folder string, index int, uid uint32) st decodeHeader(msg.Header.Get("Subject")), decodeHeader(msg.Header.Get("From")), decodeHeader(msg.Header.Get("Date")), - messageBody(raw), + messageBodyWithContext(raw, fmt.Sprintf("transfer kind=%q value=%q folder=%q index=%d uid=%d", kind, value, folder, index, uid)), ) return b.String() } @@ -1621,7 +1621,7 @@ func renderSourceMailboxReadMessage(account Account, folder string, raw RawMessa decodeHeader(msg.Header.Get("Subject")), decodeHeader(msg.Header.Get("From")), decodeHeader(msg.Header.Get("Date")), - messageBody(raw.Body), + messageBodyWithContext(raw.Body, fmt.Sprintf("source account=%q folder=%q", account.Name, folder)), ) return b.String() } @@ -1809,7 +1809,7 @@ func renderTargetMailboxReadMessage(account Account, folder string, raw RawMessa decodeHeader(msg.Header.Get("Subject")), decodeHeader(msg.Header.Get("From")), decodeHeader(msg.Header.Get("Date")), - messageBody(raw.Body), + messageBodyWithContext(raw.Body, fmt.Sprintf("target account=%q folder=%q", account.Name, folder)), ) return b.String() } diff --git a/backend/02-database.go b/backend/02-database.go index fd0923d..b972cdd 100644 --- a/backend/02-database.go +++ b/backend/02-database.go @@ -75,7 +75,13 @@ func ConnectDB(initDB bool) error { if err != nil { return err } - if _, err := db.Exec(`PRAGMA journal_mode=WAL; PRAGMA foreign_keys=ON;`); err != nil { + // SQLite pragmas are connection-local. Keep a single process-local + // connection so every query uses the busy timeout configured here. The web + // app and watcher are separate processes; WAL + busy_timeout coordinates + // their writes without failing immediately with SQLITE_BUSY. + db.SetMaxOpenConns(1) + db.SetMaxIdleConns(1) + if _, err := db.Exec(`PRAGMA busy_timeout=10000; PRAGMA journal_mode=WAL; PRAGMA foreign_keys=ON;`); err != nil { _ = db.Close() return err } @@ -115,14 +121,18 @@ func ConnectDB(initDB bool) error { account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, folder TEXT NOT NULL, message_id TEXT NOT NULL, + body_sha256 TEXT NOT NULL DEFAULT '', + mbox_done INTEGER NOT NULL DEFAULT 0, + target_done INTEGER NOT NULL DEFAULT 0, copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, - UNIQUE(account_id, folder, message_id) + UNIQUE(account_id, folder, message_id, body_sha256) )`, `CREATE TABLE IF NOT EXISTS mbox_index( account_id INTEGER NOT NULL, folder TEXT NOT NULL, seq INTEGER NOT NULL, message_id TEXT NOT NULL, + body_sha256 TEXT NOT NULL DEFAULT '', subject TEXT NOT NULL DEFAULT '', from_addr TEXT NOT NULL DEFAULT '', date TEXT NOT NULL DEFAULT '', @@ -130,7 +140,7 @@ func ConnectDB(initDB bool) error { frame_len INTEGER NOT NULL, inner_offset INTEGER NOT NULL DEFAULT 0, inner_len INTEGER NOT NULL, - UNIQUE(account_id, folder, message_id) + UNIQUE(account_id, folder, seq) )`, `CREATE TABLE IF NOT EXISTS mbox_index_state( account_id INTEGER NOT NULL, @@ -183,6 +193,14 @@ func ConnectDB(initDB bool) error { _ = db.Close() return err } + if err := ensureCopiedStageColumns(); err != nil { + _ = db.Close() + return err + } + if err := ensureMessageIdentitySchema(); err != nil { + _ = db.Close() + return err + } if err := seedArchiveMailboxesFromFS(); err != nil { _ = db.Close() return err @@ -190,6 +208,153 @@ func ConnectDB(initDB bool) error { return nil } +// ensureMessageIdentitySchema upgrades the old Message-ID-only bookkeeping in +// one transaction. No logical rows are discarded: old keys remain as rows with +// an empty body hash and continue to act as compatibility aliases. The mbox +// index is deliberately invalidated so its hashes are rebuilt from the archive +// before the next migration pass makes a copy decision. +func ensureMessageIdentitySchema() error { + copiedColumns, err := tableColumns("copied") + if err != nil { + return err + } + indexColumns, err := tableColumns("mbox_index") + if err != nil { + return err + } + upgradeCopied := !copiedColumns["body_sha256"] + upgradeIndex := !indexColumns["body_sha256"] + if !upgradeCopied && !upgradeIndex { + _, err := DB.Exec(`CREATE INDEX IF NOT EXISTS idx_copied_identity ON copied(account_id, folder, message_id, body_sha256); + CREATE INDEX IF NOT EXISTS idx_mbox_index_identity ON mbox_index(account_id, folder, message_id, body_sha256);`) + return err + } + tx, err := DB.Begin() + if err != nil { + return err + } + defer tx.Rollback() + if upgradeCopied { + statements := []string{ + `ALTER TABLE copied RENAME TO copied_pre_identity`, + `CREATE TABLE copied( + account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE, + folder TEXT NOT NULL, + message_id TEXT NOT NULL, + body_sha256 TEXT NOT NULL DEFAULT '', + mbox_done INTEGER NOT NULL DEFAULT 0, + target_done INTEGER NOT NULL DEFAULT 0, + copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, + UNIQUE(account_id, folder, message_id, body_sha256) + )`, + `INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done, copied_at) + SELECT account_id, folder, message_id, '', mbox_done, target_done, copied_at FROM copied_pre_identity`, + `DROP TABLE copied_pre_identity`, + } + for _, stmt := range statements { + if _, err := tx.Exec(stmt); err != nil { + return err + } + } + } + if upgradeIndex { + statements := []string{ + `ALTER TABLE mbox_index RENAME TO mbox_index_pre_identity`, + `CREATE TABLE mbox_index( + account_id INTEGER NOT NULL, + folder TEXT NOT NULL, + seq INTEGER NOT NULL, + message_id TEXT NOT NULL, + body_sha256 TEXT NOT NULL DEFAULT '', + subject TEXT NOT NULL DEFAULT '', + from_addr TEXT NOT NULL DEFAULT '', + date TEXT NOT NULL DEFAULT '', + file_offset INTEGER NOT NULL, + frame_len INTEGER NOT NULL, + inner_offset INTEGER NOT NULL DEFAULT 0, + inner_len INTEGER NOT NULL, + UNIQUE(account_id, folder, seq) + )`, + `INSERT INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len) + SELECT account_id, folder, seq, message_id, '', subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len FROM mbox_index_pre_identity`, + `DROP TABLE mbox_index_pre_identity`, + `UPDATE mbox_index_state SET indexed_bytes=-1`, + } + for _, stmt := range statements { + if _, err := tx.Exec(stmt); err != nil { + return err + } + } + } + if _, err := tx.Exec(`CREATE INDEX IF NOT EXISTS idx_copied_identity ON copied(account_id, folder, message_id, body_sha256)`); err != nil { + return err + } + if _, err := tx.Exec(`CREATE INDEX IF NOT EXISTS idx_mbox_index_identity ON mbox_index(account_id, folder, message_id, body_sha256)`); err != nil { + return err + } + return tx.Commit() +} + +func tableColumns(table string) (map[string]bool, error) { + rows, err := DB.Query(`PRAGMA table_info(` + table + `)`) + if err != nil { + return nil, err + } + defer rows.Close() + columns := map[string]bool{} + for rows.Next() { + var cid, notNull, pk int + var name, typ string + var defaultValue any + if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil { + return nil, err + } + columns[name] = true + } + return columns, rows.Err() +} + +func ensureCopiedStageColumns() error { + columns := map[string]bool{} + rows, err := DB.Query(`PRAGMA table_info(copied)`) + if err != nil { + return err + } + for rows.Next() { + var cid int + var name, typ string + var notNull int + var defaultValue any + var pk int + if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil { + _ = rows.Close() + return err + } + columns[name] = true + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + // Existing copied rows were created by the old all-or-nothing pipeline and + // therefore represent both stages as complete. DEFAULT 1 preserves that + // truth while all new writes below set their stage values explicitly. + if !columns["mbox_done"] { + if _, err := DB.Exec(`ALTER TABLE copied ADD COLUMN mbox_done INTEGER NOT NULL DEFAULT 1`); err != nil { + return err + } + } + if !columns["target_done"] { + if _, err := DB.Exec(`ALTER TABLE copied ADD COLUMN target_done INTEGER NOT NULL DEFAULT 1`); err != nil { + return err + } + } + return nil +} + func ensureDBFile(path string, initDB bool) error { info, err := os.Stat(path) if err == nil { @@ -217,7 +382,6 @@ func ensureAppUserColumns() error { if err != nil { return err } - defer rows.Close() for rows.Next() { var cid int var name, typ string @@ -225,6 +389,7 @@ func ensureAppUserColumns() error { var defaultValue any var pk int if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil { + _ = rows.Close() return err } if name == "display_name" { @@ -232,6 +397,10 @@ func ensureAppUserColumns() error { } } if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { return err } if !hasDisplayName { @@ -518,22 +687,129 @@ func seedArchiveMailboxesFromFS() error { } func AlreadyCopied(accountID int64, folder, messageID string) (bool, error) { - if messageID == "" { - return false, nil - } - var x int - err := DB.QueryRow(`SELECT 1 FROM copied WHERE account_id=? AND folder=? AND message_id=?`, accountID, folder, messageID).Scan(&x) - if errors.Is(err, sql.ErrNoRows) { - return false, nil - } - return err == nil, err + state, err := GetCopyState(accountID, folder, messageID) + return state.MboxDone && state.TargetDone, err } func MarkCopied(accountID int64, folder, messageID string) error { - if messageID == "" { + return MarkIdentityCopied(accountID, folder, MessageIdentity{MessageID: messageID}) +} + +type CopyState struct { + MboxDone bool + TargetDone bool +} + +func GetCopyState(accountID int64, folder, messageID string) (CopyState, error) { + return GetCopyIdentityState(accountID, folder, MessageIdentity{MessageID: messageID}) +} + +func MarkMboxCopied(accountID int64, folder, messageID string) error { + return MarkIdentityMboxCopied(accountID, folder, MessageIdentity{MessageID: messageID}) +} + +func MarkTargetCopied(accountID int64, folder, messageID string) error { + return MarkIdentityTargetCopied(accountID, folder, MessageIdentity{MessageID: messageID}) +} + +func GetCopyIdentityState(accountID int64, folder string, identity MessageIdentity) (CopyState, error) { + return GetCopyIdentityStateWithFolderAlias(accountID, folder, folder, identity) +} + +func GetCopyIdentityStateWithFolderAlias(accountID int64, folder, legacyFolder string, identity MessageIdentity) (CopyState, error) { + if identity.MessageID == "" && identity.BodySHA256 == "" { + return CopyState{}, nil + } + folders := []string{folder} + if legacyFolder != "" && legacyFolder != folder { + folders = append(folders, legacyFolder) + } + // Precise rows always win. For Message-ID-less mail the hash is the whole + // identity, irrespective of which legacy key happened to be stored beside it. + var mboxDone, targetDone int + for _, candidateFolder := range folders { + var mbox, target int + var err error + if identity.MessageID == "" { + err = DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied + WHERE account_id=? AND folder=? AND body_sha256=?`, accountID, candidateFolder, identity.BodySHA256).Scan(&mbox, &target) + } else { + err = DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied + WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=?`, + accountID, candidateFolder, identity.MessageID, identity.BodySHA256).Scan(&mbox, &target) + } + if err != nil { + return CopyState{}, err + } + mboxDone |= mbox + targetDone |= target + } + if mboxDone != 0 || targetDone != 0 { + return CopyState{MboxDone: mboxDone != 0, TargetDone: targetDone != 0}, nil + } + // Once a Message-ID has precise archive identities, a different body with + // that same ID is new and must not be hidden by the old wildcard row. + if identity.MessageID != "" && identity.BodySHA256 != "" { + var precise int + for _, candidateFolder := range folders { + var count int + if err := DB.QueryRow(`SELECT count(*) FROM copied WHERE account_id=? AND folder=? AND message_id=? AND body_sha256<>''`, + accountID, candidateFolder, identity.MessageID).Scan(&count); err != nil { + return CopyState{}, err + } + precise += count + } + if precise > 0 { + return CopyState{}, nil + } + } + aliases := []string{identity.MessageID, identity.LegacyMessageID} + if identity.MessageID == "" && identity.BodySHA256 != "" { + aliases = append(aliases, "sha256:"+identity.BodySHA256) + } + seen := map[string]bool{} + for _, alias := range aliases { + alias = strings.TrimSpace(alias) + if alias == "" || seen[alias] { + continue + } + seen[alias] = true + for _, candidateFolder := range folders { + var mbox, target int + err := DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied + WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=''`, accountID, candidateFolder, alias).Scan(&mbox, &target) + if err != nil { + return CopyState{}, err + } + mboxDone |= mbox + targetDone |= target + } + } + return CopyState{MboxDone: mboxDone != 0, TargetDone: targetDone != 0}, nil +} + +func MarkIdentityCopied(accountID int64, folder string, identity MessageIdentity) error { + return markCopyIdentity(accountID, folder, identity, true, true) +} + +func MarkIdentityMboxCopied(accountID int64, folder string, identity MessageIdentity) error { + return markCopyIdentity(accountID, folder, identity, true, false) +} + +func MarkIdentityTargetCopied(accountID int64, folder string, identity MessageIdentity) error { + return markCopyIdentity(accountID, folder, identity, false, true) +} + +func markCopyIdentity(accountID int64, folder string, identity MessageIdentity, mboxDone, targetDone bool) error { + if identity.MessageID == "" && identity.BodySHA256 == "" { return nil } - _, err := DB.Exec(`INSERT OR IGNORE INTO copied(account_id, folder, message_id) VALUES(?,?,?)`, accountID, folder, messageID) + _, err := DB.Exec(`INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done) + VALUES(?,?,?,?,?,?) + ON CONFLICT(account_id, folder, message_id, body_sha256) DO UPDATE SET + mbox_done=MAX(copied.mbox_done, excluded.mbox_done), + target_done=MAX(copied.target_done, excluded.target_done), + copied_at=CURRENT_TIMESTAMP`, accountID, folder, identity.MessageID, identity.BodySHA256, boolInt(mboxDone), boolInt(targetDone)) return err } @@ -542,6 +818,7 @@ type MboxIndexEntry struct { Folder string Seq int MessageID string + BodySHA256 string Subject string From string Date string @@ -552,7 +829,7 @@ type MboxIndexEntry struct { } func SaveMboxIndex(e MboxIndexEntry) error { - if DB == nil || e.MessageID == "" { + if DB == nil || (e.MessageID == "" && e.BodySHA256 == "") { return nil } var seq int @@ -561,9 +838,9 @@ func SaveMboxIndex(e MboxIndexEntry) error { } else { _ = DB.QueryRow(`SELECT COALESCE(MAX(seq), -1) + 1 FROM mbox_index WHERE account_id=? AND folder=?`, e.AccountID, e.Folder).Scan(&seq) } - _, err := DB.Exec(`INSERT OR IGNORE INTO mbox_index(account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len) - VALUES(?,?,?,?,?,?,?,?,?,?,?)`, - e.AccountID, e.Folder, seq, e.MessageID, e.Subject, e.From, e.Date, e.FileOffset, e.FrameLen, e.InnerOffset, e.InnerLen) + _, err := DB.Exec(`INSERT OR REPLACE INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len) + VALUES(?,?,?,?,?,?,?,?,?,?,?,?)`, + e.AccountID, e.Folder, seq, e.MessageID, e.BodySHA256, e.Subject, e.From, e.Date, e.FileOffset, e.FrameLen, e.InnerOffset, e.InnerLen) return err } @@ -576,6 +853,53 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry, return err } defer tx.Rollback() + folderAliases := []string{folder} + aliasRows, err := tx.Query(`SELECT folder FROM copied WHERE account_id=? UNION SELECT folder FROM mbox_index WHERE account_id=?`, accountID, accountID) + if err != nil { + return err + } + for aliasRows.Next() { + var candidate string + if err := aliasRows.Scan(&candidate); err != nil { + _ = aliasRows.Close() + return err + } + if candidate != folder && safeMboxName(candidate) == folder { + folderAliases = append(folderAliases, candidate) + } + } + if err := aliasRows.Err(); err != nil { + _ = aliasRows.Close() + return err + } + if err := aliasRows.Close(); err != nil { + return err + } + legacyIDsBySeq := map[int]string{} + for _, candidateFolder := range folderAliases { + rows, err := tx.Query(`SELECT seq, message_id FROM mbox_index WHERE account_id=? AND folder=?`, accountID, candidateFolder) + if err != nil { + return err + } + for rows.Next() { + var seq int + var messageID string + if err := rows.Scan(&seq, &messageID); err != nil { + _ = rows.Close() + return err + } + if _, exists := legacyIDsBySeq[seq]; !exists || candidateFolder == folder { + legacyIDsBySeq[seq] = messageID + } + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + } if _, err := tx.Exec(`DELETE FROM mbox_index WHERE account_id=? AND folder=?`, accountID, folder); err != nil { return err } @@ -583,15 +907,33 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry, entry.AccountID = accountID entry.Folder = folder entry.Seq = i - if entry.MessageID == "" { + if entry.MessageID == "" && entry.BodySHA256 == "" { continue } - if _, err := tx.Exec(`INSERT INTO mbox_index(account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len) - VALUES(?,?,?,?,?,?,?,?,?,?,?)`, - entry.AccountID, entry.Folder, entry.Seq, entry.MessageID, entry.Subject, entry.From, entry.Date, + if _, err := tx.Exec(`INSERT INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len) + VALUES(?,?,?,?,?,?,?,?,?,?,?,?)`, + entry.AccountID, entry.Folder, entry.Seq, entry.MessageID, entry.BodySHA256, entry.Subject, entry.From, entry.Date, entry.FileOffset, entry.FrameLen, entry.InnerOffset, entry.InnerLen); err != nil { return err } + // A full rebuild recovers records which reached the old append-first + // pipeline but whose index/copied writes lost a SQLITE_BUSY race. In that + // pipeline the target append happened before the mbox append, so recovered + // records are complete in both stages. Marking them prevents a later watch + // run from appending duplicates to either destination. + state, found, err := copyStateForReindex(tx, entry.AccountID, folderAliases, entry, legacyIDsBySeq[i]) + if err != nil { + return err + } + if !found { + state = CopyState{MboxDone: true, TargetDone: true} + } + if _, err := tx.Exec(`INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done) + VALUES(?,?,?,?,?,?) + ON CONFLICT(account_id, folder, message_id, body_sha256) DO UPDATE SET mbox_done=1`, + entry.AccountID, entry.Folder, entry.MessageID, entry.BodySHA256, 1, boolInt(state.TargetDone)); err != nil { + return err + } } if _, err := tx.Exec(`INSERT INTO mbox_index_state(account_id, folder, indexed_bytes) VALUES(?,?,?) @@ -602,6 +944,88 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry, return tx.Commit() } +// RemoveMboxIndexFolderAliases removes only superseded index metadata such as +// "INBOX.Newbies " after the actual archive file has been rebuilt under its +// canonical filesystem name "INBOX.Newbies". copied compatibility aliases and +// all mbox files remain untouched. +func RemoveMboxIndexFolderAliases(accountID int64, canonicalFolders map[string]bool) error { + rows, err := DB.Query(`SELECT folder FROM mbox_index WHERE account_id=? UNION SELECT folder FROM mbox_index_state WHERE account_id=?`, accountID, accountID) + if err != nil { + return err + } + var stale []string + for rows.Next() { + var folder string + if err := rows.Scan(&folder); err != nil { + _ = rows.Close() + return err + } + if !canonicalFolders[folder] && canonicalFolders[safeMboxName(folder)] { + stale = append(stale, folder) + } + } + if err := rows.Err(); err != nil { + _ = rows.Close() + return err + } + if err := rows.Close(); err != nil { + return err + } + if len(stale) == 0 { + return nil + } + tx, err := DB.Begin() + if err != nil { + return err + } + defer tx.Rollback() + for _, folder := range stale { + if _, err := tx.Exec(`DELETE FROM mbox_index WHERE account_id=? AND folder=?`, accountID, folder); err != nil { + return err + } + if _, err := tx.Exec(`DELETE FROM mbox_index_state WHERE account_id=? AND folder=?`, accountID, folder); err != nil { + return err + } + } + return tx.Commit() +} + +func copyStateForReindex(tx *sql.Tx, accountID int64, folders []string, entry MboxIndexEntry, legacyMessageID string) (CopyState, bool, error) { + type candidate struct{ messageID, bodyHash string } + candidates := []candidate{{entry.MessageID, entry.BodySHA256}} + if legacyMessageID != "" { + candidates = append(candidates, candidate{legacyMessageID, ""}) + } + if entry.MessageID != "" { + candidates = append(candidates, candidate{entry.MessageID, ""}) + } else if entry.BodySHA256 != "" { + candidates = append(candidates, candidate{"sha256:" + entry.BodySHA256, ""}) + } + seen := map[candidate]bool{} + var state CopyState + found := false + for _, c := range candidates { + if seen[c] { + continue + } + seen[c] = true + for _, folder := range folders { + var count, mboxDone, targetDone int + if err := tx.QueryRow(`SELECT count(*), COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) + FROM copied WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=?`, + accountID, folder, c.messageID, c.bodyHash).Scan(&count, &mboxDone, &targetDone); err != nil { + return CopyState{}, false, err + } + if count > 0 { + found = true + state.MboxDone = state.MboxDone || mboxDone != 0 + state.TargetDone = state.TargetDone || targetDone != 0 + } + } + } + return state, found, nil +} + func UpdateMboxIndexState(accountID int64, folder string, indexedBytes int64) error { if DB == nil || accountID == 0 { return nil @@ -629,7 +1053,7 @@ func ListMboxIndex(accountID int64, folder string) ([]MboxIndexEntry, error) { if DB == nil || accountID == 0 { return nil, sql.ErrNoRows } - rows, err := DB.Query(`SELECT account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len + rows, err := DB.Query(`SELECT account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len FROM mbox_index WHERE account_id=? AND folder=? ORDER BY seq`, accountID, folder) if err != nil { return nil, err @@ -638,7 +1062,7 @@ func ListMboxIndex(accountID int64, folder string) ([]MboxIndexEntry, error) { var out []MboxIndexEntry for rows.Next() { var e MboxIndexEntry - if err := rows.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen); err != nil { + if err := rows.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.BodySHA256, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen); err != nil { return nil, err } out = append(out, e) @@ -650,10 +1074,10 @@ func GetMboxIndex(accountID int64, folder string, seq int) (MboxIndexEntry, erro if DB == nil || accountID == 0 { return MboxIndexEntry{}, sql.ErrNoRows } - row := DB.QueryRow(`SELECT account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len + row := DB.QueryRow(`SELECT account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len FROM mbox_index WHERE account_id=? AND folder=? AND seq=?`, accountID, folder, seq) var e MboxIndexEntry - err := row.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen) + err := row.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.BodySHA256, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen) return e, err } diff --git a/backend/04-imap-source.go b/backend/04-imap-source.go index 5b710a4..46daaad 100644 --- a/backend/04-imap-source.go +++ b/backend/04-imap-source.go @@ -1,6 +1,7 @@ package backend import ( + "bytes" "crypto/sha256" "crypto/tls" "errors" @@ -25,6 +26,17 @@ type RawMessage struct { InternalDate time.Time // Original-Zeit, per APPEND erhalten } +// MessageIdentity is stable across the IMAP and mbox representations. A real +// Message-ID is not sufficient on its own: broken mailers sometimes reuse it +// for byte-different messages. Messages without a Message-ID are identified by +// BodySHA256 alone. LegacyMessageID keeps the old raw-IMAP hash addressable +// while existing databases are migrated without re-copying mail. +type MessageIdentity struct { + MessageID string + BodySHA256 string + LegacyMessageID string +} + type MessageHeader struct { UID uint32 MessageID string @@ -475,6 +487,43 @@ func messageID(body []byte) string { return fmt.Sprintf("sha256:%x", sha256.Sum256(body)) } +func identityForRawMessage(m RawMessage) MessageIdentity { + id := normalizeMessageID(m.MessageID) + legacyID := id + if strings.HasPrefix(strings.ToLower(id), "sha256:") { + id = "" + } + if id == "" { + if msg, err := mail.ReadMessage(bytes.NewReader(m.Body)); err == nil { + id = normalizeMessageID(msg.Header.Get("Message-ID")) + } + } + return MessageIdentity{ + MessageID: id, + BodySHA256: bodySHA256(m.Body), + LegacyMessageID: legacyID, + } +} + +// canonicalMessageBytes mirrors the bytes which can be recovered from the +// current mbox writer/reader pair: line endings are LF, trailing record +// separators are removed and mbox's >From escaping is undone. Hashing this +// representation on both paths prevents the raw-IMAP/mbox hash split. +func canonicalMessageBytes(raw []byte) []byte { + raw = bytes.ReplaceAll(raw, []byte("\r\n"), []byte("\n")) + lines := bytes.Split(raw, []byte("\n")) + for i, line := range lines { + if bytes.HasPrefix(line, []byte(">From ")) { + lines[i] = line[1:] + } + } + return bytes.TrimRight(bytes.Join(lines, []byte("\n")), "\n") +} + +func bodySHA256(raw []byte) string { + return fmt.Sprintf("%x", sha256.Sum256(canonicalMessageBytes(raw))) +} + func normalizeMessageID(id string) string { return strings.Trim(strings.TrimSpace(id), "<>") } diff --git a/backend/04-imap-source_test.go b/backend/04-imap-source_test.go index 1d05415..ef5d5ad 100644 --- a/backend/04-imap-source_test.go +++ b/backend/04-imap-source_test.go @@ -24,3 +24,34 @@ func TestMessageIDFallsBackToBodyHash(t *testing.T) { t.Fatal("messageID hash fallback is not stable") } } + +func TestBodySHA256IsStableAcrossMboxNormalization(t *testing.T) { + raw := []byte("From: a@example.com\r\nSubject: Test\r\n\r\nFirst\r\nFrom escaped\r\n\r\n") + record := mboxRecord(RawMessage{Body: raw}) + parts := splitMboxRecordsWithOffsets(record) + if len(parts) != 1 { + t.Fatalf("mbox parts=%d, want 1", len(parts)) + } + if got, want := bodySHA256(parts[0].Message), bodySHA256(raw); got != want { + t.Fatalf("mbox hash=%s, raw hash=%s", got, want) + } +} + +func TestIdentityUsesMessageIDAndBodyHash(t *testing.T) { + first := identityForRawMessage(RawMessage{MessageID: "", Body: []byte("Message-ID: \r\n\r\nfirst")}) + second := identityForRawMessage(RawMessage{MessageID: "", Body: []byte("Message-ID: \r\n\r\nsecond")}) + if first.MessageID != second.MessageID { + t.Fatalf("message IDs differ: %#v %#v", first, second) + } + if first.BodySHA256 == second.BodySHA256 { + t.Fatalf("different bodies have same identity hash: %#v %#v", first, second) + } +} + +func TestIdentityWithoutMessageIDKeepsLegacyRawHash(t *testing.T) { + body := []byte("From: a@example.com\r\n\r\nbody") + identity := identityForRawMessage(RawMessage{MessageID: messageID(body), Body: body}) + if identity.MessageID != "" || identity.BodySHA256 == "" || !strings.HasPrefix(identity.LegacyMessageID, "sha256:") { + t.Fatalf("unexpected identity: %#v", identity) + } +} diff --git a/backend/06-mbox.go b/backend/06-mbox.go index e005786..5963fc1 100644 --- a/backend/06-mbox.go +++ b/backend/06-mbox.go @@ -6,6 +6,7 @@ import ( "fmt" "html" "io" + "log" "mime" "mime/multipart" "mime/quotedprintable" @@ -118,7 +119,7 @@ func mboxAppendInfoFromMessage(m RawMessage, record []byte) MboxAppendInfo { parts := splitMboxRecordsWithOffsets(record) if len(parts) > 0 { info.InnerOffset = parts[0].Offset - info.InnerLen = int64(len(parts[0].Message)) + info.InnerLen = int64(parts[0].Length) } msg, err := mail.ReadMessage(bytes.NewReader(m.Body)) if err != nil { @@ -274,6 +275,7 @@ func ReindexArchives(name string) error { func reindexAccountArchive(account Account) error { dir := filepath.Join(Cfg.MboxRoot, accountMboxDir(account)) + canonicalFolders := map[string]bool{} for _, path := range archiveMboxPaths(dir) { folder := folderNameFromMboxPath(path) if folder == "" { @@ -282,8 +284,9 @@ func reindexAccountArchive(account Account) error { if err := ReindexMboxFile(account.ID, folder, path); err != nil { return fmt.Errorf("%s %s: %w", account.Name, folder, err) } + canonicalFolders[folder] = true } - return nil + return RemoveMboxIndexFolderAliases(account.ID, canonicalFolders) } func reindexPlainMbox(path string) ([]MboxIndexEntry, error) { @@ -294,7 +297,7 @@ func reindexPlainMbox(path string) ([]MboxIndexEntry, error) { var out []MboxIndexEntry for _, part := range splitMboxRecordsWithOffsets(b) { entry := indexEntryFromMboxRecord(part.Message, part.Offset, int64(part.Length), 0, int64(part.Length)) - if entry.MessageID != "" { + if entry.MessageID != "" || entry.BodySHA256 != "" { out = append(out, entry) } } @@ -318,8 +321,8 @@ func reindexZstdMbox(path string) ([]MboxIndexEntry, error) { return nil, err } for _, part := range splitMboxRecordsWithOffsets(record) { - entry := indexEntryFromMboxRecord(part.Message, frame.Offset, int64(frame.Length), part.Offset, int64(len(part.Message))) - if entry.MessageID != "" { + entry := indexEntryFromMboxRecord(part.Message, frame.Offset, int64(frame.Length), part.Offset, int64(part.Length)) + if entry.MessageID != "" || entry.BodySHA256 != "" { out = append(out, entry) } } @@ -334,39 +337,44 @@ type mboxRecordPart struct { } func splitMboxRecordsWithOffsets(b []byte) []mboxRecordPart { - normalized := bytes.ReplaceAll(b, []byte("\r\n"), []byte("\n")) - lines := bytes.SplitAfter(normalized, []byte("\n")) var out []mboxRecordPart - var cur bytes.Buffer - var curStart int64 - var pos int64 - inMsg := false - for _, lineWithNL := range lines { - line := bytes.TrimSuffix(lineWithNL, []byte("\n")) - if bytes.HasPrefix(line, []byte("From ")) { - if inMsg && cur.Len() > 0 { - msg := bytes.TrimRight(cur.Bytes(), "\n") - out = append(out, mboxRecordPart{Offset: curStart, Length: int(pos - curStart), Message: append([]byte(nil), msg...)}) - cur.Reset() - } - inMsg = true - pos += int64(len(lineWithNL)) - curStart = pos - continue + messageStart := -1 + appendRecord := func(end int) { + if messageStart < 0 || end <= messageStart { + return } - if inMsg { - if bytes.HasPrefix(line, []byte(">From ")) { - line = line[1:] - } - _, _ = cur.Write(line) - _ = cur.WriteByte('\n') + raw := b[messageStart:end] + message := unescapeMboxMessage(raw) + if len(message) == 0 { + return } - pos += int64(len(lineWithNL)) + out = append(out, mboxRecordPart{ + Offset: int64(messageStart), + Length: end - messageStart, + Message: append([]byte(nil), message...), + }) } - if inMsg && cur.Len() > 0 { - msg := bytes.TrimRight(cur.Bytes(), "\n") - out = append(out, mboxRecordPart{Offset: curStart, Length: int(pos - curStart), Message: append([]byte(nil), msg...)}) + for lineStart := 0; lineStart < len(b); { + relNL := bytes.IndexByte(b[lineStart:], '\n') + lineEnd, nextLine := len(b), len(b) + if relNL >= 0 { + lineEnd = lineStart + relNL + nextLine = lineEnd + 1 + } + contentEnd := lineEnd + if contentEnd > lineStart && b[contentEnd-1] == '\r' { + contentEnd-- + } + if bytes.HasPrefix(b[lineStart:contentEnd], []byte("From ")) { + appendRecord(lineStart) + messageStart = nextLine + } + if nextLine >= len(b) { + break + } + lineStart = nextLine } + appendRecord(len(b)) return out } @@ -399,15 +407,12 @@ func splitZstdFrames(b []byte) []zstdFramePart { } func indexEntryFromMboxRecord(raw []byte, fileOffset, frameLen, innerOffset, innerLen int64) MboxIndexEntry { - entry := MboxIndexEntry{FileOffset: fileOffset, FrameLen: frameLen, InnerOffset: innerOffset, InnerLen: innerLen} + entry := MboxIndexEntry{BodySHA256: bodySHA256(raw), FileOffset: fileOffset, FrameLen: frameLen, InnerOffset: innerOffset, InnerLen: innerLen} msg, err := mail.ReadMessage(bytes.NewReader(raw)) if err != nil { return entry } entry.MessageID = normalizeMessageID(msg.Header.Get("Message-ID")) - if entry.MessageID == "" { - entry.MessageID = messageID(raw) - } entry.Subject = decodeHeader(msg.Header.Get("Subject")) entry.From = decodeHeader(msg.Header.Get("From")) entry.Date = decodeHeader(msg.Header.Get("Date")) @@ -415,19 +420,7 @@ func indexEntryFromMboxRecord(raw []byte, fileOffset, frameLen, innerOffset, inn } func unescapeMboxMessage(raw []byte) []byte { - raw = bytes.ReplaceAll(raw, []byte("\r\n"), []byte("\n")) - lines := bytes.Split(raw, []byte("\n")) - var b bytes.Buffer - for i, line := range lines { - if i > 0 { - _ = b.WriteByte('\n') - } - if bytes.HasPrefix(line, []byte(">From ")) { - line = line[1:] - } - _, _ = b.Write(line) - } - return bytes.TrimRight(b.Bytes(), "\n") + return canonicalMessageBytes(raw) } func readMboxMessagesBytes(b []byte) [][]byte { @@ -522,9 +515,6 @@ func readMboxMessageFromIndex(path string, index int) ([]byte, bool, error) { if err != nil { return nil, false, nil } - if !strings.HasSuffix(strings.ToLower(path), ".zst") { - return nil, false, nil - } f, err := os.Open(path) if err != nil { return nil, true, err @@ -533,10 +523,28 @@ func readMboxMessageFromIndex(path string, index int) ([]byte, bool, error) { if _, err := f.Seek(entry.FileOffset, io.SeekStart); err != nil { return nil, true, err } + if entry.FrameLen <= 0 || entry.FrameLen > int64(^uint(0)>>1) { + return nil, true, fmt.Errorf("invalid indexed mbox length %d", entry.FrameLen) + } frame := make([]byte, entry.FrameLen) if _, err := io.ReadFull(f, frame); err != nil { return nil, true, err } + if !strings.HasSuffix(strings.ToLower(path), ".zst") { + // Entries written during migration point at the complete mbox record + // (separator included, InnerOffset > 0). Entries produced by a full + // rebuild point directly at the raw message bytes (InnerOffset == 0). + // Both variants are read by their persisted byte range, never by the + // message's positional number in a freshly split file. + if entry.InnerOffset > 0 { + messages := readMboxMessagesBytes(frame) + if len(messages) != 1 { + return nil, true, fmt.Errorf("invalid indexed mbox record at offset %d", entry.FileOffset) + } + return messages[0], true, nil + } + return unescapeMboxMessage(frame), true, nil + } dec, err := zstd.NewReader(nil) if err != nil { return nil, true, err @@ -605,6 +613,29 @@ func decodeHeader(v string) string { } func messageBody(raw []byte) string { + return messageBodyWithContext(raw, "") +} + +func messageBodyWithContext(raw []byte, context string) string { + return renderMessageBodySafely(raw, context, func() string { + return messageBodyUnsafe(raw) + }) +} + +func renderMessageBodySafely(raw []byte, context string, render func() string) (body string) { + defer func() { + if recovered := recover(); recovered != nil { + if strings.TrimSpace(context) == "" { + context = "unbekannter Vorschaupfad" + } + log.Printf("mail preview rendering panic (%s): %v", context, recovered) + body = "Darstellung fehlgeschlagen; Rohfassung:\n\n" + string(raw) + } + }() + return render() +} + +func messageBodyUnsafe(raw []byte) string { msg, err := mail.ReadMessage(bytes.NewReader(raw)) if err != nil { return string(raw) @@ -738,11 +769,12 @@ func markHTMLBreaks(s string) string { } func replaceCaseInsensitive(s, old, new string) string { + if old == "" { + return s + } var b strings.Builder - lower := strings.ToLower(s) - needle := strings.ToLower(old) for { - i := strings.Index(lower, needle) + i := asciiFoldIndex(s, old) if i < 0 { b.WriteString(s) return b.String() @@ -751,29 +783,56 @@ func replaceCaseInsensitive(s, old, new string) string { b.WriteString(new) cut := i + len(old) s = s[cut:] - lower = lower[cut:] } } func stripHTMLBlock(s, tag string) string { - lower := strings.ToLower(s) open := "<" + tag close := "" for { - start := strings.Index(lower, open) + start := asciiFoldIndex(s, open) if start < 0 { return s } - end := strings.Index(lower[start:], close) + end := asciiFoldIndex(s[start:], close) if end < 0 { return s[:start] } end += start + len(close) s = s[:start] + " " + s[end:] - lower = strings.ToLower(s) } } +// asciiFoldIndex searches an ASCII needle case-insensitively in s. It scans +// the original bytes, so the returned index is always valid for slicing s even +// when s contains Unicode characters whose lowercase form has another byte +// length (for example Turkish dotted I or the Kelvin sign). +func asciiFoldIndex(s, needle string) int { + if needle == "" || len(s) < len(needle) { + return -1 + } + for i := 0; i+len(needle) <= len(s); i++ { + matched := true + for j := 0; j < len(needle); j++ { + if asciiLower(s[i+j]) != asciiLower(needle[j]) { + matched = false + break + } + } + if matched { + return i + } + } + return -1 +} + +func asciiLower(c byte) byte { + if c >= 'A' && c <= 'Z' { + return c + ('a' - 'A') + } + return c +} + func decodeTransfer(body []byte, enc string) []byte { switch strings.ToLower(strings.TrimSpace(enc)) { case "quoted-printable": diff --git a/backend/06-mbox_test.go b/backend/06-mbox_test.go index ec495ff..5522bb7 100644 --- a/backend/06-mbox_test.go +++ b/backend/06-mbox_test.go @@ -2,6 +2,7 @@ package backend import ( "bytes" + "database/sql" "os" "path/filepath" "strings" @@ -240,6 +241,291 @@ func TestReadMboxMessagesBytesClonesBufferRecords(t *testing.T) { } } +func TestPlainIndexedReadUsesOffsetAcrossMissingMiddleIndex(t *testing.T) { + oldCfg, oldDB := Cfg, DB + root := t.TempDir() + t.Cleanup(func() { + if DB != nil { + _ = DB.Close() + } + Cfg, DB = oldCfg, oldDB + }) + + Cfg = Config{DBPath: filepath.Join(root, "mail-graveyard.db"), MboxRoot: filepath.Join(root, "backup")} + if err := ConnectDB(true); err != nil { + t.Fatal(err) + } + if err := SaveArchiveMailbox("archive"); err != nil { + t.Fatal(err) + } + if err := SaveAccount(Account{ + Name: "test-account", SrcHost: "source.example", SrcPort: 993, SrcSecurity: "tls", SrcUser: "source@example.com", SrcPass: "x", + DstHost: "target.example", DstPort: 993, DstSecurity: "tls", DstUser: "target@example.com", DstPass: "x", MboxDir: "archive", Active: true, + }); err != nil { + t.Fatal(err) + } + account, err := GetAccount("test-account") + if err != nil { + t.Fatal(err) + } + writer, err := NewMboxWriter(filepath.Join(Cfg.MboxRoot, "archive")) + if err != nil { + t.Fatal(err) + } + firstInfo, err := writer.Append("INBOX", testRawMessage("first@example.com", "First")) + if err != nil { + t.Fatal(err) + } + if err := SaveMboxIndex(MboxIndexEntry{AccountID: account.ID, Folder: "INBOX", MessageID: "first@example.com", Subject: "First", FileOffset: firstInfo.FileOffset, FrameLen: firstInfo.FrameLen, InnerOffset: firstInfo.InnerOffset, InnerLen: firstInfo.InnerLen}); err != nil { + t.Fatal(err) + } + if _, err := writer.Append("INBOX", testRawMessage("missing@example.com", "Missing middle")); err != nil { + t.Fatal(err) + } + thirdInfo, err := writer.Append("INBOX", testRawMessage("third@example.com", "Third")) + if err != nil { + t.Fatal(err) + } + if err := SaveMboxIndex(MboxIndexEntry{AccountID: account.ID, Folder: "INBOX", MessageID: "third@example.com", Subject: "Third", FileOffset: thirdInfo.FileOffset, FrameLen: thirdInfo.FrameLen, InnerOffset: thirdInfo.InnerOffset, InnerLen: thirdInfo.InnerLen}); err != nil { + t.Fatal(err) + } + if err := UpdateMboxIndexState(account.ID, "INBOX", thirdInfo.FileOffset+thirdInfo.FrameLen); err != nil { + t.Fatal(err) + } + + raw, err := ReadMboxMessage(thirdInfo.Path, 1) + if err != nil { + t.Fatal(err) + } + if !bytes.Contains(raw, []byte("Subject: Third")) || bytes.Contains(raw, []byte("Missing middle")) { + t.Fatalf("indexed read drifted to the file position: %q", raw) + } + + if err := ReindexMboxFile(account.ID, "INBOX", thirdInfo.Path); err != nil { + t.Fatal(err) + } + for i, subject := range []string{"First", "Missing middle", "Third"} { + raw, err := ReadMboxMessage(thirdInfo.Path, i) + if err != nil { + t.Fatalf("rebuilt index %d: %v", i, err) + } + if !bytes.Contains(raw, []byte("Subject: "+subject)) { + t.Fatalf("rebuilt index %d returned wrong message: %q", i, raw) + } + } +} + +func TestConnectDBConfiguresBusyTimeoutAndCopyStages(t *testing.T) { + oldCfg, oldDB := Cfg, DB + root := t.TempDir() + t.Cleanup(func() { + if DB != nil { + _ = DB.Close() + } + Cfg, DB = oldCfg, oldDB + }) + Cfg = Config{DBPath: filepath.Join(root, "mail-graveyard.db"), MboxRoot: filepath.Join(root, "backup")} + if err := ConnectDB(true); err != nil { + t.Fatal(err) + } + var timeout int + if err := DB.QueryRow(`PRAGMA busy_timeout`).Scan(&timeout); err != nil { + t.Fatal(err) + } + if timeout != 10000 { + t.Fatalf("busy_timeout=%d, want 10000", timeout) + } + if err := SaveAccount(Account{ + Name: "stage-account", SrcHost: "source.example", SrcPort: 993, SrcSecurity: "tls", SrcUser: "source@example.com", SrcPass: "x", + DstHost: "target.example", DstPort: 993, DstSecurity: "tls", DstUser: "target@example.com", DstPass: "x", Active: true, + }); err != nil { + t.Fatal(err) + } + account, err := GetAccount("stage-account") + if err != nil { + t.Fatal(err) + } + if err := MarkMboxCopied(account.ID, "INBOX", "stage@example.com"); err != nil { + t.Fatal(err) + } + state, err := GetCopyState(account.ID, "INBOX", "stage@example.com") + if err != nil { + t.Fatal(err) + } + if !state.MboxDone || state.TargetDone { + t.Fatalf("unexpected mbox-only state: %#v", state) + } + if err := MarkTargetCopied(account.ID, "INBOX", "stage@example.com"); err != nil { + t.Fatal(err) + } + state, err = GetCopyState(account.ID, "INBOX", "stage@example.com") + if err != nil { + t.Fatal(err) + } + if !state.MboxDone || !state.TargetDone { + t.Fatalf("unexpected completed state: %#v", state) + } + first := MessageIdentity{MessageID: "reused@example.com", BodySHA256: bodySHA256([]byte("first"))} + second := MessageIdentity{MessageID: "reused@example.com", BodySHA256: bodySHA256([]byte("second"))} + if err := MarkIdentityCopied(account.ID, "INBOX", first); err != nil { + t.Fatal(err) + } + state, err = GetCopyIdentityState(account.ID, "INBOX", second) + if err != nil { + t.Fatal(err) + } + if state.MboxDone || state.TargetDone { + t.Fatalf("byte-different message with reused ID was hidden: %#v", state) + } +} + +func TestIdentitySchemaUpgradePreservesLegacyAliases(t *testing.T) { + oldCfg, oldDB := Cfg, DB + root := t.TempDir() + path := filepath.Join(root, "legacy.db") + rawDB, err := sql.Open("sqlite", path) + if err != nil { + t.Fatal(err) + } + legacySchema := []string{ + `CREATE TABLE accounts(id INTEGER PRIMARY KEY)`, + `INSERT INTO accounts(id) VALUES(7)`, + `CREATE TABLE copied(account_id INTEGER NOT NULL, folder TEXT NOT NULL, message_id TEXT NOT NULL, mbox_done INTEGER NOT NULL DEFAULT 1, target_done INTEGER NOT NULL DEFAULT 1, copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(account_id, folder, message_id))`, + `INSERT INTO copied(account_id, folder, message_id, mbox_done, target_done) VALUES(7, 'INBOX.Newbies ', 'sha256:legacy-raw', 1, 0)`, + `CREATE TABLE mbox_index(account_id INTEGER NOT NULL, folder TEXT NOT NULL, seq INTEGER NOT NULL, message_id TEXT NOT NULL, subject TEXT NOT NULL DEFAULT '', from_addr TEXT NOT NULL DEFAULT '', date TEXT NOT NULL DEFAULT '', file_offset INTEGER NOT NULL, frame_len INTEGER NOT NULL, inner_offset INTEGER NOT NULL DEFAULT 0, inner_len INTEGER NOT NULL, UNIQUE(account_id, folder, message_id))`, + `INSERT INTO mbox_index(account_id, folder, seq, message_id, file_offset, frame_len, inner_len) VALUES(7, 'INBOX.Newbies ', 0, 'sha256:legacy-raw', 0, 10, 10)`, + `CREATE TABLE mbox_index_state(account_id INTEGER NOT NULL, folder TEXT NOT NULL, indexed_bytes INTEGER NOT NULL DEFAULT 0, UNIQUE(account_id, folder))`, + `INSERT INTO mbox_index_state(account_id, folder, indexed_bytes) VALUES(7, 'INBOX.Newbies ', 10)`, + } + for _, stmt := range legacySchema { + if _, err := rawDB.Exec(stmt); err != nil { + _ = rawDB.Close() + t.Fatal(err) + } + } + if err := rawDB.Close(); err != nil { + t.Fatal(err) + } + if DB != nil { + _ = DB.Close() + } + t.Cleanup(func() { + if DB != nil { + _ = DB.Close() + } + Cfg, DB = oldCfg, oldDB + }) + Cfg = Config{DBPath: path, MboxRoot: filepath.Join(root, "backup")} + if err := ConnectDB(false); err != nil { + t.Fatal(err) + } + var messageID, bodyHash string + var mboxDone, targetDone int + if err := DB.QueryRow(`SELECT message_id, body_sha256, mbox_done, target_done FROM copied WHERE account_id=7`).Scan(&messageID, &bodyHash, &mboxDone, &targetDone); err != nil { + t.Fatal(err) + } + if messageID != "sha256:legacy-raw" || bodyHash != "" || mboxDone != 1 || targetDone != 0 { + t.Fatalf("legacy alias changed during upgrade: id=%q hash=%q stages=%d/%d", messageID, bodyHash, mboxDone, targetDone) + } + var indexedBytes int64 + if err := DB.QueryRow(`SELECT indexed_bytes FROM mbox_index_state WHERE account_id=7 AND folder='INBOX.Newbies '`).Scan(&indexedBytes); err != nil { + t.Fatal(err) + } + if indexedBytes != -1 { + t.Fatalf("identity-less index was not invalidated: %d", indexedBytes) + } + stableHash := bodySHA256([]byte("archived body")) + if err := ReplaceMboxIndex(7, "INBOX.Newbies", []MboxIndexEntry{{BodySHA256: stableHash, FrameLen: 10, InnerLen: 10}}, 10); err != nil { + t.Fatal(err) + } + if err := DB.QueryRow(`SELECT mbox_done, target_done FROM copied WHERE account_id=7 AND folder='INBOX.Newbies' AND message_id='' AND body_sha256=?`, stableHash).Scan(&mboxDone, &targetDone); err != nil { + t.Fatal(err) + } + if mboxDone != 1 || targetDone != 0 { + t.Fatalf("reindex lost pending target state from legacy alias: %d/%d", mboxDone, targetDone) + } + if err := RemoveMboxIndexFolderAliases(7, map[string]bool{"INBOX.Newbies": true}); err != nil { + t.Fatal(err) + } + var staleIndex, preservedAlias int + if err := DB.QueryRow(`SELECT count(*) FROM mbox_index WHERE account_id=7 AND folder='INBOX.Newbies '`).Scan(&staleIndex); err != nil { + t.Fatal(err) + } + if err := DB.QueryRow(`SELECT count(*) FROM copied WHERE account_id=7 AND folder='INBOX.Newbies ' AND message_id='sha256:legacy-raw'`).Scan(&preservedAlias); err != nil { + t.Fatal(err) + } + if staleIndex != 0 || preservedAlias != 1 { + t.Fatalf("folder alias cleanup stale_index=%d preserved_copy_alias=%d", staleIndex, preservedAlias) + } +} + +func TestHTMLToTextHandlesUnicodeBeforeCaseInsensitiveTags(t *testing.T) { + tests := []struct { + name string + in string + want []string + drop []string + }{ + { + name: "turkish dotted i", + in: "

Turkish İ test


x", + want: []string{"Turkish İ test", "x"}, + }, + { + name: "kelvin sign", + in: "
Kelvin K test

x", + want: []string{"Kelvin K test", "x"}, + }, + { + name: "mixed blocks and repeated replacements", + in: "İ

first

K
second

third", + want: []string{"İ", "first", "K", "second", "third"}, + drop: []string{"bad"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got := htmlToText(tt.in) + for _, want := range tt.want { + if !strings.Contains(got, want) { + t.Fatalf("htmlToText(%q) = %q, missing %q", tt.in, got, want) + } + } + for _, drop := range tt.drop { + if strings.Contains(got, drop) { + t.Fatalf("htmlToText(%q) = %q, unexpectedly contains %q", tt.in, got, drop) + } + } + }) + } +} + +func TestASCIIFoldIndexReturnsOriginalByteOffset(t *testing.T) { + for _, tt := range []struct { + s string + needle string + want int + }{ + {"İİ
x", "
", len("İİ")}, + {"KK", "", len("KK")}, + {"prefix
suffix", "
", len("prefix")}, + } { + if got := asciiFoldIndex(tt.s, tt.needle); got != tt.want { + t.Fatalf("asciiFoldIndex(%q, %q) = %d, want %d", tt.s, tt.needle, got, tt.want) + } + } +} + +func TestMessageBodyPanicFallsBackToRawMessage(t *testing.T) { + raw := []byte("Subject: Evidence\r\n\r\nraw body") + got := renderMessageBodySafely(raw, "test account=archive folder=INBOX seq=7", func() string { + panic("synthetic parser failure") + }) + if !strings.Contains(got, "Darstellung fehlgeschlagen") || !strings.Contains(got, string(raw)) { + t.Fatalf("panic fallback did not preserve raw message: %q", got) + } +} + func testRawMessage(id, subject string) RawMessage { return RawMessage{ MessageID: id, diff --git a/backend/07-migrate.go b/backend/07-migrate.go index d866da5..97ccc59 100644 --- a/backend/07-migrate.go +++ b/backend/07-migrate.go @@ -3,6 +3,7 @@ package backend import ( "fmt" "log" + "os" "path/filepath" "strings" "time" @@ -16,8 +17,10 @@ import ( // Posteingang) landen im vorhandenen Rollen-Ordner des Ziels statt als // Dublette; Pfad-Trenner wird umgehaengt. Dann EnsureFolder + Mails streamen. // 3. Pro Mail: Message-ID gegen copied-Cache pruefen -> schon da? ueberspringen. -// Sonst DOPPELT schreiben: TargetMailbox.Append (IMAP) + MboxWriter.Append -// (lokal), dann MarkCopied. +// Sonst zuerst ins lokale mbox-Sicherheitsarchiv schreiben und diesen +// Zustand festhalten; erst danach folgt TargetMailbox.Append. Beide Stufen +// sind getrennt wiederaufnehmbar, damit ein Zielausfall nie das Archiv +// verhindert und ein Nachlauf keine mbox-Dublette erzeugt. // 4. Fortschritt in jobs-Tabelle schreiben (Browser-Anzeige). // // name = Konto-Name oder "all". watch = true -> Delta-Schleife bis Abbruch @@ -89,11 +92,6 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error { return fmt.Errorf("open source %s: %w", a.Name, err) } defer src.Close() - dst, err := OpenIMAPTarget(a) - if err != nil { - return fmt.Errorf("open target %s: %w", a.Name, err) - } - defer dst.Close() archiveName := accountMboxDir(a) if archiveName == "" { return fmt.Errorf("keine Archiv-Mailbox fuer %s ausgewaehlt", a.Name) @@ -109,86 +107,187 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error { _ = finishJob(jobID, 0, 0, 1, "error") return err } - targets, err := dst.Folders() - if err != nil { - _ = finishJob(jobID, 0, 0, 1, "error") - return err - } folderMap, err := FolderMap(a.ID) if err != nil { _ = finishJob(jobID, 0, 0, 1, "error") return err } + var dst TargetMailbox + var targets []TargetFolder + if opened, openErr := OpenIMAPTarget(a); openErr != nil { + log.Printf("%s target unavailable; continuing archive-only: %v", a.Name, openErr) + } else { + dst = opened + targets, err = dst.Folders() + if err != nil { + log.Printf("%s target folders unavailable; continuing archive-only: %v", a.Name, err) + _ = dst.Close() + dst = nil + targets = nil + } + } + defer func() { + if dst != nil { + _ = dst.Close() + } + }() total, done, errs := 0, 0, 0 + archived, targeted, targetPending := 0, 0, 0 + targetUnavailableLogged := false for _, folder := range folders { if len(selectedFolders) > 0 && !selectedFolders[folder.Name] { continue } - dstFolder := MapSourceToTarget(folder.Name, folder.Attrs, folder.Delim, dst.Delim(), targets, folderMap[folder.Name]) + dstDelim := "/" + if dst != nil { + dstDelim = dst.Delim() + } + dstFolder := MapSourceToTarget(folder.Name, folder.Attrs, folder.Delim, dstDelim, targets, folderMap[folder.Name]) if dstFolder == "" { dstFolder = folder.Name } - if err := dst.EnsureFolder(dstFolder); err != nil { - errs++ - log.Printf("%s %s ensure target %s error: %v", a.Name, folder.Name, dstFolder, err) - continue + targetReady := false + if dst != nil { + if err := dst.EnsureFolder(dstFolder); err != nil { + log.Printf("%s %s ensure target %s error; retrying connection: %v", a.Name, folder.Name, dstFolder, err) + _ = dst.Close() + dst = nil + if reopened, reopenErr := OpenIMAPTarget(a); reopenErr != nil { + log.Printf("%s target reconnect failed; continuing archive-only: %v", a.Name, reopenErr) + } else if ensureErr := reopened.EnsureFolder(dstFolder); ensureErr != nil { + log.Printf("%s %s ensure target %s after reconnect failed; continuing archive-only: %v", a.Name, folder.Name, dstFolder, ensureErr) + _ = reopened.Close() + } else { + dst = reopened + targetReady = true + } + } else { + targetReady = true + } } folderDone, folderErrs := 0, 0 + folderArchived, folderTargeted, folderPending := 0, 0, 0 folderTotal := 0 + identityFolder := safeMboxName(folder.Name) + // After an identity-schema upgrade the indexed byte count is invalidated. + // Rebuild from the actual archive before consulting copied, otherwise a + // legacy Message-ID row could hide a byte-different message. + archivePath := filepath.Join(mboxDir, safeMboxName(folder.Name)+mboxFileExt()) + if _, statErr := os.Stat(archivePath); statErr == nil { + if indexErr := EnsureMboxIndexed(archivePath); indexErr != nil { + errs++ + folderErrs++ + log.Printf("%s %s archive identity index error: %v", a.Name, folder.Name, indexErr) + continue + } + } else if !os.IsNotExist(statErr) { + errs++ + folderErrs++ + log.Printf("%s %s archive stat error: %v", a.Name, folder.Name, statErr) + continue + } if err := src.Fetch(folder.Name, func(m RawMessage) error { total++ folderTotal++ - already, err := AlreadyCopied(a.ID, folder.Name, m.MessageID) + identity := identityForRawMessage(m) + state, err := GetCopyIdentityStateWithFolderAlias(a.ID, identityFolder, folder.Name, identity) if err != nil { errs++ folderErrs++ log.Printf("%s %s dedup error %s: %v", a.Name, folder.Name, m.MessageID, err) return nil } - if already { + if state.MboxDone && state.TargetDone { return nil } - if err := dst.Append(dstFolder, m); err != nil { - errs++ - folderErrs++ - log.Printf("%s %s append to %s error %s: %v", a.Name, folder.Name, dstFolder, m.MessageID, err) - return nil + if !state.MboxDone { + appendInfo, err := mbox.Append(folder.Name, m) + if err != nil { + errs++ + folderErrs++ + log.Printf("%s %s mbox error %s: %v", a.Name, folder.Name, m.MessageID, err) + return nil + } + if err := SaveMboxIndex(MboxIndexEntry{ + AccountID: a.ID, + Folder: identityFolder, + MessageID: identity.MessageID, + BodySHA256: identity.BodySHA256, + Subject: appendInfo.Subject, + From: appendInfo.From, + Date: appendInfo.Date, + FileOffset: appendInfo.FileOffset, + FrameLen: appendInfo.FrameLen, + InnerOffset: appendInfo.InnerOffset, + InnerLen: appendInfo.InnerLen, + }); err != nil { + errs++ + folderErrs++ + log.Printf("%s %s mbox index error %s: %v", a.Name, folder.Name, m.MessageID, err) + return nil + } + if err := UpdateMboxIndexState(a.ID, identityFolder, appendInfo.FileOffset+appendInfo.FrameLen); err != nil { + errs++ + folderErrs++ + log.Printf("%s %s mbox index state error %s: %v", a.Name, folder.Name, m.MessageID, err) + return nil + } + if err := MarkIdentityMboxCopied(a.ID, identityFolder, identity); err != nil { + errs++ + folderErrs++ + log.Printf("%s %s mbox stage mark error %s: %v", a.Name, folder.Name, m.MessageID, err) + return nil + } + state.MboxDone = true + archived++ + folderArchived++ } - appendInfo, err := mbox.Append(folder.Name, m) - if err != nil { - errs++ - folderErrs++ - log.Printf("%s %s mbox error %s: %v", a.Name, folder.Name, m.MessageID, err) - return nil - } - if err := SaveMboxIndex(MboxIndexEntry{ - AccountID: a.ID, - Folder: folder.Name, - MessageID: m.MessageID, - Subject: appendInfo.Subject, - From: appendInfo.From, - Date: appendInfo.Date, - FileOffset: appendInfo.FileOffset, - FrameLen: appendInfo.FrameLen, - InnerOffset: appendInfo.InnerOffset, - InnerLen: appendInfo.InnerLen, - }); err != nil { - errs++ - folderErrs++ - log.Printf("%s %s mbox index error %s: %v", a.Name, folder.Name, m.MessageID, err) - return nil - } - if err := UpdateMboxIndexState(a.ID, folder.Name, appendInfo.FileOffset+appendInfo.FrameLen); err != nil { - errs++ - folderErrs++ - log.Printf("%s %s mbox index state error %s: %v", a.Name, folder.Name, m.MessageID, err) - return nil - } - if err := MarkCopied(a.ID, folder.Name, m.MessageID); err != nil { - errs++ - folderErrs++ - log.Printf("%s %s mark error %s: %v", a.Name, folder.Name, m.MessageID, err) - return nil + if !state.TargetDone { + if dst == nil || !targetReady { + errs++ + folderErrs++ + targetPending++ + folderPending++ + if !targetUnavailableLogged { + log.Printf("%s target unavailable; messages remain safely archived with target_done=0", a.Name) + targetUnavailableLogged = true + } + return nil + } + if err := dst.Append(dstFolder, m); err != nil { + log.Printf("%s %s append to %s error %s; retrying connection: %v", a.Name, folder.Name, dstFolder, m.MessageID, err) + _ = dst.Close() + dst = nil + reopened, reopenErr := OpenIMAPTarget(a) + if reopenErr == nil { + reopenErr = reopened.EnsureFolder(dstFolder) + } + if reopenErr == nil { + reopenErr = reopened.Append(dstFolder, m) + } + if reopenErr != nil { + if reopened != nil { + _ = reopened.Close() + } + errs++ + folderErrs++ + targetPending++ + folderPending++ + targetReady = false + log.Printf("%s target reconnect/append failed; continuing archive-only: %v", a.Name, reopenErr) + return nil + } + dst = reopened + targetReady = true + } + if err := MarkIdentityTargetCopied(a.ID, identityFolder, identity); err != nil { + errs++ + folderErrs++ + log.Printf("%s %s target stage mark error %s: %v", a.Name, folder.Name, m.MessageID, err) + return nil + } + targeted++ + folderTargeted++ } done++ folderDone++ @@ -198,14 +297,16 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error { folderErrs++ log.Printf("%s %s fetch error: %v", a.Name, folder.Name, err) } - log.Printf("migration %s %s -> %s: total=%d copied=%d errors=%d", a.Name, folder.Name, dstFolder, folderTotal, folderDone, folderErrs) + log.Printf("migration %s %s -> %s: total=%d complete=%d archived=%d target=%d pending=%d errors=%d", + a.Name, folder.Name, dstFolder, folderTotal, folderDone, folderArchived, folderTargeted, folderPending, folderErrs) } state := "success" if errs > 0 { state = "partial" } _ = finishJob(jobID, total, done, errs, state) - log.Printf("migration %s all folders: total=%d copied=%d errors=%d", a.Name, total, done, errs) + log.Printf("migration %s all folders: total=%d complete=%d archived=%d target=%d pending=%d errors=%d", + a.Name, total, done, archived, targeted, targetPending, errs) if errs > 0 { return fmt.Errorf("migration %s finished with %d errors", a.Name, errs) } diff --git a/backend/08-viewer.go b/backend/08-viewer.go index e3fd2b4..1228693 100644 --- a/backend/08-viewer.go +++ b/backend/08-viewer.go @@ -62,7 +62,7 @@ func messageHandler(w http.ResponseWriter, r *http.Request) { decodeHeader(msg.Header.Get("Subject")), decodeHeader(msg.Header.Get("From")), decodeHeader(msg.Header.Get("Date")), - messageBody(raw), + messageBodyWithContext(raw, fmt.Sprintf("archive account=%q folder=%q seq=%d", account, folder, index)), ) } @@ -393,7 +393,7 @@ func renderTargetMatch(b *strings.Builder, match forwardedMatch) { decodeHeader(msg.Header.Get("Subject")), decodeHeader(msg.Header.Get("From")), decodeHeader(msg.Header.Get("Date"))+" | "+match.Location(), - messageBody(match.Raw), + messageBodyWithContext(match.Raw, "target "+match.Location()), ) } @@ -487,7 +487,8 @@ func filterEntries(path string, entries []MboxEntry, q string) []MboxEntry { continue } raw, err := ReadMboxMessage(path, entry.Index) - if err == nil && strings.Contains(strings.ToLower(messageBody(raw)), needle) { + if err == nil && strings.Contains(strings.ToLower(messageBodyWithContext(raw, + fmt.Sprintf("archive search path=%q seq=%d", path, entry.Index))), needle) { out = append(out, entry) } } @@ -523,7 +524,7 @@ func findTargetIMAPCopy(account, targetAccount string, raw []byte) (forwardedMat if err != nil { return forwardedMatch{}, false, err } - match, ok, scanErr := scanTargetForMessage(dst, targetUser, id) + match, ok, scanErr := scanTargetForMessage(dst, targetUser, id, bodySHA256(raw)) closeErr := dst.Close() if scanErr != nil { return forwardedMatch{}, false, scanErr @@ -538,7 +539,7 @@ func findTargetIMAPCopy(account, targetAccount string, raw []byte) (forwardedMat return forwardedMatch{}, false, nil } -func scanTargetForMessage(dst TargetMailbox, targetUser, messageID string) (forwardedMatch, bool, error) { +func scanTargetForMessage(dst TargetMailbox, targetUser, messageID, bodyHash string) (forwardedMatch, bool, error) { messageID = normalizeMessageID(messageID) folders, err := dst.Folders() if err != nil { @@ -559,6 +560,9 @@ func scanTargetForMessage(dst TargetMailbox, targetUser, messageID string) (forw if err != nil { continue } + if bodySHA256(msg.Body) != bodyHash { + continue + } return forwardedMatch{ Account: "Ziel: " + targetUser, Folder: folder.Name, @@ -582,6 +586,7 @@ func findForwardedCopy(account, targetAccount string, raw []byte) (forwardedMatc if err != nil { return forwardedMatch{}, false } + bodyHash := bodySHA256(raw) for _, entry := range entries { if !entry.IsDir() || entry.Name() == account { continue @@ -600,7 +605,7 @@ func findForwardedCopy(account, targetAccount string, raw []byte) (forwardedMatc if err != nil { continue } - if messageIDHeader(candidate) == id { + if messageIDHeader(candidate) == id && bodySHA256(candidate) == bodyHash { return forwardedMatch{ Account: entry.Name(), Folder: folderNameFromMboxPath(path), @@ -713,7 +718,7 @@ func appendManualTargetCopy(account, folder, targetAccount string, raw []byte) ( return "", err } if a.ID != 0 { - _ = MarkCopied(a.ID, folder, id) + _ = MarkIdentityCopied(a.ID, folder, identityForRawMessage(msg)) } return "Manuell kopiert nach " + strings.TrimSpace(a.DstUser) + " / " + dstFolder + ".", nil } diff --git a/backend/13-target-dedup.go b/backend/13-target-dedup.go index b1e3999..8790972 100644 --- a/backend/13-target-dedup.go +++ b/backend/13-target-dedup.go @@ -15,21 +15,24 @@ type TargetDedupReport struct { ExtraCopies int Candidates int WithoutID int + VariantGroups int Deleted int FolderReports []TargetDedupFolderReport } type TargetDedupFolderReport struct { - Folder string - Groups int - ExtraCopies int - WithoutID int - Deletes []TargetDedupDelete + Folder string + Groups int + ExtraCopies int + WithoutID int + VariantGroups int + Deletes []TargetDedupDelete } type TargetDedupDelete struct { - UID uint32 - MessageID string + UID uint32 + MessageID string + BodySHA256 string } func DedupTarget(name string, folders []string, apply bool) (TargetDedupReport, error) { @@ -61,13 +64,14 @@ func DedupTarget(name string, folders []string, apply bool) (TargetDedupReport, report.ExtraCopies += folderReport.ExtraCopies report.Candidates += len(folderReport.Deletes) report.WithoutID += folderReport.WithoutID + report.VariantGroups += folderReport.VariantGroups if apply { report.Deleted += len(folderReport.Deletes) } report.FolderReports = append(report.FolderReports, folderReport) } - log.Printf("target dedup %s apply=%v folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d", - report.Account, report.Apply, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID) + log.Printf("target dedup %s apply=%v folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d protected_variant_groups=%d", + report.Account, report.Apply, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID, report.VariantGroups) return report, nil } @@ -91,12 +95,35 @@ func dedupTargetFolder(dst TargetMailbox, folder string, apply bool) (TargetDedu if len(uids) < 2 { continue } - sort.Slice(uids, func(i, j int) bool { return uids[i] < uids[j] }) - report.Groups++ - report.ExtraCopies += len(uids) - 1 - for _, uid := range uids[1:] { - report.Deletes = append(report.Deletes, TargetDedupDelete{UID: uid, MessageID: id}) - deleteUIDs = append(deleteUIDs, uid) + byBody := map[string][]uint32{} + for _, uid := range uids { + msg, err := dst.FetchOne(folder, uid) + if err != nil { + return report, fmt.Errorf("%s fetch uid %d for safe dedup: %w", folder, uid, err) + } + hash := bodySHA256(msg.Body) + byBody[hash] = append(byBody[hash], uid) + } + if len(byBody) > 1 { + report.VariantGroups++ + } + hashes := make([]string, 0, len(byBody)) + for hash := range byBody { + hashes = append(hashes, hash) + } + sort.Strings(hashes) + for _, hash := range hashes { + bodyUIDs := byBody[hash] + if len(bodyUIDs) < 2 { + continue + } + sort.Slice(bodyUIDs, func(i, j int) bool { return bodyUIDs[i] < bodyUIDs[j] }) + report.Groups++ + report.ExtraCopies += len(bodyUIDs) - 1 + for _, uid := range bodyUIDs[1:] { + report.Deletes = append(report.Deletes, TargetDedupDelete{UID: uid, MessageID: id, BodySHA256: hash}) + deleteUIDs = append(deleteUIDs, uid) + } } } sort.Slice(report.Deletes, func(i, j int) bool { @@ -108,14 +135,14 @@ func dedupTargetFolder(dst TargetMailbox, folder string, apply bool) (TargetDedu if apply && len(deleteUIDs) > 0 { sort.Slice(deleteUIDs, func(i, j int) bool { return deleteUIDs[i] < deleteUIDs[j] }) for _, del := range report.Deletes { - log.Printf("target dedup delete folder=%s uid=%d message_id=%s", folder, del.UID, del.MessageID) + log.Printf("target dedup delete folder=%s uid=%d message_id=%s body_sha256=%s", folder, del.UID, del.MessageID, del.BodySHA256) } if err := dst.DeleteUIDs(folder, deleteUIDs); err != nil { return report, fmt.Errorf("%s delete uids: %w", folder, err) } } - if report.Groups > 0 || report.WithoutID > 0 { - log.Printf("target dedup folder=%s groups=%d extra=%d without_id=%d apply=%v", folder, report.Groups, report.ExtraCopies, report.WithoutID, apply) + if report.Groups > 0 || report.WithoutID > 0 || report.VariantGroups > 0 { + log.Printf("target dedup folder=%s groups=%d extra=%d without_id=%d protected_variant_groups=%d apply=%v", folder, report.Groups, report.ExtraCopies, report.WithoutID, report.VariantGroups, apply) } return report, nil } @@ -125,13 +152,13 @@ func LogTargetDedupReport(report TargetDedupReport) { if report.Apply { mode = "APPLY" } - log.Printf("target dedup report %s account=%s folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d", - mode, report.Account, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID) + log.Printf("target dedup report %s account=%s folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d protected_variant_groups=%d", + mode, report.Account, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID, report.VariantGroups) for _, folder := range report.FolderReports { - if folder.Groups == 0 && folder.WithoutID == 0 { + if folder.Groups == 0 && folder.WithoutID == 0 && folder.VariantGroups == 0 { continue } - log.Printf("target dedup report folder=%s groups=%d extra=%d without_id=%d", folder.Folder, folder.Groups, folder.ExtraCopies, folder.WithoutID) + log.Printf("target dedup report folder=%s groups=%d extra=%d without_id=%d protected_variant_groups=%d", folder.Folder, folder.Groups, folder.ExtraCopies, folder.WithoutID, folder.VariantGroups) for _, del := range folder.Deletes { log.Printf("target dedup report candidate folder=%s uid=%d message_id=%s", folder.Folder, del.UID, del.MessageID) } diff --git a/backend/13-target-dedup_test.go b/backend/13-target-dedup_test.go new file mode 100644 index 0000000..d7ed764 --- /dev/null +++ b/backend/13-target-dedup_test.go @@ -0,0 +1,70 @@ +package backend + +import ( + "fmt" + "reflect" + "testing" +) + +type dedupTargetStub struct { + headers []MessageHeader + bodies map[uint32][]byte + deleted []uint32 +} + +func (s *dedupTargetStub) Folders() ([]TargetFolder, error) { return nil, nil } +func (s *dedupTargetStub) Delim() string { return "/" } +func (s *dedupTargetStub) EnsureFolder(string) error { return nil } +func (s *dedupTargetStub) Append(string, RawMessage) error { return nil } +func (s *dedupTargetStub) Headers(string, int, int) ([]MessageHeader, error) { + return s.headers, nil +} +func (s *dedupTargetStub) AllHeaders(string) ([]MessageHeader, error) { return s.headers, nil } +func (s *dedupTargetStub) FetchOne(_ string, uid uint32) (RawMessage, error) { + body, ok := s.bodies[uid] + if !ok { + return RawMessage{}, fmt.Errorf("missing uid %d", uid) + } + return RawMessage{Body: body}, nil +} +func (s *dedupTargetStub) DeleteUIDs(_ string, uids []uint32) error { + s.deleted = append(s.deleted, uids...) + return nil +} +func (s *dedupTargetStub) Close() error { return nil } + +func TestDedupTargetFolderProtectsDifferentBodiesWithSameMessageID(t *testing.T) { + sameA := []byte("Message-ID: \r\n\r\nA") + sameB := []byte("Message-ID: \r\n\r\nB") + dst := &dedupTargetStub{ + headers: []MessageHeader{ + {UID: 1, MessageID: "same@example.com"}, + {UID: 2, MessageID: "same@example.com"}, + {UID: 3, MessageID: "same@example.com"}, + }, + bodies: map[uint32][]byte{1: sameA, 2: sameA, 3: sameB}, + } + report, err := dedupTargetFolder(dst, "INBOX", true) + if err != nil { + t.Fatal(err) + } + if report.Groups != 1 || report.ExtraCopies != 1 || report.VariantGroups != 1 { + t.Fatalf("unexpected report: %#v", report) + } + if !reflect.DeepEqual(dst.deleted, []uint32{2}) { + t.Fatalf("deleted=%v, want only exact duplicate UID 2", dst.deleted) + } +} + +func TestDedupTargetFolderAbortsBeforeDeleteWhenBodyCannotBeVerified(t *testing.T) { + dst := &dedupTargetStub{ + headers: []MessageHeader{{UID: 1, MessageID: "same@example.com"}, {UID: 2, MessageID: "same@example.com"}}, + bodies: map[uint32][]byte{1: []byte("Message-ID: \r\n\r\nA")}, + } + if _, err := dedupTargetFolder(dst, "INBOX", true); err == nil { + t.Fatal("expected safe dedup to fail when a body cannot be fetched") + } + if len(dst.deleted) != 0 { + t.Fatalf("deleted despite incomplete verification: %v", dst.deleted) + } +} diff --git a/bug-hashkey-instabil.md b/bug-hashkey-instabil.md new file mode 100644 index 0000000..7ce96ff --- /dev/null +++ b/bug-hashkey-instabil.md @@ -0,0 +1,84 @@ +# 🟠 Instabiler Ersatzschlüssel: 4.024 Doppel-Einträge in `copied` + +**Kein Datenverlust, keine Dubletten in Postfächern.** Aber die Buchhaltung +stimmt nicht — und beim Aufräumen lauert eine Falle (siehe „⚠️ NICHT TUN"). + +## Befund + +``` +Datei-Records: 58.051 ═ mbox_index: 58.051 ✓ (Drift 0) +copied: 62.073 ← 4.022 mehr +``` +`copied`-Zeilen ohne passenden `mbox_index`-Eintrag: **4.024**, davon +**4.021 mit `sha256:`-Schlüssel**. Verteilung exakt auf die Konten mit +`--reindex --rebuild`: gb +2533, rb +719, byrne +564, crystal +112, roby +92. +(gb hatte gemessen 2.537 Mails ohne Message-ID — passt auf ~4 genau.) + +## Ursache + +Mails **ohne `Message-ID`-Header** bekommen `sha256:` als Ersatzschlüssel. +Der Hash wird aber über **unterschiedliche Bytes** gebildet: + +| Pfad | hasht | +|---|---| +| Migration (`07-migrate.go`) | die **IMAP-Rohbytes** der Mail | +| Reindex/Rebuild (`06-mbox.go`) | den **mbox-Record aus der Datei** | + +Die mbox-Fassung ist nicht byte-gleich mit der IMAP-Fassung: Der Writer setzt +`>From `-Quoting und normalisiert Zeilenenden. → **anderer Hash → anderer +Schlüssel → dieselbe Mail steht zweimal in `copied`.** + +Betroffen sind alle **6.623** Mails ohne Message-ID (so viele hat `mbox_index` +mit `sha256:`-Schlüssel). + +## ⚠️ NICHT TUN: die „Waisen" einfach löschen + +Die 4.024 Waisen sind **nicht** der Müll — sie sind die Zeilen, auf die der +**Migrationspfad tatsächlich matcht** (er rechnet den IMAP-Hash aus). Die vom +Rebuild ergänzten Zeilen (mbox-Hash) sind die redundanten. + +**Werden die Waisen gelöscht, hält die Migration diese 4.021 Mails für +unkopiert und schreibt sie erneut ins Ziel-Postfach → echte Dubletten in +fremden Postfächern.** Genau der Schaden, den wir vermeiden wollen. + +Aus demselben Grund darf die **Schlüsselberechnung nicht einfach umgestellt** +werden: Für Konten **ohne** Rebuild (kolmer, palamari, colak, …) existiert nur +der IMAP-Hash. Ein Wechsel auf den mbox-Hash lässt dort **alle** Message-ID-losen +Mails als „neu" erscheinen → Massen-Neukopie. + +## Fix (in dieser Reihenfolge, sonst knallt es) + +1. **Schlüssel vereinheitlichen:** Der Ersatzschlüssel muss in **beiden** Pfaden + über **dieselben** Bytes gehen. Sauberste Wahl: über die **mbox-normalisierte + Fassung** (also das, was tatsächlich im Archiv steht) — dann liefern + Migration und Reindex/Rebuild dauerhaft denselben Wert, auch nach jedem + künftigen Rebuild. + → In `07-migrate.go` den Hash **nach** der mbox-Normalisierung bilden, nicht + über die IMAP-Rohbytes. +2. **Einmalige `copied`-Migration** (zwingend zusammen mit 1., nicht danach): + Für jedes Konto/Ordner `copied` aus `mbox_index` neu ableiten + (`message_id` aus dem Index, `mbox_done=1`), `target_done` aus dem + bestehenden Stand übernehmen (die Mails **sind** im Ziel — nicht auf 0 + zurücksetzen!). Anschließend die alten IMAP-Hash-Zeilen entfernen. + **In einer Transaktion**, mit Zählung vorher/nachher. +3. **Abnahme:** `copied` == `mbox_index` (58.051 == 58.051), und der erste + Watch-/Migrationslauf danach meldet **`archived=0 target=0 pending=0`** — + also keine einzige Neukopie. Meldet er mehr, sofort stoppen: dann matcht der + Schlüssel nicht und es entstehen gerade Dubletten. + +## Alternativ (wenn 1.+2. zu heikel erscheinen) +Zustand **so lassen**. Er ist funktional korrekt: Der Migrationspfad matcht auf +die IMAP-Hash-Zeilen, es wird nichts doppelt kopiert (live bestätigt: +`complete=0 archived=0 target=0 pending=0 errors=0`). Kosten: `copied` ist um +~7 % aufgebläht und die Zahl taugt nicht als Beleg für „so viel ist archiviert". +Dann aber **in `INSTALL.md` dokumentieren**, warum die Zahlen auseinandergehen — +sonst rätselt in zwei Jahren jemand daran. + +## Hängt zusammen mit +Der noch offenen `body_sha256`-Bereinigung (byte-verschiedene Mails mit gleicher +Message-ID, ~109 Nachzügler): Beide drehen sich um „was ist die Identität einer +Mail". Am besten **in einem Zug** lösen und **eine** Schlüsselregel festlegen: +- Message-ID vorhanden → `message_id` **+ `body_sha256`** (fängt die + byte-verschiedenen Fälle), +- keine Message-ID → nur `body_sha256`, +- **`body_sha256` immer über die mbox-normalisierte Fassung.** diff --git a/bug-htmltotext-panic.md b/bug-htmltotext-panic.md new file mode 100644 index 0000000..4f9055f --- /dev/null +++ b/bug-htmltotext-panic.md @@ -0,0 +1,106 @@ +# 🔴 Vorschau stürzt bei ~0,1 % der Mails ab (htmlToText, Unicode) + +Gefunden beim Verifizieren des Index-Drift-Fixes. **Der Drift-Fix selbst ist +sauber** (2000 Stichproben über 8 Ordner, 0 falsche Zuordnungen, Drift 0 über +alle 69 Ordner) — das hier ist ein **separater, neuer Befund**. + +## Symptom + +``` +2026/07/16 09:45:44 http: panic serving: runtime error: + slice bounds out of range [:207] with length 175 + backend.replaceCaseInsensitive 06-mbox.go:770 + backend.markHTMLBreaks 06-mbox.go:755 + backend.htmlToText 06-mbox.go:690 + backend.extractTextBody / messageBody / messageHandler +``` +Die betroffene Mail ist **nicht anzeigbar** — die Verbindung stirbt. Gemessen: +**1 von 2000 Stichproben (~0,1 %)**, hochgerechnet ~58 der 57.771 Mails. +Beispiel: `kolmer/INBOX seq=2378 "[www.mtp24.pl] Weryfikacja adresu"` — eine +polnische Mail. Auslöser ist Nicht-ASCII. + +## Ursache — dieselbe Klasse wie der Buffer-Bug: zwei Koordinatensysteme + +[`06-mbox.go:760`](backend/06-mbox.go): +```go +func replaceCaseInsensitive(s, old, new string) string { + lower := strings.ToLower(s) // <-- kann ANDERE Byte-Laenge haben als s! + needle := strings.ToLower(old) + for { + i := strings.Index(lower, needle) // Index in LOWER + ... + b.WriteString(s[:i]) // auf S angewendet <-- FALSCH + cut := i + len(old) + s = s[cut:] + lower = lower[cut:] // beide mit demselben cut -> driften auseinander + } +} +``` + +`strings.ToLower` ist **nicht byte-laengen-erhaltend**: +- `İ` (U+0130, 2 Bytes) → `i̇` (3 Bytes) = **+1** +- `K` (U+212A Kelvin, 3 Bytes) → `k` (1 Byte) = **−2** + +Sobald ein solches Zeichen vorkommt, passen die Indizes aus `lower` nicht mehr +auf `s`. Mit jedem `cut` driftet es weiter, bis `i > len(s)` → Panic. Das erklaert +`[:207] with length 175` exakt. + +**`stripHTMLBlock` direkt darunter hat denselben Fehler** (`lower`-Index auf `s` +angewendet) — bitte mitfixen, auch wenn er noch nicht gecrasht ist. + +## Fix + +Die Needles sind **alle ASCII** (`
`, `

`, ``, …). Also gar keine +lowercase-Kopie bauen, sondern direkt auf dem Original suchen — mit +ASCII-Case-Folding: + +```go +// asciiFoldIndex sucht needle (ASCII, bereits lowercase) case-insensitiv in s +// und liefert einen Index, der IMMER fuer s gueltig ist. +func asciiFoldIndex(s, needle string) int { + n := len(needle) + if n == 0 || len(s) < n { return -1 } + for i := 0; i+n <= len(s); i++ { + ok := true + for j := 0; j < n; j++ { + c := s[i+j] + if 'A' <= c && c <= 'Z' { c += 'a' - 'A' } + if c != needle[j] { ok = false; break } + } + if ok { return i } + } + return -1 +} +``` +Damit entfaellt das zweite Koordinatensystem komplett — es gibt nur noch `s`. +(Alternative: vorkompilierte `regexp.MustCompile("(?i)"+regexp.QuoteMeta(old))` +— ebenfalls korrekt, aber langsamer und schwerer als noetig.) + +## Zusaetzlich: `recover()` im Vorschau-Pfad — Pflicht für ein Beweis-Archiv + +`htmlToText` verarbeitet **beliebiges, fremdes HTML** aus 20 Jahren Mail +(Spam, kaputte Mailer, alle Sprachen). Ein Parser-Bug darf **nie** dazu fuehren, +dass eine archivierte Mail unsichtbar wird. + +In `messageHandler` (bzw. um `messageBody`) ein `defer recover()`: +- bei Panic → **nicht** die Verbindung sterben lassen, +- stattdessen den **Rohtext** der Mail anzeigen (bzw. Header + Hinweis + „Darstellung fehlgeschlagen, Rohfassung unten"), +- und den Vorfall loggen (Konto/Ordner/seq), damit wir solche Mails finden. + +**Grundsatz:** Man muss jede archivierte Mail *irgendwie* ansehen koennen — +notfalls roh. Lieber haesslich als unsichtbar. + +## Abnahme + +1. Unit-Test, der den Bug faengt (mit ASCII-only Test wuerde er durchrutschen!): + ```go + in := "

Turkish İ test


x" // U+0130 aendert die Byte-Laenge + out := htmlToText(in) // darf NICHT panicken + ``` + Zusaetzlich ein Fall mit `K` (U+212A) und ein gemischter (mehrere Vorkommen, + damit die Drift ueber mehrere Schleifendurchlaeufe sichtbar wuerde). +2. `go test ./...` gruen. +3. Live: `kolmer/INBOX seq=2378` (`[www.mtp24.pl] Weryfikacja adresu`) muss sich + oeffnen lassen — heute stirbt die Verbindung. +4. Ich (Claude) fahre danach den Sweep erneut: **0 Abstuerze** in 2000+ Proben. diff --git a/bug-index-drift.md b/bug-index-drift.md new file mode 100644 index 0000000..62cacd2 --- /dev/null +++ b/bug-index-drift.md @@ -0,0 +1,88 @@ +# 🔴 KRITISCH — Vorschau/Liste um 1 verschoben (Index-Drift durch SQLITE_BUSY) + +**Der Reader-Aliasing-Fix (6c4073a) war richtig, aber es gibt einen ZWEITEN, +unabhaengigen Bug.** Klick auf „iTagPro" zeigt „Orivelle Pen" (die Nachbarzeile). + +## Root Cause — bewiesen, nicht vermutet + +**Zwei Fehler, die sich verketten:** + +1. **6 Mails sind in der mbox-DATEI, aber nicht im `mbox_index`.** + Ursache: `PRAGMA busy_timeout` steht immer noch auf **0** (der + `rettungslauf-fixes.md`-Brief ist nicht umgesetzt). Beim Rettungslauf gab es + pro betroffenem Ordner `SQLITE_BUSY`: `mbox.Append` schrieb den Record in die + Datei, der darauffolgende **Index-INSERT scheiterte sofort**. Ergebnis: + Datei-Record vorhanden, Index-Zeile fehlt. + +2. **Der plain-Reader liest nach DATEI-POSITION statt nach `file_offset`.** + [`06-mbox.go` `readMboxMessageFromIndex`](backend/06-mbox.go): holt zwar die + Index-Zeile (mit korrektem `file_offset`), wirft sie fuer plain aber weg + (`if !strings.HasSuffix(path, ".zst") { return nil, false, nil }`) und faellt + auf `msgs[index]` = Datei-Position zurueck. Ab der fehlenden Index-Zeile ist + Position != seq → alles um 1 verschoben. + +**Der Beweis (drift-scan ueber alle 66 Ordner):** +``` +byrne/INBOX Datei 5869 Index 5868 +1 (Rettungslauf errors=1) +crystal/INBOX Datei 4874 Index 4873 +1 (errors=2 -> +crystal/INBOX.Sent Datei 2146 Index 2145 +1 genau 2 Ordner) +gb/INBOX Datei 6675 Index 6674 +1 (errors=1) +rb/INBOX Datei 8628 Index 8627 +1 (errors=1) +roby/INBOX Datei 2718 Index 2717 +1 (errors=1) +``` +**Drift-Anzahl == SQLITE_BUSY-Fehleranzahl, exakt.** Die 60 fehlerfreien Ordner +haben 0 Drift. Und: Lesen ueber `file_offset` liefert fuer gb **4129/4129** +Mails korrekt (0 echte Fehler) — die gespeicherten Offsets sind die Wahrheit. + +## Fix (drei Teile, in dieser Reihenfolge) + +### A. `busy_timeout` setzen — PFLICHT, sonst passiert es wieder +`ConnectDB` (02-database.go): `PRAGMA busy_timeout = 10000;` neben `journal_mode=WAL`. + +### B. Plain-mbox ueber `file_offset` lesen (der eigentliche Fix) +`readMboxMessageFromIndex` **darf fuer plain nicht mehr auf die Positions- +Zerlegung zurueckfallen.** Analog zum zstd-Pfad: +- Index-Zeile holen → `file_offset` seeken → Record bis zum `file_offset` der + **naechsten** Zeile (bzw. EOF) lesen, `TrimRight("\n")`. +- Die fragile Neu-Zerlegung `readMboxMessages`+`msgs[index]` **ganz aus dem + Vorschau-/Export-/Copy-Pfad entfernen.** Zwei getrennte Zerleger (Writer- + Offsets vs. Reader-Resplit), die auseinanderlaufen koennen, sind die Wurzel. +- Damit lesen Liste UND Vorschau aus **derselben** Quelle (`mbox_index`) → sie + koennen nicht mehr auseinanderlaufen. + +### C. Die 6 verlorenen Index-Zeilen nachziehen — VOLLER Reindex aus der Datei +Das bestehende `--reindex` haengt nur ab `indexed_bytes` an — es findet die +**mittendrin** fehlenden Mails nicht. Es braucht einen **Full-Rebuild**: +- `--reindex --rebuild `: `mbox_index` fuer den Ordner **loeschen** + und aus der Datei **komplett neu** aufbauen (jeder Record → seq, file_offset, + message_id, subject, date). Danach ist Index-Anzahl == Datei-Records. +- **Wichtig — `copied` mitziehen:** Diese 6 Mails stehen auch **nicht** in + `copied` (der Busy-Fehler hat auch `MarkCopied` verhindert). Ein Watch-Lauf + wuerde sie sonst erneut ins Ziel-Postfach **und** erneut in die mbox kopieren + (Dublette in beiden!). Beim Rebuild deshalb pro Record die Message-ID (+ ggf. + `body_sha256`) in `copied` mit `mbox_done=1` eintragen. + +## Sekundaerbefunde (mitnehmen, kein Blocker) +- **Ungueltiges UTF-8 in `mbox_index.subject`** (z.B. eine Tesla-Newsletter-Mail + mit kaputtem Byte) — bricht sogar `sqlite3`-Abfragen. Subject vor dem Speichern + nach UTF-8 sanitisieren (ungueltige Bytes ersetzen). +- **Envelope-Message-ID vs. Roh-Header:** In ~8 gb-Faellen steht im Index ein + `sha256:`-Fallback, obwohl die Datei einen echten `Message-ID`-Header hat. + Beim Indizieren zuerst den Roh-Header parsen, bevor auf Hash zurueckgefallen + wird — sonst greift Dedup fuer diese Mails nicht. + +## Abnahme +1. `go test ./...` gruen; `PRAGMA busy_timeout` = 10000. +2. Test, der die Drift faengt: mbox mit N Records schreiben, eine Index-Zeile + **in der Mitte** loeschen, dann Vorschau von seq > Luecke abrufen → muss die + **richtige** Mail liefern (heute die Nachbarmail). +3. `--reindex --rebuild all`, danach **drift-scan == 0** fuer ALLE 66 Ordner + (Datei-Records == Index == Listen-Eintraege). +4. Watch-Lauf nach dem Rebuild: `copied=0` fuer alle Konten (keine Neukopie der + nachgezogenen Mails). +5. Voller Sweep (Betreff **und** Datum) ueber gb/INBOX und crystal/INBOX: + **0 Abweichungen**. + +Ich (Claude) fahre nach dem Fix den drift-scan ueber alle Ordner UND einen +Voll-Sweep gegen `mbox_index` — diesmal ueber ALLE betroffenen Postfaecher, +nicht nur eines. diff --git a/rettungslauf-fixes.md b/rettungslauf-fixes.md new file mode 100644 index 0000000..e79f20f --- /dev/null +++ b/rettungslauf-fixes.md @@ -0,0 +1,137 @@ +# Codex-Brief — Fixes aus dem echten Rettungslauf (14.07., 15 GB / 58k Mails) + +Der Lauf gegen die 18 echten `@dr-gold.de`-Postfaecher hat vier Schwaechen +freigelegt, die im Testbetrieb nie sichtbar waren. **Fix 1 und 2 sind Pflicht, +bevor wir den Nachschlag fahren** — sonst reisst es an derselben Stelle wieder. + +Kontext: Der Quell-Hoster verliert derzeit Mails. Das lokale mbox-Archiv ist das +Sicherheitsnetz. **Alles, was das Archiv gefaehrdet, ist ein Notfall.** + +--- + +## 1. 🔴 `busy_timeout = 0` — SQLite scheitert sofort statt zu warten + +**Befund:** `PRAGMA busy_timeout` steht auf **0**. Bei Lock-Konflikten (Watcher + +Web-App + Migration auf derselben DB) bricht die Abfrage **sofort** ab: +``` +mbox index error : database is locked (5) (SQLITE_BUSY) +``` +Sechsmal im Rettungslauf passiert. + +**Fix:** In `ConnectDB` (02-database.go) neben `journal_mode=WAL` setzen: +```sql +PRAGMA busy_timeout = 10000; -- 10s warten statt sofort scheitern +``` +Einzeiler, grosse Wirkung. + +--- + +## 2. 🔴 Das Archiv haengt am Ziel — und das ist verkehrt herum + +**Der schwerste Befund.** Heutige Reihenfolge in `migrateAccount`: +``` +AlreadyCopied -> dst.Append (All-Inkl) -> mbox.Append (lokal) -> MarkCopied + ^^^^^^^^^^ scheitert das hier, wird per `return nil` abgebrochen + -> die Mail landet AUCH NICHT im lokalen Archiv +``` + +**Real passiert bei `petarus`:** Die Ziel-Verbindung starb mitten im Lauf +(`use of closed network connection`). Danach schlug jedes `EnsureFolder`/`Append` +fehl → **~620 von 1135 Mails wurden weder ins Ziel noch ins Archiv geschrieben.** +Das Sicherheitsnetz hat genau dann gerissen, als es gebraucht wurde. + +**Das Archiv muss zuerst kommen und vom Ziel unabhaengig sein.** Dafuer braucht +`copied` getrennte Zustaende — sonst erzeugt ein Retry mbox-Dubletten: + +```sql +ALTER TABLE copied ADD COLUMN mbox_done INTEGER NOT NULL DEFAULT 1; +ALTER TABLE copied ADD COLUMN target_done INTEGER NOT NULL DEFAULT 1; +-- Default 1: Bestandszeilen sind beides-fertig, das stimmt. +``` + +**Neuer Ablauf:** +``` +1. Zeile vorhanden und mbox_done=1 und target_done=1 -> ueberspringen +2. mbox.Append (ZUERST — das Archiv ist das Sicherheitsnetz) + scheitert -> Fehler zaehlen, Mail ueberspringen, nichts markieren +3. Zeile schreiben/aktualisieren: mbox_done=1, target_done=0 +4. dst.Append (Ziel) + OK -> target_done=1 + scheitert -> Fehler zaehlen, WEITERMACHEN. Die Mail IST archiviert. +``` + +**Was das loest:** +- Ein sterbendes Ziel kostet nie wieder eine Archiv-Kopie. +- Ein zweiter Lauf sieht `mbox_done=1, target_done=0` → **ueberspringt den + mbox-Append** (keine Dublette!) und **holt nur das Ziel nach**. + +`MarkCopied` bleibt sinngemaess das Letzte je Stufe. Ein Absturz zwischen +mbox-Write und Zeilen-Update kopiert die Mail beim naechsten Lauf erneut ins +Archiv — dieselbe bewusste Entscheidung wie bisher, nur jetzt pro Stufe. + +--- + +## 3. 🟠 Kein Reconnect zum Ziel + +Bei einem 2-GB-Postfach haelt All-Inkl die IMAP-Sitzung nicht durch. Stirbt sie, +scheitert **jede** weitere Operation — der Rest des Kontos faellt weg. + +**Fix:** Verbindungsfehler erkennen (`use of closed network connection`, EOF, +`broken pipe`) und **die Ziel-Verbindung neu aufbauen** (1–2 Versuche mit kurzer +Pause). Gelingt das nicht: +- **nicht abbrechen**, sondern fuer den Rest des Kontos im **Archiv-Only-Modus** + weiterlaufen (dank Fix 2 ist das gefahrlos — die Mails landen sicher im mbox, + `target_done=0`), und +- am Ende **laut melden**: `ZIEL NICHT ERREICHBAR — N Mails nur archiviert, + Nachlauf noetig`. + +Lieber ein vollstaendiges Archiv mit offenem Ziel-Nachlauf als beides halb. + +--- + +## 4. 🟠 Gleiche Message-ID im selben Ordner → Mail wird still verworfen + +**Befund:** Bei mehreren Konten ist `copied < total` **ohne** Fehler +(neubauer 1, simsek 5, palamari 8, gb u. a. — zusammen ~25 Mails). Ursache: Die +gleiche Message-ID kommt im selben Ordner zweimal vor (Re-Import, kaputte Mailer), +und der Dedup wirft die zweite weg. + +Fuer ein **Beweis-Archiv ist das nicht hinnehmbar**: Zwei Mails mit derselben +Message-ID koennen **unterschiedlichen Inhalt** haben. Wir duerfen nie still eine +byte-verschiedene Mail verwerfen. + +**Fix:** `copied` um `body_sha256` erweitern, Unique-Key auf +`(account_id, folder, message_id, body_sha256)`. +- Gleiche ID **und** gleiche Bytes → echte Dublette, ueberspringen (wie bisher). +- Gleiche ID, **andere** Bytes → **andere Mail**, archivieren. +- Bestandszeilen: `body_sha256` leer lassen und beim Vergleich als Wildcard + behandeln (kein Reindex-Zwang, keine Re-Kopie des Bestands). + +--- + +## 5. 🟢 Phantom-Ordner tolerieren + +`LIST` meldet `INBOX.Posteingang`, `SELECT` sagt `NO Mailbox doesn't exist`. +Kaputte Quelle (passt zum Hoster-Problem). Heute: Fehler. Kuenftig: **Warnung**, +Ordner ueberspringen, Lauf sauber weiterfuehren — das ist kein Fehler unseres +Tools. + +--- + +## Abnahme + +1. `go test ./...` gruen. +2. `PRAGMA busy_timeout` → 10000. +3. **Der entscheidende Test — Ziel-Ausfall simulieren:** Migration eines + Testkontos starten, waehrend des Laufs das Ziel unerreichbar machen + (z. B. falscher `dst_host`/Port). Erwartung: + - **Alle** Quell-Mails landen trotzdem **vollstaendig im lokalen mbox**. + - `target_done=0` fuer diese Mails. + - Lauf meldet laut „Ziel nicht erreichbar, Nachlauf noetig". + - **Zweiter Lauf mit erreichbarem Ziel:** kopiert **nichts** ins mbox + (keine Dublette!) und holt **nur** das Ziel nach. + Das ist die Regression, die uns bei `petarus` ~620 Mails gekostet haette. +4. Nachschlag `petarus` (`--folders` ordnerweise) → alle 1135 Mails im Archiv. + +Ich (Claude) fahre danach den Verifikationslauf: Quell-Anzahl vs. Archiv-Anzahl +je Konto, und ziehe alle Luecken nach.