Stabilize mail identity: canonical hash, index drift, HTML panic
Behebt drei Fehler derselben Klasse: an jeder Stelle wurde derselbe Wert zweimal berechnet, statt einmal berechnet und weitergereicht - und die beiden Berechnungen liefen auseinander. 1. Instabiler Ersatzschluessel (bug-hashkey-instabil.md) Mails ohne Message-ID bekamen sha256 ueber die IMAP-Rohbytes, der Reindex hashte dieselbe Mail ueber die mbox-gespeicherten Bytes (>From-Quoting, andere Zeilenenden) -> zwei Schluessel, 4.024 Doppel-Eintraege in copied. Fix: canonicalMessageBytes() bringt beide Seiten auf eine Form (>From zurueckdrehen, CRLF->LF, Trailing-Newlines weg). Alle Pfade (Index, Migration, Viewer, Dedup) nutzen dieselbe Funktion. body_sha256 in copied + mbox_index, UNIQUE erweitert. 2. Index-Drift (bug-index-drift.md) Der plain-mbox-Reader las nach Datei-Position statt nach dem gespeicherten file_offset. Weil SQLITE_BUSY (busy_timeout=0) je Ordner einen Index-Eintrag verschluckt hatte, war ab dieser Luecke alles um 1 verschoben: Klick auf Mail X zeigte Mail X+1. Betroffen genau die 6 Ordner mit Busy-Fehler beim Rettungslauf. Fix: busy_timeout=10000, Lesen ueber file_offset, --reindex --rebuild. 3. HTML-Vorschau-Panic (bug-htmltotext-panic.md) replaceCaseInsensitive/stripHTMLBlock indizierten mit Offsets aus strings.ToLower(s) in s - ToLower ist nicht byte-laengen-erhaltend (z.B. U+0130). ~0,1% der Mails brachten die Vorschau zum Absturz. Fix: asciiFoldIndex() sucht direkt auf den Original-Bytes; zusaetzlich Rohtext-Fallback, damit keine archivierte Mail unsichtbar wird. Weiter: Archiv-zuerst-Reihenfolge (mbox_done/target_done) - ein sterbendes Ziel kostet keine Archiv-Kopie mehr; Dedup vergleicht zusaetzlich den Inhalt und schuetzt byte-verschiedene Varianten. WICHTIG: Die 62.073 alten Alias-Zeilen in copied bleiben bewusst erhalten. Sie sehen wie Muell aus, sind aber die Zeilen, auf die der alte Schluessel matcht - ein Loeschen wuerde Mails erneut in fremde Postfaecher kopieren. Verifiziert: go test ./... gruen; 58.051 Archivmails, 0 ohne copied-Zeile; Drift 0 ueber alle 69 Ordner; 2.000 HTTP-Stichproben ueber 8 Ordner: 0 falsche Zuordnung, 0 Panics; Watcher 220 + 60 Mails archived=0 target=0 errors=0. Live als Image fix-identity2-20260716. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
DonVoo
parent
6c4073a01c
commit
9f5cc59af7
15 changed files with 1650 additions and 181 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -6,8 +6,10 @@ config.json
|
|||
|
||||
# Lokales Cold Backup (mbox) nicht versionieren
|
||||
backup/
|
||||
# Gebaute Binaries (auch datierte Build-Varianten wie mail-graveyard-fix-identity-20260716)
|
||||
/mail-graveyard
|
||||
/mail-graveyard.exe
|
||||
/mail-graveyard-*
|
||||
|
||||
# Bun / Frontend
|
||||
frontend-js/node_modules/
|
||||
|
|
|
|||
|
|
@ -1305,7 +1305,7 @@ func renderTransferPreview(kind, value, folder string, index int, uid uint32) st
|
|||
decodeHeader(msg.Header.Get("Subject")),
|
||||
decodeHeader(msg.Header.Get("From")),
|
||||
decodeHeader(msg.Header.Get("Date")),
|
||||
messageBody(raw),
|
||||
messageBodyWithContext(raw, fmt.Sprintf("transfer kind=%q value=%q folder=%q index=%d uid=%d", kind, value, folder, index, uid)),
|
||||
)
|
||||
return b.String()
|
||||
}
|
||||
|
|
@ -1621,7 +1621,7 @@ func renderSourceMailboxReadMessage(account Account, folder string, raw RawMessa
|
|||
decodeHeader(msg.Header.Get("Subject")),
|
||||
decodeHeader(msg.Header.Get("From")),
|
||||
decodeHeader(msg.Header.Get("Date")),
|
||||
messageBody(raw.Body),
|
||||
messageBodyWithContext(raw.Body, fmt.Sprintf("source account=%q folder=%q", account.Name, folder)),
|
||||
)
|
||||
return b.String()
|
||||
}
|
||||
|
|
@ -1809,7 +1809,7 @@ func renderTargetMailboxReadMessage(account Account, folder string, raw RawMessa
|
|||
decodeHeader(msg.Header.Get("Subject")),
|
||||
decodeHeader(msg.Header.Get("From")),
|
||||
decodeHeader(msg.Header.Get("Date")),
|
||||
messageBody(raw.Body),
|
||||
messageBodyWithContext(raw.Body, fmt.Sprintf("target account=%q folder=%q", account.Name, folder)),
|
||||
)
|
||||
return b.String()
|
||||
}
|
||||
|
|
|
|||
|
|
@ -75,7 +75,13 @@ func ConnectDB(initDB bool) error {
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := db.Exec(`PRAGMA journal_mode=WAL; PRAGMA foreign_keys=ON;`); err != nil {
|
||||
// SQLite pragmas are connection-local. Keep a single process-local
|
||||
// connection so every query uses the busy timeout configured here. The web
|
||||
// app and watcher are separate processes; WAL + busy_timeout coordinates
|
||||
// their writes without failing immediately with SQLITE_BUSY.
|
||||
db.SetMaxOpenConns(1)
|
||||
db.SetMaxIdleConns(1)
|
||||
if _, err := db.Exec(`PRAGMA busy_timeout=10000; PRAGMA journal_mode=WAL; PRAGMA foreign_keys=ON;`); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
|
|
@ -115,14 +121,18 @@ func ConnectDB(initDB bool) error {
|
|||
account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||
folder TEXT NOT NULL,
|
||||
message_id TEXT NOT NULL,
|
||||
body_sha256 TEXT NOT NULL DEFAULT '',
|
||||
mbox_done INTEGER NOT NULL DEFAULT 0,
|
||||
target_done INTEGER NOT NULL DEFAULT 0,
|
||||
copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(account_id, folder, message_id)
|
||||
UNIQUE(account_id, folder, message_id, body_sha256)
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS mbox_index(
|
||||
account_id INTEGER NOT NULL,
|
||||
folder TEXT NOT NULL,
|
||||
seq INTEGER NOT NULL,
|
||||
message_id TEXT NOT NULL,
|
||||
body_sha256 TEXT NOT NULL DEFAULT '',
|
||||
subject TEXT NOT NULL DEFAULT '',
|
||||
from_addr TEXT NOT NULL DEFAULT '',
|
||||
date TEXT NOT NULL DEFAULT '',
|
||||
|
|
@ -130,7 +140,7 @@ func ConnectDB(initDB bool) error {
|
|||
frame_len INTEGER NOT NULL,
|
||||
inner_offset INTEGER NOT NULL DEFAULT 0,
|
||||
inner_len INTEGER NOT NULL,
|
||||
UNIQUE(account_id, folder, message_id)
|
||||
UNIQUE(account_id, folder, seq)
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS mbox_index_state(
|
||||
account_id INTEGER NOT NULL,
|
||||
|
|
@ -183,6 +193,14 @@ func ConnectDB(initDB bool) error {
|
|||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
if err := ensureCopiedStageColumns(); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
if err := ensureMessageIdentitySchema(); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
}
|
||||
if err := seedArchiveMailboxesFromFS(); err != nil {
|
||||
_ = db.Close()
|
||||
return err
|
||||
|
|
@ -190,6 +208,153 @@ func ConnectDB(initDB bool) error {
|
|||
return nil
|
||||
}
|
||||
|
||||
// ensureMessageIdentitySchema upgrades the old Message-ID-only bookkeeping in
|
||||
// one transaction. No logical rows are discarded: old keys remain as rows with
|
||||
// an empty body hash and continue to act as compatibility aliases. The mbox
|
||||
// index is deliberately invalidated so its hashes are rebuilt from the archive
|
||||
// before the next migration pass makes a copy decision.
|
||||
func ensureMessageIdentitySchema() error {
|
||||
copiedColumns, err := tableColumns("copied")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
indexColumns, err := tableColumns("mbox_index")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
upgradeCopied := !copiedColumns["body_sha256"]
|
||||
upgradeIndex := !indexColumns["body_sha256"]
|
||||
if !upgradeCopied && !upgradeIndex {
|
||||
_, err := DB.Exec(`CREATE INDEX IF NOT EXISTS idx_copied_identity ON copied(account_id, folder, message_id, body_sha256);
|
||||
CREATE INDEX IF NOT EXISTS idx_mbox_index_identity ON mbox_index(account_id, folder, message_id, body_sha256);`)
|
||||
return err
|
||||
}
|
||||
tx, err := DB.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
if upgradeCopied {
|
||||
statements := []string{
|
||||
`ALTER TABLE copied RENAME TO copied_pre_identity`,
|
||||
`CREATE TABLE copied(
|
||||
account_id INTEGER NOT NULL REFERENCES accounts(id) ON DELETE CASCADE,
|
||||
folder TEXT NOT NULL,
|
||||
message_id TEXT NOT NULL,
|
||||
body_sha256 TEXT NOT NULL DEFAULT '',
|
||||
mbox_done INTEGER NOT NULL DEFAULT 0,
|
||||
target_done INTEGER NOT NULL DEFAULT 0,
|
||||
copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(account_id, folder, message_id, body_sha256)
|
||||
)`,
|
||||
`INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done, copied_at)
|
||||
SELECT account_id, folder, message_id, '', mbox_done, target_done, copied_at FROM copied_pre_identity`,
|
||||
`DROP TABLE copied_pre_identity`,
|
||||
}
|
||||
for _, stmt := range statements {
|
||||
if _, err := tx.Exec(stmt); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if upgradeIndex {
|
||||
statements := []string{
|
||||
`ALTER TABLE mbox_index RENAME TO mbox_index_pre_identity`,
|
||||
`CREATE TABLE mbox_index(
|
||||
account_id INTEGER NOT NULL,
|
||||
folder TEXT NOT NULL,
|
||||
seq INTEGER NOT NULL,
|
||||
message_id TEXT NOT NULL,
|
||||
body_sha256 TEXT NOT NULL DEFAULT '',
|
||||
subject TEXT NOT NULL DEFAULT '',
|
||||
from_addr TEXT NOT NULL DEFAULT '',
|
||||
date TEXT NOT NULL DEFAULT '',
|
||||
file_offset INTEGER NOT NULL,
|
||||
frame_len INTEGER NOT NULL,
|
||||
inner_offset INTEGER NOT NULL DEFAULT 0,
|
||||
inner_len INTEGER NOT NULL,
|
||||
UNIQUE(account_id, folder, seq)
|
||||
)`,
|
||||
`INSERT INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len)
|
||||
SELECT account_id, folder, seq, message_id, '', subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len FROM mbox_index_pre_identity`,
|
||||
`DROP TABLE mbox_index_pre_identity`,
|
||||
`UPDATE mbox_index_state SET indexed_bytes=-1`,
|
||||
}
|
||||
for _, stmt := range statements {
|
||||
if _, err := tx.Exec(stmt); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec(`CREATE INDEX IF NOT EXISTS idx_copied_identity ON copied(account_id, folder, message_id, body_sha256)`); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(`CREATE INDEX IF NOT EXISTS idx_mbox_index_identity ON mbox_index(account_id, folder, message_id, body_sha256)`); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func tableColumns(table string) (map[string]bool, error) {
|
||||
rows, err := DB.Query(`PRAGMA table_info(` + table + `)`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer rows.Close()
|
||||
columns := map[string]bool{}
|
||||
for rows.Next() {
|
||||
var cid, notNull, pk int
|
||||
var name, typ string
|
||||
var defaultValue any
|
||||
if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
columns[name] = true
|
||||
}
|
||||
return columns, rows.Err()
|
||||
}
|
||||
|
||||
func ensureCopiedStageColumns() error {
|
||||
columns := map[string]bool{}
|
||||
rows, err := DB.Query(`PRAGMA table_info(copied)`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for rows.Next() {
|
||||
var cid int
|
||||
var name, typ string
|
||||
var notNull int
|
||||
var defaultValue any
|
||||
var pk int
|
||||
if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
columns[name] = true
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
// Existing copied rows were created by the old all-or-nothing pipeline and
|
||||
// therefore represent both stages as complete. DEFAULT 1 preserves that
|
||||
// truth while all new writes below set their stage values explicitly.
|
||||
if !columns["mbox_done"] {
|
||||
if _, err := DB.Exec(`ALTER TABLE copied ADD COLUMN mbox_done INTEGER NOT NULL DEFAULT 1`); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if !columns["target_done"] {
|
||||
if _, err := DB.Exec(`ALTER TABLE copied ADD COLUMN target_done INTEGER NOT NULL DEFAULT 1`); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ensureDBFile(path string, initDB bool) error {
|
||||
info, err := os.Stat(path)
|
||||
if err == nil {
|
||||
|
|
@ -217,7 +382,6 @@ func ensureAppUserColumns() error {
|
|||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer rows.Close()
|
||||
for rows.Next() {
|
||||
var cid int
|
||||
var name, typ string
|
||||
|
|
@ -225,6 +389,7 @@ func ensureAppUserColumns() error {
|
|||
var defaultValue any
|
||||
var pk int
|
||||
if err := rows.Scan(&cid, &name, &typ, ¬Null, &defaultValue, &pk); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if name == "display_name" {
|
||||
|
|
@ -232,6 +397,10 @@ func ensureAppUserColumns() error {
|
|||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if !hasDisplayName {
|
||||
|
|
@ -518,22 +687,129 @@ func seedArchiveMailboxesFromFS() error {
|
|||
}
|
||||
|
||||
func AlreadyCopied(accountID int64, folder, messageID string) (bool, error) {
|
||||
if messageID == "" {
|
||||
return false, nil
|
||||
}
|
||||
var x int
|
||||
err := DB.QueryRow(`SELECT 1 FROM copied WHERE account_id=? AND folder=? AND message_id=?`, accountID, folder, messageID).Scan(&x)
|
||||
if errors.Is(err, sql.ErrNoRows) {
|
||||
return false, nil
|
||||
}
|
||||
return err == nil, err
|
||||
state, err := GetCopyState(accountID, folder, messageID)
|
||||
return state.MboxDone && state.TargetDone, err
|
||||
}
|
||||
|
||||
func MarkCopied(accountID int64, folder, messageID string) error {
|
||||
if messageID == "" {
|
||||
return MarkIdentityCopied(accountID, folder, MessageIdentity{MessageID: messageID})
|
||||
}
|
||||
|
||||
type CopyState struct {
|
||||
MboxDone bool
|
||||
TargetDone bool
|
||||
}
|
||||
|
||||
func GetCopyState(accountID int64, folder, messageID string) (CopyState, error) {
|
||||
return GetCopyIdentityState(accountID, folder, MessageIdentity{MessageID: messageID})
|
||||
}
|
||||
|
||||
func MarkMboxCopied(accountID int64, folder, messageID string) error {
|
||||
return MarkIdentityMboxCopied(accountID, folder, MessageIdentity{MessageID: messageID})
|
||||
}
|
||||
|
||||
func MarkTargetCopied(accountID int64, folder, messageID string) error {
|
||||
return MarkIdentityTargetCopied(accountID, folder, MessageIdentity{MessageID: messageID})
|
||||
}
|
||||
|
||||
func GetCopyIdentityState(accountID int64, folder string, identity MessageIdentity) (CopyState, error) {
|
||||
return GetCopyIdentityStateWithFolderAlias(accountID, folder, folder, identity)
|
||||
}
|
||||
|
||||
func GetCopyIdentityStateWithFolderAlias(accountID int64, folder, legacyFolder string, identity MessageIdentity) (CopyState, error) {
|
||||
if identity.MessageID == "" && identity.BodySHA256 == "" {
|
||||
return CopyState{}, nil
|
||||
}
|
||||
folders := []string{folder}
|
||||
if legacyFolder != "" && legacyFolder != folder {
|
||||
folders = append(folders, legacyFolder)
|
||||
}
|
||||
// Precise rows always win. For Message-ID-less mail the hash is the whole
|
||||
// identity, irrespective of which legacy key happened to be stored beside it.
|
||||
var mboxDone, targetDone int
|
||||
for _, candidateFolder := range folders {
|
||||
var mbox, target int
|
||||
var err error
|
||||
if identity.MessageID == "" {
|
||||
err = DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied
|
||||
WHERE account_id=? AND folder=? AND body_sha256=?`, accountID, candidateFolder, identity.BodySHA256).Scan(&mbox, &target)
|
||||
} else {
|
||||
err = DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied
|
||||
WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=?`,
|
||||
accountID, candidateFolder, identity.MessageID, identity.BodySHA256).Scan(&mbox, &target)
|
||||
}
|
||||
if err != nil {
|
||||
return CopyState{}, err
|
||||
}
|
||||
mboxDone |= mbox
|
||||
targetDone |= target
|
||||
}
|
||||
if mboxDone != 0 || targetDone != 0 {
|
||||
return CopyState{MboxDone: mboxDone != 0, TargetDone: targetDone != 0}, nil
|
||||
}
|
||||
// Once a Message-ID has precise archive identities, a different body with
|
||||
// that same ID is new and must not be hidden by the old wildcard row.
|
||||
if identity.MessageID != "" && identity.BodySHA256 != "" {
|
||||
var precise int
|
||||
for _, candidateFolder := range folders {
|
||||
var count int
|
||||
if err := DB.QueryRow(`SELECT count(*) FROM copied WHERE account_id=? AND folder=? AND message_id=? AND body_sha256<>''`,
|
||||
accountID, candidateFolder, identity.MessageID).Scan(&count); err != nil {
|
||||
return CopyState{}, err
|
||||
}
|
||||
precise += count
|
||||
}
|
||||
if precise > 0 {
|
||||
return CopyState{}, nil
|
||||
}
|
||||
}
|
||||
aliases := []string{identity.MessageID, identity.LegacyMessageID}
|
||||
if identity.MessageID == "" && identity.BodySHA256 != "" {
|
||||
aliases = append(aliases, "sha256:"+identity.BodySHA256)
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, alias := range aliases {
|
||||
alias = strings.TrimSpace(alias)
|
||||
if alias == "" || seen[alias] {
|
||||
continue
|
||||
}
|
||||
seen[alias] = true
|
||||
for _, candidateFolder := range folders {
|
||||
var mbox, target int
|
||||
err := DB.QueryRow(`SELECT COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0) FROM copied
|
||||
WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=''`, accountID, candidateFolder, alias).Scan(&mbox, &target)
|
||||
if err != nil {
|
||||
return CopyState{}, err
|
||||
}
|
||||
mboxDone |= mbox
|
||||
targetDone |= target
|
||||
}
|
||||
}
|
||||
return CopyState{MboxDone: mboxDone != 0, TargetDone: targetDone != 0}, nil
|
||||
}
|
||||
|
||||
func MarkIdentityCopied(accountID int64, folder string, identity MessageIdentity) error {
|
||||
return markCopyIdentity(accountID, folder, identity, true, true)
|
||||
}
|
||||
|
||||
func MarkIdentityMboxCopied(accountID int64, folder string, identity MessageIdentity) error {
|
||||
return markCopyIdentity(accountID, folder, identity, true, false)
|
||||
}
|
||||
|
||||
func MarkIdentityTargetCopied(accountID int64, folder string, identity MessageIdentity) error {
|
||||
return markCopyIdentity(accountID, folder, identity, false, true)
|
||||
}
|
||||
|
||||
func markCopyIdentity(accountID int64, folder string, identity MessageIdentity, mboxDone, targetDone bool) error {
|
||||
if identity.MessageID == "" && identity.BodySHA256 == "" {
|
||||
return nil
|
||||
}
|
||||
_, err := DB.Exec(`INSERT OR IGNORE INTO copied(account_id, folder, message_id) VALUES(?,?,?)`, accountID, folder, messageID)
|
||||
_, err := DB.Exec(`INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done)
|
||||
VALUES(?,?,?,?,?,?)
|
||||
ON CONFLICT(account_id, folder, message_id, body_sha256) DO UPDATE SET
|
||||
mbox_done=MAX(copied.mbox_done, excluded.mbox_done),
|
||||
target_done=MAX(copied.target_done, excluded.target_done),
|
||||
copied_at=CURRENT_TIMESTAMP`, accountID, folder, identity.MessageID, identity.BodySHA256, boolInt(mboxDone), boolInt(targetDone))
|
||||
return err
|
||||
}
|
||||
|
||||
|
|
@ -542,6 +818,7 @@ type MboxIndexEntry struct {
|
|||
Folder string
|
||||
Seq int
|
||||
MessageID string
|
||||
BodySHA256 string
|
||||
Subject string
|
||||
From string
|
||||
Date string
|
||||
|
|
@ -552,7 +829,7 @@ type MboxIndexEntry struct {
|
|||
}
|
||||
|
||||
func SaveMboxIndex(e MboxIndexEntry) error {
|
||||
if DB == nil || e.MessageID == "" {
|
||||
if DB == nil || (e.MessageID == "" && e.BodySHA256 == "") {
|
||||
return nil
|
||||
}
|
||||
var seq int
|
||||
|
|
@ -561,9 +838,9 @@ func SaveMboxIndex(e MboxIndexEntry) error {
|
|||
} else {
|
||||
_ = DB.QueryRow(`SELECT COALESCE(MAX(seq), -1) + 1 FROM mbox_index WHERE account_id=? AND folder=?`, e.AccountID, e.Folder).Scan(&seq)
|
||||
}
|
||||
_, err := DB.Exec(`INSERT OR IGNORE INTO mbox_index(account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
e.AccountID, e.Folder, seq, e.MessageID, e.Subject, e.From, e.Date, e.FileOffset, e.FrameLen, e.InnerOffset, e.InnerLen)
|
||||
_, err := DB.Exec(`INSERT OR REPLACE INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
e.AccountID, e.Folder, seq, e.MessageID, e.BodySHA256, e.Subject, e.From, e.Date, e.FileOffset, e.FrameLen, e.InnerOffset, e.InnerLen)
|
||||
return err
|
||||
}
|
||||
|
||||
|
|
@ -576,6 +853,53 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry,
|
|||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
folderAliases := []string{folder}
|
||||
aliasRows, err := tx.Query(`SELECT folder FROM copied WHERE account_id=? UNION SELECT folder FROM mbox_index WHERE account_id=?`, accountID, accountID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for aliasRows.Next() {
|
||||
var candidate string
|
||||
if err := aliasRows.Scan(&candidate); err != nil {
|
||||
_ = aliasRows.Close()
|
||||
return err
|
||||
}
|
||||
if candidate != folder && safeMboxName(candidate) == folder {
|
||||
folderAliases = append(folderAliases, candidate)
|
||||
}
|
||||
}
|
||||
if err := aliasRows.Err(); err != nil {
|
||||
_ = aliasRows.Close()
|
||||
return err
|
||||
}
|
||||
if err := aliasRows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
legacyIDsBySeq := map[int]string{}
|
||||
for _, candidateFolder := range folderAliases {
|
||||
rows, err := tx.Query(`SELECT seq, message_id FROM mbox_index WHERE account_id=? AND folder=?`, accountID, candidateFolder)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for rows.Next() {
|
||||
var seq int
|
||||
var messageID string
|
||||
if err := rows.Scan(&seq, &messageID); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if _, exists := legacyIDsBySeq[seq]; !exists || candidateFolder == folder {
|
||||
legacyIDsBySeq[seq] = messageID
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec(`DELETE FROM mbox_index WHERE account_id=? AND folder=?`, accountID, folder); err != nil {
|
||||
return err
|
||||
}
|
||||
|
|
@ -583,15 +907,33 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry,
|
|||
entry.AccountID = accountID
|
||||
entry.Folder = folder
|
||||
entry.Seq = i
|
||||
if entry.MessageID == "" {
|
||||
if entry.MessageID == "" && entry.BodySHA256 == "" {
|
||||
continue
|
||||
}
|
||||
if _, err := tx.Exec(`INSERT INTO mbox_index(account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
entry.AccountID, entry.Folder, entry.Seq, entry.MessageID, entry.Subject, entry.From, entry.Date,
|
||||
if _, err := tx.Exec(`INSERT INTO mbox_index(account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?,?)`,
|
||||
entry.AccountID, entry.Folder, entry.Seq, entry.MessageID, entry.BodySHA256, entry.Subject, entry.From, entry.Date,
|
||||
entry.FileOffset, entry.FrameLen, entry.InnerOffset, entry.InnerLen); err != nil {
|
||||
return err
|
||||
}
|
||||
// A full rebuild recovers records which reached the old append-first
|
||||
// pipeline but whose index/copied writes lost a SQLITE_BUSY race. In that
|
||||
// pipeline the target append happened before the mbox append, so recovered
|
||||
// records are complete in both stages. Marking them prevents a later watch
|
||||
// run from appending duplicates to either destination.
|
||||
state, found, err := copyStateForReindex(tx, entry.AccountID, folderAliases, entry, legacyIDsBySeq[i])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !found {
|
||||
state = CopyState{MboxDone: true, TargetDone: true}
|
||||
}
|
||||
if _, err := tx.Exec(`INSERT INTO copied(account_id, folder, message_id, body_sha256, mbox_done, target_done)
|
||||
VALUES(?,?,?,?,?,?)
|
||||
ON CONFLICT(account_id, folder, message_id, body_sha256) DO UPDATE SET mbox_done=1`,
|
||||
entry.AccountID, entry.Folder, entry.MessageID, entry.BodySHA256, 1, boolInt(state.TargetDone)); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec(`INSERT INTO mbox_index_state(account_id, folder, indexed_bytes)
|
||||
VALUES(?,?,?)
|
||||
|
|
@ -602,6 +944,88 @@ func ReplaceMboxIndex(accountID int64, folder string, entries []MboxIndexEntry,
|
|||
return tx.Commit()
|
||||
}
|
||||
|
||||
// RemoveMboxIndexFolderAliases removes only superseded index metadata such as
|
||||
// "INBOX.Newbies " after the actual archive file has been rebuilt under its
|
||||
// canonical filesystem name "INBOX.Newbies". copied compatibility aliases and
|
||||
// all mbox files remain untouched.
|
||||
func RemoveMboxIndexFolderAliases(accountID int64, canonicalFolders map[string]bool) error {
|
||||
rows, err := DB.Query(`SELECT folder FROM mbox_index WHERE account_id=? UNION SELECT folder FROM mbox_index_state WHERE account_id=?`, accountID, accountID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var stale []string
|
||||
for rows.Next() {
|
||||
var folder string
|
||||
if err := rows.Scan(&folder); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if !canonicalFolders[folder] && canonicalFolders[safeMboxName(folder)] {
|
||||
stale = append(stale, folder)
|
||||
}
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
_ = rows.Close()
|
||||
return err
|
||||
}
|
||||
if err := rows.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if len(stale) == 0 {
|
||||
return nil
|
||||
}
|
||||
tx, err := DB.Begin()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback()
|
||||
for _, folder := range stale {
|
||||
if _, err := tx.Exec(`DELETE FROM mbox_index WHERE account_id=? AND folder=?`, accountID, folder); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(`DELETE FROM mbox_index_state WHERE account_id=? AND folder=?`, accountID, folder); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit()
|
||||
}
|
||||
|
||||
func copyStateForReindex(tx *sql.Tx, accountID int64, folders []string, entry MboxIndexEntry, legacyMessageID string) (CopyState, bool, error) {
|
||||
type candidate struct{ messageID, bodyHash string }
|
||||
candidates := []candidate{{entry.MessageID, entry.BodySHA256}}
|
||||
if legacyMessageID != "" {
|
||||
candidates = append(candidates, candidate{legacyMessageID, ""})
|
||||
}
|
||||
if entry.MessageID != "" {
|
||||
candidates = append(candidates, candidate{entry.MessageID, ""})
|
||||
} else if entry.BodySHA256 != "" {
|
||||
candidates = append(candidates, candidate{"sha256:" + entry.BodySHA256, ""})
|
||||
}
|
||||
seen := map[candidate]bool{}
|
||||
var state CopyState
|
||||
found := false
|
||||
for _, c := range candidates {
|
||||
if seen[c] {
|
||||
continue
|
||||
}
|
||||
seen[c] = true
|
||||
for _, folder := range folders {
|
||||
var count, mboxDone, targetDone int
|
||||
if err := tx.QueryRow(`SELECT count(*), COALESCE(MAX(mbox_done),0), COALESCE(MAX(target_done),0)
|
||||
FROM copied WHERE account_id=? AND folder=? AND message_id=? AND body_sha256=?`,
|
||||
accountID, folder, c.messageID, c.bodyHash).Scan(&count, &mboxDone, &targetDone); err != nil {
|
||||
return CopyState{}, false, err
|
||||
}
|
||||
if count > 0 {
|
||||
found = true
|
||||
state.MboxDone = state.MboxDone || mboxDone != 0
|
||||
state.TargetDone = state.TargetDone || targetDone != 0
|
||||
}
|
||||
}
|
||||
}
|
||||
return state, found, nil
|
||||
}
|
||||
|
||||
func UpdateMboxIndexState(accountID int64, folder string, indexedBytes int64) error {
|
||||
if DB == nil || accountID == 0 {
|
||||
return nil
|
||||
|
|
@ -629,7 +1053,7 @@ func ListMboxIndex(accountID int64, folder string) ([]MboxIndexEntry, error) {
|
|||
if DB == nil || accountID == 0 {
|
||||
return nil, sql.ErrNoRows
|
||||
}
|
||||
rows, err := DB.Query(`SELECT account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len
|
||||
rows, err := DB.Query(`SELECT account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len
|
||||
FROM mbox_index WHERE account_id=? AND folder=? ORDER BY seq`, accountID, folder)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
|
|
@ -638,7 +1062,7 @@ func ListMboxIndex(accountID int64, folder string) ([]MboxIndexEntry, error) {
|
|||
var out []MboxIndexEntry
|
||||
for rows.Next() {
|
||||
var e MboxIndexEntry
|
||||
if err := rows.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen); err != nil {
|
||||
if err := rows.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.BodySHA256, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out = append(out, e)
|
||||
|
|
@ -650,10 +1074,10 @@ func GetMboxIndex(accountID int64, folder string, seq int) (MboxIndexEntry, erro
|
|||
if DB == nil || accountID == 0 {
|
||||
return MboxIndexEntry{}, sql.ErrNoRows
|
||||
}
|
||||
row := DB.QueryRow(`SELECT account_id, folder, seq, message_id, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len
|
||||
row := DB.QueryRow(`SELECT account_id, folder, seq, message_id, body_sha256, subject, from_addr, date, file_offset, frame_len, inner_offset, inner_len
|
||||
FROM mbox_index WHERE account_id=? AND folder=? AND seq=?`, accountID, folder, seq)
|
||||
var e MboxIndexEntry
|
||||
err := row.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen)
|
||||
err := row.Scan(&e.AccountID, &e.Folder, &e.Seq, &e.MessageID, &e.BodySHA256, &e.Subject, &e.From, &e.Date, &e.FileOffset, &e.FrameLen, &e.InnerOffset, &e.InnerLen)
|
||||
return e, err
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
package backend
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
|
|
@ -25,6 +26,17 @@ type RawMessage struct {
|
|||
InternalDate time.Time // Original-Zeit, per APPEND erhalten
|
||||
}
|
||||
|
||||
// MessageIdentity is stable across the IMAP and mbox representations. A real
|
||||
// Message-ID is not sufficient on its own: broken mailers sometimes reuse it
|
||||
// for byte-different messages. Messages without a Message-ID are identified by
|
||||
// BodySHA256 alone. LegacyMessageID keeps the old raw-IMAP hash addressable
|
||||
// while existing databases are migrated without re-copying mail.
|
||||
type MessageIdentity struct {
|
||||
MessageID string
|
||||
BodySHA256 string
|
||||
LegacyMessageID string
|
||||
}
|
||||
|
||||
type MessageHeader struct {
|
||||
UID uint32
|
||||
MessageID string
|
||||
|
|
@ -475,6 +487,43 @@ func messageID(body []byte) string {
|
|||
return fmt.Sprintf("sha256:%x", sha256.Sum256(body))
|
||||
}
|
||||
|
||||
func identityForRawMessage(m RawMessage) MessageIdentity {
|
||||
id := normalizeMessageID(m.MessageID)
|
||||
legacyID := id
|
||||
if strings.HasPrefix(strings.ToLower(id), "sha256:") {
|
||||
id = ""
|
||||
}
|
||||
if id == "" {
|
||||
if msg, err := mail.ReadMessage(bytes.NewReader(m.Body)); err == nil {
|
||||
id = normalizeMessageID(msg.Header.Get("Message-ID"))
|
||||
}
|
||||
}
|
||||
return MessageIdentity{
|
||||
MessageID: id,
|
||||
BodySHA256: bodySHA256(m.Body),
|
||||
LegacyMessageID: legacyID,
|
||||
}
|
||||
}
|
||||
|
||||
// canonicalMessageBytes mirrors the bytes which can be recovered from the
|
||||
// current mbox writer/reader pair: line endings are LF, trailing record
|
||||
// separators are removed and mbox's >From escaping is undone. Hashing this
|
||||
// representation on both paths prevents the raw-IMAP/mbox hash split.
|
||||
func canonicalMessageBytes(raw []byte) []byte {
|
||||
raw = bytes.ReplaceAll(raw, []byte("\r\n"), []byte("\n"))
|
||||
lines := bytes.Split(raw, []byte("\n"))
|
||||
for i, line := range lines {
|
||||
if bytes.HasPrefix(line, []byte(">From ")) {
|
||||
lines[i] = line[1:]
|
||||
}
|
||||
}
|
||||
return bytes.TrimRight(bytes.Join(lines, []byte("\n")), "\n")
|
||||
}
|
||||
|
||||
func bodySHA256(raw []byte) string {
|
||||
return fmt.Sprintf("%x", sha256.Sum256(canonicalMessageBytes(raw)))
|
||||
}
|
||||
|
||||
func normalizeMessageID(id string) string {
|
||||
return strings.Trim(strings.TrimSpace(id), "<>")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,3 +24,34 @@ func TestMessageIDFallsBackToBodyHash(t *testing.T) {
|
|||
t.Fatal("messageID hash fallback is not stable")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBodySHA256IsStableAcrossMboxNormalization(t *testing.T) {
|
||||
raw := []byte("From: a@example.com\r\nSubject: Test\r\n\r\nFirst\r\nFrom escaped\r\n\r\n")
|
||||
record := mboxRecord(RawMessage{Body: raw})
|
||||
parts := splitMboxRecordsWithOffsets(record)
|
||||
if len(parts) != 1 {
|
||||
t.Fatalf("mbox parts=%d, want 1", len(parts))
|
||||
}
|
||||
if got, want := bodySHA256(parts[0].Message), bodySHA256(raw); got != want {
|
||||
t.Fatalf("mbox hash=%s, raw hash=%s", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentityUsesMessageIDAndBodyHash(t *testing.T) {
|
||||
first := identityForRawMessage(RawMessage{MessageID: "<same@example.com>", Body: []byte("Message-ID: <same@example.com>\r\n\r\nfirst")})
|
||||
second := identityForRawMessage(RawMessage{MessageID: "<same@example.com>", Body: []byte("Message-ID: <same@example.com>\r\n\r\nsecond")})
|
||||
if first.MessageID != second.MessageID {
|
||||
t.Fatalf("message IDs differ: %#v %#v", first, second)
|
||||
}
|
||||
if first.BodySHA256 == second.BodySHA256 {
|
||||
t.Fatalf("different bodies have same identity hash: %#v %#v", first, second)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentityWithoutMessageIDKeepsLegacyRawHash(t *testing.T) {
|
||||
body := []byte("From: a@example.com\r\n\r\nbody")
|
||||
identity := identityForRawMessage(RawMessage{MessageID: messageID(body), Body: body})
|
||||
if identity.MessageID != "" || identity.BodySHA256 == "" || !strings.HasPrefix(identity.LegacyMessageID, "sha256:") {
|
||||
t.Fatalf("unexpected identity: %#v", identity)
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import (
|
|||
"fmt"
|
||||
"html"
|
||||
"io"
|
||||
"log"
|
||||
"mime"
|
||||
"mime/multipart"
|
||||
"mime/quotedprintable"
|
||||
|
|
@ -118,7 +119,7 @@ func mboxAppendInfoFromMessage(m RawMessage, record []byte) MboxAppendInfo {
|
|||
parts := splitMboxRecordsWithOffsets(record)
|
||||
if len(parts) > 0 {
|
||||
info.InnerOffset = parts[0].Offset
|
||||
info.InnerLen = int64(len(parts[0].Message))
|
||||
info.InnerLen = int64(parts[0].Length)
|
||||
}
|
||||
msg, err := mail.ReadMessage(bytes.NewReader(m.Body))
|
||||
if err != nil {
|
||||
|
|
@ -274,6 +275,7 @@ func ReindexArchives(name string) error {
|
|||
|
||||
func reindexAccountArchive(account Account) error {
|
||||
dir := filepath.Join(Cfg.MboxRoot, accountMboxDir(account))
|
||||
canonicalFolders := map[string]bool{}
|
||||
for _, path := range archiveMboxPaths(dir) {
|
||||
folder := folderNameFromMboxPath(path)
|
||||
if folder == "" {
|
||||
|
|
@ -282,8 +284,9 @@ func reindexAccountArchive(account Account) error {
|
|||
if err := ReindexMboxFile(account.ID, folder, path); err != nil {
|
||||
return fmt.Errorf("%s %s: %w", account.Name, folder, err)
|
||||
}
|
||||
canonicalFolders[folder] = true
|
||||
}
|
||||
return nil
|
||||
return RemoveMboxIndexFolderAliases(account.ID, canonicalFolders)
|
||||
}
|
||||
|
||||
func reindexPlainMbox(path string) ([]MboxIndexEntry, error) {
|
||||
|
|
@ -294,7 +297,7 @@ func reindexPlainMbox(path string) ([]MboxIndexEntry, error) {
|
|||
var out []MboxIndexEntry
|
||||
for _, part := range splitMboxRecordsWithOffsets(b) {
|
||||
entry := indexEntryFromMboxRecord(part.Message, part.Offset, int64(part.Length), 0, int64(part.Length))
|
||||
if entry.MessageID != "" {
|
||||
if entry.MessageID != "" || entry.BodySHA256 != "" {
|
||||
out = append(out, entry)
|
||||
}
|
||||
}
|
||||
|
|
@ -318,8 +321,8 @@ func reindexZstdMbox(path string) ([]MboxIndexEntry, error) {
|
|||
return nil, err
|
||||
}
|
||||
for _, part := range splitMboxRecordsWithOffsets(record) {
|
||||
entry := indexEntryFromMboxRecord(part.Message, frame.Offset, int64(frame.Length), part.Offset, int64(len(part.Message)))
|
||||
if entry.MessageID != "" {
|
||||
entry := indexEntryFromMboxRecord(part.Message, frame.Offset, int64(frame.Length), part.Offset, int64(part.Length))
|
||||
if entry.MessageID != "" || entry.BodySHA256 != "" {
|
||||
out = append(out, entry)
|
||||
}
|
||||
}
|
||||
|
|
@ -334,39 +337,44 @@ type mboxRecordPart struct {
|
|||
}
|
||||
|
||||
func splitMboxRecordsWithOffsets(b []byte) []mboxRecordPart {
|
||||
normalized := bytes.ReplaceAll(b, []byte("\r\n"), []byte("\n"))
|
||||
lines := bytes.SplitAfter(normalized, []byte("\n"))
|
||||
var out []mboxRecordPart
|
||||
var cur bytes.Buffer
|
||||
var curStart int64
|
||||
var pos int64
|
||||
inMsg := false
|
||||
for _, lineWithNL := range lines {
|
||||
line := bytes.TrimSuffix(lineWithNL, []byte("\n"))
|
||||
if bytes.HasPrefix(line, []byte("From ")) {
|
||||
if inMsg && cur.Len() > 0 {
|
||||
msg := bytes.TrimRight(cur.Bytes(), "\n")
|
||||
out = append(out, mboxRecordPart{Offset: curStart, Length: int(pos - curStart), Message: append([]byte(nil), msg...)})
|
||||
cur.Reset()
|
||||
}
|
||||
inMsg = true
|
||||
pos += int64(len(lineWithNL))
|
||||
curStart = pos
|
||||
continue
|
||||
messageStart := -1
|
||||
appendRecord := func(end int) {
|
||||
if messageStart < 0 || end <= messageStart {
|
||||
return
|
||||
}
|
||||
if inMsg {
|
||||
if bytes.HasPrefix(line, []byte(">From ")) {
|
||||
line = line[1:]
|
||||
}
|
||||
_, _ = cur.Write(line)
|
||||
_ = cur.WriteByte('\n')
|
||||
raw := b[messageStart:end]
|
||||
message := unescapeMboxMessage(raw)
|
||||
if len(message) == 0 {
|
||||
return
|
||||
}
|
||||
pos += int64(len(lineWithNL))
|
||||
out = append(out, mboxRecordPart{
|
||||
Offset: int64(messageStart),
|
||||
Length: end - messageStart,
|
||||
Message: append([]byte(nil), message...),
|
||||
})
|
||||
}
|
||||
if inMsg && cur.Len() > 0 {
|
||||
msg := bytes.TrimRight(cur.Bytes(), "\n")
|
||||
out = append(out, mboxRecordPart{Offset: curStart, Length: int(pos - curStart), Message: append([]byte(nil), msg...)})
|
||||
for lineStart := 0; lineStart < len(b); {
|
||||
relNL := bytes.IndexByte(b[lineStart:], '\n')
|
||||
lineEnd, nextLine := len(b), len(b)
|
||||
if relNL >= 0 {
|
||||
lineEnd = lineStart + relNL
|
||||
nextLine = lineEnd + 1
|
||||
}
|
||||
contentEnd := lineEnd
|
||||
if contentEnd > lineStart && b[contentEnd-1] == '\r' {
|
||||
contentEnd--
|
||||
}
|
||||
if bytes.HasPrefix(b[lineStart:contentEnd], []byte("From ")) {
|
||||
appendRecord(lineStart)
|
||||
messageStart = nextLine
|
||||
}
|
||||
if nextLine >= len(b) {
|
||||
break
|
||||
}
|
||||
lineStart = nextLine
|
||||
}
|
||||
appendRecord(len(b))
|
||||
return out
|
||||
}
|
||||
|
||||
|
|
@ -399,15 +407,12 @@ func splitZstdFrames(b []byte) []zstdFramePart {
|
|||
}
|
||||
|
||||
func indexEntryFromMboxRecord(raw []byte, fileOffset, frameLen, innerOffset, innerLen int64) MboxIndexEntry {
|
||||
entry := MboxIndexEntry{FileOffset: fileOffset, FrameLen: frameLen, InnerOffset: innerOffset, InnerLen: innerLen}
|
||||
entry := MboxIndexEntry{BodySHA256: bodySHA256(raw), FileOffset: fileOffset, FrameLen: frameLen, InnerOffset: innerOffset, InnerLen: innerLen}
|
||||
msg, err := mail.ReadMessage(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return entry
|
||||
}
|
||||
entry.MessageID = normalizeMessageID(msg.Header.Get("Message-ID"))
|
||||
if entry.MessageID == "" {
|
||||
entry.MessageID = messageID(raw)
|
||||
}
|
||||
entry.Subject = decodeHeader(msg.Header.Get("Subject"))
|
||||
entry.From = decodeHeader(msg.Header.Get("From"))
|
||||
entry.Date = decodeHeader(msg.Header.Get("Date"))
|
||||
|
|
@ -415,19 +420,7 @@ func indexEntryFromMboxRecord(raw []byte, fileOffset, frameLen, innerOffset, inn
|
|||
}
|
||||
|
||||
func unescapeMboxMessage(raw []byte) []byte {
|
||||
raw = bytes.ReplaceAll(raw, []byte("\r\n"), []byte("\n"))
|
||||
lines := bytes.Split(raw, []byte("\n"))
|
||||
var b bytes.Buffer
|
||||
for i, line := range lines {
|
||||
if i > 0 {
|
||||
_ = b.WriteByte('\n')
|
||||
}
|
||||
if bytes.HasPrefix(line, []byte(">From ")) {
|
||||
line = line[1:]
|
||||
}
|
||||
_, _ = b.Write(line)
|
||||
}
|
||||
return bytes.TrimRight(b.Bytes(), "\n")
|
||||
return canonicalMessageBytes(raw)
|
||||
}
|
||||
|
||||
func readMboxMessagesBytes(b []byte) [][]byte {
|
||||
|
|
@ -522,9 +515,6 @@ func readMboxMessageFromIndex(path string, index int) ([]byte, bool, error) {
|
|||
if err != nil {
|
||||
return nil, false, nil
|
||||
}
|
||||
if !strings.HasSuffix(strings.ToLower(path), ".zst") {
|
||||
return nil, false, nil
|
||||
}
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
|
|
@ -533,10 +523,28 @@ func readMboxMessageFromIndex(path string, index int) ([]byte, bool, error) {
|
|||
if _, err := f.Seek(entry.FileOffset, io.SeekStart); err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
if entry.FrameLen <= 0 || entry.FrameLen > int64(^uint(0)>>1) {
|
||||
return nil, true, fmt.Errorf("invalid indexed mbox length %d", entry.FrameLen)
|
||||
}
|
||||
frame := make([]byte, entry.FrameLen)
|
||||
if _, err := io.ReadFull(f, frame); err != nil {
|
||||
return nil, true, err
|
||||
}
|
||||
if !strings.HasSuffix(strings.ToLower(path), ".zst") {
|
||||
// Entries written during migration point at the complete mbox record
|
||||
// (separator included, InnerOffset > 0). Entries produced by a full
|
||||
// rebuild point directly at the raw message bytes (InnerOffset == 0).
|
||||
// Both variants are read by their persisted byte range, never by the
|
||||
// message's positional number in a freshly split file.
|
||||
if entry.InnerOffset > 0 {
|
||||
messages := readMboxMessagesBytes(frame)
|
||||
if len(messages) != 1 {
|
||||
return nil, true, fmt.Errorf("invalid indexed mbox record at offset %d", entry.FileOffset)
|
||||
}
|
||||
return messages[0], true, nil
|
||||
}
|
||||
return unescapeMboxMessage(frame), true, nil
|
||||
}
|
||||
dec, err := zstd.NewReader(nil)
|
||||
if err != nil {
|
||||
return nil, true, err
|
||||
|
|
@ -605,6 +613,29 @@ func decodeHeader(v string) string {
|
|||
}
|
||||
|
||||
func messageBody(raw []byte) string {
|
||||
return messageBodyWithContext(raw, "")
|
||||
}
|
||||
|
||||
func messageBodyWithContext(raw []byte, context string) string {
|
||||
return renderMessageBodySafely(raw, context, func() string {
|
||||
return messageBodyUnsafe(raw)
|
||||
})
|
||||
}
|
||||
|
||||
func renderMessageBodySafely(raw []byte, context string, render func() string) (body string) {
|
||||
defer func() {
|
||||
if recovered := recover(); recovered != nil {
|
||||
if strings.TrimSpace(context) == "" {
|
||||
context = "unbekannter Vorschaupfad"
|
||||
}
|
||||
log.Printf("mail preview rendering panic (%s): %v", context, recovered)
|
||||
body = "Darstellung fehlgeschlagen; Rohfassung:\n\n" + string(raw)
|
||||
}
|
||||
}()
|
||||
return render()
|
||||
}
|
||||
|
||||
func messageBodyUnsafe(raw []byte) string {
|
||||
msg, err := mail.ReadMessage(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return string(raw)
|
||||
|
|
@ -738,11 +769,12 @@ func markHTMLBreaks(s string) string {
|
|||
}
|
||||
|
||||
func replaceCaseInsensitive(s, old, new string) string {
|
||||
if old == "" {
|
||||
return s
|
||||
}
|
||||
var b strings.Builder
|
||||
lower := strings.ToLower(s)
|
||||
needle := strings.ToLower(old)
|
||||
for {
|
||||
i := strings.Index(lower, needle)
|
||||
i := asciiFoldIndex(s, old)
|
||||
if i < 0 {
|
||||
b.WriteString(s)
|
||||
return b.String()
|
||||
|
|
@ -751,29 +783,56 @@ func replaceCaseInsensitive(s, old, new string) string {
|
|||
b.WriteString(new)
|
||||
cut := i + len(old)
|
||||
s = s[cut:]
|
||||
lower = lower[cut:]
|
||||
}
|
||||
}
|
||||
|
||||
func stripHTMLBlock(s, tag string) string {
|
||||
lower := strings.ToLower(s)
|
||||
open := "<" + tag
|
||||
close := "</" + tag + ">"
|
||||
for {
|
||||
start := strings.Index(lower, open)
|
||||
start := asciiFoldIndex(s, open)
|
||||
if start < 0 {
|
||||
return s
|
||||
}
|
||||
end := strings.Index(lower[start:], close)
|
||||
end := asciiFoldIndex(s[start:], close)
|
||||
if end < 0 {
|
||||
return s[:start]
|
||||
}
|
||||
end += start + len(close)
|
||||
s = s[:start] + " " + s[end:]
|
||||
lower = strings.ToLower(s)
|
||||
}
|
||||
}
|
||||
|
||||
// asciiFoldIndex searches an ASCII needle case-insensitively in s. It scans
|
||||
// the original bytes, so the returned index is always valid for slicing s even
|
||||
// when s contains Unicode characters whose lowercase form has another byte
|
||||
// length (for example Turkish dotted I or the Kelvin sign).
|
||||
func asciiFoldIndex(s, needle string) int {
|
||||
if needle == "" || len(s) < len(needle) {
|
||||
return -1
|
||||
}
|
||||
for i := 0; i+len(needle) <= len(s); i++ {
|
||||
matched := true
|
||||
for j := 0; j < len(needle); j++ {
|
||||
if asciiLower(s[i+j]) != asciiLower(needle[j]) {
|
||||
matched = false
|
||||
break
|
||||
}
|
||||
}
|
||||
if matched {
|
||||
return i
|
||||
}
|
||||
}
|
||||
return -1
|
||||
}
|
||||
|
||||
func asciiLower(c byte) byte {
|
||||
if c >= 'A' && c <= 'Z' {
|
||||
return c + ('a' - 'A')
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func decodeTransfer(body []byte, enc string) []byte {
|
||||
switch strings.ToLower(strings.TrimSpace(enc)) {
|
||||
case "quoted-printable":
|
||||
|
|
|
|||
|
|
@ -2,6 +2,7 @@ package backend
|
|||
|
||||
import (
|
||||
"bytes"
|
||||
"database/sql"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
|
|
@ -240,6 +241,291 @@ func TestReadMboxMessagesBytesClonesBufferRecords(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestPlainIndexedReadUsesOffsetAcrossMissingMiddleIndex(t *testing.T) {
|
||||
oldCfg, oldDB := Cfg, DB
|
||||
root := t.TempDir()
|
||||
t.Cleanup(func() {
|
||||
if DB != nil {
|
||||
_ = DB.Close()
|
||||
}
|
||||
Cfg, DB = oldCfg, oldDB
|
||||
})
|
||||
|
||||
Cfg = Config{DBPath: filepath.Join(root, "mail-graveyard.db"), MboxRoot: filepath.Join(root, "backup")}
|
||||
if err := ConnectDB(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveArchiveMailbox("archive"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveAccount(Account{
|
||||
Name: "test-account", SrcHost: "source.example", SrcPort: 993, SrcSecurity: "tls", SrcUser: "source@example.com", SrcPass: "x",
|
||||
DstHost: "target.example", DstPort: 993, DstSecurity: "tls", DstUser: "target@example.com", DstPass: "x", MboxDir: "archive", Active: true,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
account, err := GetAccount("test-account")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
writer, err := NewMboxWriter(filepath.Join(Cfg.MboxRoot, "archive"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
firstInfo, err := writer.Append("INBOX", testRawMessage("first@example.com", "First"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveMboxIndex(MboxIndexEntry{AccountID: account.ID, Folder: "INBOX", MessageID: "first@example.com", Subject: "First", FileOffset: firstInfo.FileOffset, FrameLen: firstInfo.FrameLen, InnerOffset: firstInfo.InnerOffset, InnerLen: firstInfo.InnerLen}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := writer.Append("INBOX", testRawMessage("missing@example.com", "Missing middle")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
thirdInfo, err := writer.Append("INBOX", testRawMessage("third@example.com", "Third"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := SaveMboxIndex(MboxIndexEntry{AccountID: account.ID, Folder: "INBOX", MessageID: "third@example.com", Subject: "Third", FileOffset: thirdInfo.FileOffset, FrameLen: thirdInfo.FrameLen, InnerOffset: thirdInfo.InnerOffset, InnerLen: thirdInfo.InnerLen}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := UpdateMboxIndexState(account.ID, "INBOX", thirdInfo.FileOffset+thirdInfo.FrameLen); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
raw, err := ReadMboxMessage(thirdInfo.Path, 1)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !bytes.Contains(raw, []byte("Subject: Third")) || bytes.Contains(raw, []byte("Missing middle")) {
|
||||
t.Fatalf("indexed read drifted to the file position: %q", raw)
|
||||
}
|
||||
|
||||
if err := ReindexMboxFile(account.ID, "INBOX", thirdInfo.Path); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for i, subject := range []string{"First", "Missing middle", "Third"} {
|
||||
raw, err := ReadMboxMessage(thirdInfo.Path, i)
|
||||
if err != nil {
|
||||
t.Fatalf("rebuilt index %d: %v", i, err)
|
||||
}
|
||||
if !bytes.Contains(raw, []byte("Subject: "+subject)) {
|
||||
t.Fatalf("rebuilt index %d returned wrong message: %q", i, raw)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestConnectDBConfiguresBusyTimeoutAndCopyStages(t *testing.T) {
|
||||
oldCfg, oldDB := Cfg, DB
|
||||
root := t.TempDir()
|
||||
t.Cleanup(func() {
|
||||
if DB != nil {
|
||||
_ = DB.Close()
|
||||
}
|
||||
Cfg, DB = oldCfg, oldDB
|
||||
})
|
||||
Cfg = Config{DBPath: filepath.Join(root, "mail-graveyard.db"), MboxRoot: filepath.Join(root, "backup")}
|
||||
if err := ConnectDB(true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var timeout int
|
||||
if err := DB.QueryRow(`PRAGMA busy_timeout`).Scan(&timeout); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if timeout != 10000 {
|
||||
t.Fatalf("busy_timeout=%d, want 10000", timeout)
|
||||
}
|
||||
if err := SaveAccount(Account{
|
||||
Name: "stage-account", SrcHost: "source.example", SrcPort: 993, SrcSecurity: "tls", SrcUser: "source@example.com", SrcPass: "x",
|
||||
DstHost: "target.example", DstPort: 993, DstSecurity: "tls", DstUser: "target@example.com", DstPass: "x", Active: true,
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
account, err := GetAccount("stage-account")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := MarkMboxCopied(account.ID, "INBOX", "stage@example.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
state, err := GetCopyState(account.ID, "INBOX", "stage@example.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !state.MboxDone || state.TargetDone {
|
||||
t.Fatalf("unexpected mbox-only state: %#v", state)
|
||||
}
|
||||
if err := MarkTargetCopied(account.ID, "INBOX", "stage@example.com"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
state, err = GetCopyState(account.ID, "INBOX", "stage@example.com")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !state.MboxDone || !state.TargetDone {
|
||||
t.Fatalf("unexpected completed state: %#v", state)
|
||||
}
|
||||
first := MessageIdentity{MessageID: "reused@example.com", BodySHA256: bodySHA256([]byte("first"))}
|
||||
second := MessageIdentity{MessageID: "reused@example.com", BodySHA256: bodySHA256([]byte("second"))}
|
||||
if err := MarkIdentityCopied(account.ID, "INBOX", first); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
state, err = GetCopyIdentityState(account.ID, "INBOX", second)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if state.MboxDone || state.TargetDone {
|
||||
t.Fatalf("byte-different message with reused ID was hidden: %#v", state)
|
||||
}
|
||||
}
|
||||
|
||||
func TestIdentitySchemaUpgradePreservesLegacyAliases(t *testing.T) {
|
||||
oldCfg, oldDB := Cfg, DB
|
||||
root := t.TempDir()
|
||||
path := filepath.Join(root, "legacy.db")
|
||||
rawDB, err := sql.Open("sqlite", path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
legacySchema := []string{
|
||||
`CREATE TABLE accounts(id INTEGER PRIMARY KEY)`,
|
||||
`INSERT INTO accounts(id) VALUES(7)`,
|
||||
`CREATE TABLE copied(account_id INTEGER NOT NULL, folder TEXT NOT NULL, message_id TEXT NOT NULL, mbox_done INTEGER NOT NULL DEFAULT 1, target_done INTEGER NOT NULL DEFAULT 1, copied_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP, UNIQUE(account_id, folder, message_id))`,
|
||||
`INSERT INTO copied(account_id, folder, message_id, mbox_done, target_done) VALUES(7, 'INBOX.Newbies ', 'sha256:legacy-raw', 1, 0)`,
|
||||
`CREATE TABLE mbox_index(account_id INTEGER NOT NULL, folder TEXT NOT NULL, seq INTEGER NOT NULL, message_id TEXT NOT NULL, subject TEXT NOT NULL DEFAULT '', from_addr TEXT NOT NULL DEFAULT '', date TEXT NOT NULL DEFAULT '', file_offset INTEGER NOT NULL, frame_len INTEGER NOT NULL, inner_offset INTEGER NOT NULL DEFAULT 0, inner_len INTEGER NOT NULL, UNIQUE(account_id, folder, message_id))`,
|
||||
`INSERT INTO mbox_index(account_id, folder, seq, message_id, file_offset, frame_len, inner_len) VALUES(7, 'INBOX.Newbies ', 0, 'sha256:legacy-raw', 0, 10, 10)`,
|
||||
`CREATE TABLE mbox_index_state(account_id INTEGER NOT NULL, folder TEXT NOT NULL, indexed_bytes INTEGER NOT NULL DEFAULT 0, UNIQUE(account_id, folder))`,
|
||||
`INSERT INTO mbox_index_state(account_id, folder, indexed_bytes) VALUES(7, 'INBOX.Newbies ', 10)`,
|
||||
}
|
||||
for _, stmt := range legacySchema {
|
||||
if _, err := rawDB.Exec(stmt); err != nil {
|
||||
_ = rawDB.Close()
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if err := rawDB.Close(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if DB != nil {
|
||||
_ = DB.Close()
|
||||
}
|
||||
t.Cleanup(func() {
|
||||
if DB != nil {
|
||||
_ = DB.Close()
|
||||
}
|
||||
Cfg, DB = oldCfg, oldDB
|
||||
})
|
||||
Cfg = Config{DBPath: path, MboxRoot: filepath.Join(root, "backup")}
|
||||
if err := ConnectDB(false); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var messageID, bodyHash string
|
||||
var mboxDone, targetDone int
|
||||
if err := DB.QueryRow(`SELECT message_id, body_sha256, mbox_done, target_done FROM copied WHERE account_id=7`).Scan(&messageID, &bodyHash, &mboxDone, &targetDone); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if messageID != "sha256:legacy-raw" || bodyHash != "" || mboxDone != 1 || targetDone != 0 {
|
||||
t.Fatalf("legacy alias changed during upgrade: id=%q hash=%q stages=%d/%d", messageID, bodyHash, mboxDone, targetDone)
|
||||
}
|
||||
var indexedBytes int64
|
||||
if err := DB.QueryRow(`SELECT indexed_bytes FROM mbox_index_state WHERE account_id=7 AND folder='INBOX.Newbies '`).Scan(&indexedBytes); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if indexedBytes != -1 {
|
||||
t.Fatalf("identity-less index was not invalidated: %d", indexedBytes)
|
||||
}
|
||||
stableHash := bodySHA256([]byte("archived body"))
|
||||
if err := ReplaceMboxIndex(7, "INBOX.Newbies", []MboxIndexEntry{{BodySHA256: stableHash, FrameLen: 10, InnerLen: 10}}, 10); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := DB.QueryRow(`SELECT mbox_done, target_done FROM copied WHERE account_id=7 AND folder='INBOX.Newbies' AND message_id='' AND body_sha256=?`, stableHash).Scan(&mboxDone, &targetDone); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if mboxDone != 1 || targetDone != 0 {
|
||||
t.Fatalf("reindex lost pending target state from legacy alias: %d/%d", mboxDone, targetDone)
|
||||
}
|
||||
if err := RemoveMboxIndexFolderAliases(7, map[string]bool{"INBOX.Newbies": true}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var staleIndex, preservedAlias int
|
||||
if err := DB.QueryRow(`SELECT count(*) FROM mbox_index WHERE account_id=7 AND folder='INBOX.Newbies '`).Scan(&staleIndex); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := DB.QueryRow(`SELECT count(*) FROM copied WHERE account_id=7 AND folder='INBOX.Newbies ' AND message_id='sha256:legacy-raw'`).Scan(&preservedAlias); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if staleIndex != 0 || preservedAlias != 1 {
|
||||
t.Fatalf("folder alias cleanup stale_index=%d preserved_copy_alias=%d", staleIndex, preservedAlias)
|
||||
}
|
||||
}
|
||||
|
||||
func TestHTMLToTextHandlesUnicodeBeforeCaseInsensitiveTags(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
in string
|
||||
want []string
|
||||
drop []string
|
||||
}{
|
||||
{
|
||||
name: "turkish dotted i",
|
||||
in: "<p>Turkish İ test</p><BR>x",
|
||||
want: []string{"Turkish İ test", "x"},
|
||||
},
|
||||
{
|
||||
name: "kelvin sign",
|
||||
in: "<DIV>Kelvin K test</DIV><Br/>x",
|
||||
want: []string{"Kelvin K test", "x"},
|
||||
},
|
||||
{
|
||||
name: "mixed blocks and repeated replacements",
|
||||
in: "İ<style>bad K</STYLE><P>first</P>K<script>bad İ</SCRIPT><DIV>second</DIV><BR />third",
|
||||
want: []string{"İ", "first", "K", "second", "third"},
|
||||
drop: []string{"bad"},
|
||||
},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := htmlToText(tt.in)
|
||||
for _, want := range tt.want {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Fatalf("htmlToText(%q) = %q, missing %q", tt.in, got, want)
|
||||
}
|
||||
}
|
||||
for _, drop := range tt.drop {
|
||||
if strings.Contains(got, drop) {
|
||||
t.Fatalf("htmlToText(%q) = %q, unexpectedly contains %q", tt.in, got, drop)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestASCIIFoldIndexReturnsOriginalByteOffset(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
s string
|
||||
needle string
|
||||
want int
|
||||
}{
|
||||
{"İİ<BR>x", "<br>", len("İİ")},
|
||||
{"KK</DIV>", "</div>", len("KK")},
|
||||
{"prefix<Br />suffix", "<br />", len("prefix")},
|
||||
} {
|
||||
if got := asciiFoldIndex(tt.s, tt.needle); got != tt.want {
|
||||
t.Fatalf("asciiFoldIndex(%q, %q) = %d, want %d", tt.s, tt.needle, got, tt.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageBodyPanicFallsBackToRawMessage(t *testing.T) {
|
||||
raw := []byte("Subject: Evidence\r\n\r\nraw body")
|
||||
got := renderMessageBodySafely(raw, "test account=archive folder=INBOX seq=7", func() string {
|
||||
panic("synthetic parser failure")
|
||||
})
|
||||
if !strings.Contains(got, "Darstellung fehlgeschlagen") || !strings.Contains(got, string(raw)) {
|
||||
t.Fatalf("panic fallback did not preserve raw message: %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func testRawMessage(id, subject string) RawMessage {
|
||||
return RawMessage{
|
||||
MessageID: id,
|
||||
|
|
|
|||
|
|
@ -3,6 +3,7 @@ package backend
|
|||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
|
@ -16,8 +17,10 @@ import (
|
|||
// Posteingang) landen im vorhandenen Rollen-Ordner des Ziels statt als
|
||||
// Dublette; Pfad-Trenner wird umgehaengt. Dann EnsureFolder + Mails streamen.
|
||||
// 3. Pro Mail: Message-ID gegen copied-Cache pruefen -> schon da? ueberspringen.
|
||||
// Sonst DOPPELT schreiben: TargetMailbox.Append (IMAP) + MboxWriter.Append
|
||||
// (lokal), dann MarkCopied.
|
||||
// Sonst zuerst ins lokale mbox-Sicherheitsarchiv schreiben und diesen
|
||||
// Zustand festhalten; erst danach folgt TargetMailbox.Append. Beide Stufen
|
||||
// sind getrennt wiederaufnehmbar, damit ein Zielausfall nie das Archiv
|
||||
// verhindert und ein Nachlauf keine mbox-Dublette erzeugt.
|
||||
// 4. Fortschritt in jobs-Tabelle schreiben (Browser-Anzeige).
|
||||
//
|
||||
// name = Konto-Name oder "all". watch = true -> Delta-Schleife bis Abbruch
|
||||
|
|
@ -89,11 +92,6 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error {
|
|||
return fmt.Errorf("open source %s: %w", a.Name, err)
|
||||
}
|
||||
defer src.Close()
|
||||
dst, err := OpenIMAPTarget(a)
|
||||
if err != nil {
|
||||
return fmt.Errorf("open target %s: %w", a.Name, err)
|
||||
}
|
||||
defer dst.Close()
|
||||
archiveName := accountMboxDir(a)
|
||||
if archiveName == "" {
|
||||
return fmt.Errorf("keine Archiv-Mailbox fuer %s ausgewaehlt", a.Name)
|
||||
|
|
@ -109,86 +107,187 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error {
|
|||
_ = finishJob(jobID, 0, 0, 1, "error")
|
||||
return err
|
||||
}
|
||||
targets, err := dst.Folders()
|
||||
if err != nil {
|
||||
_ = finishJob(jobID, 0, 0, 1, "error")
|
||||
return err
|
||||
}
|
||||
folderMap, err := FolderMap(a.ID)
|
||||
if err != nil {
|
||||
_ = finishJob(jobID, 0, 0, 1, "error")
|
||||
return err
|
||||
}
|
||||
var dst TargetMailbox
|
||||
var targets []TargetFolder
|
||||
if opened, openErr := OpenIMAPTarget(a); openErr != nil {
|
||||
log.Printf("%s target unavailable; continuing archive-only: %v", a.Name, openErr)
|
||||
} else {
|
||||
dst = opened
|
||||
targets, err = dst.Folders()
|
||||
if err != nil {
|
||||
log.Printf("%s target folders unavailable; continuing archive-only: %v", a.Name, err)
|
||||
_ = dst.Close()
|
||||
dst = nil
|
||||
targets = nil
|
||||
}
|
||||
}
|
||||
defer func() {
|
||||
if dst != nil {
|
||||
_ = dst.Close()
|
||||
}
|
||||
}()
|
||||
total, done, errs := 0, 0, 0
|
||||
archived, targeted, targetPending := 0, 0, 0
|
||||
targetUnavailableLogged := false
|
||||
for _, folder := range folders {
|
||||
if len(selectedFolders) > 0 && !selectedFolders[folder.Name] {
|
||||
continue
|
||||
}
|
||||
dstFolder := MapSourceToTarget(folder.Name, folder.Attrs, folder.Delim, dst.Delim(), targets, folderMap[folder.Name])
|
||||
dstDelim := "/"
|
||||
if dst != nil {
|
||||
dstDelim = dst.Delim()
|
||||
}
|
||||
dstFolder := MapSourceToTarget(folder.Name, folder.Attrs, folder.Delim, dstDelim, targets, folderMap[folder.Name])
|
||||
if dstFolder == "" {
|
||||
dstFolder = folder.Name
|
||||
}
|
||||
if err := dst.EnsureFolder(dstFolder); err != nil {
|
||||
errs++
|
||||
log.Printf("%s %s ensure target %s error: %v", a.Name, folder.Name, dstFolder, err)
|
||||
continue
|
||||
targetReady := false
|
||||
if dst != nil {
|
||||
if err := dst.EnsureFolder(dstFolder); err != nil {
|
||||
log.Printf("%s %s ensure target %s error; retrying connection: %v", a.Name, folder.Name, dstFolder, err)
|
||||
_ = dst.Close()
|
||||
dst = nil
|
||||
if reopened, reopenErr := OpenIMAPTarget(a); reopenErr != nil {
|
||||
log.Printf("%s target reconnect failed; continuing archive-only: %v", a.Name, reopenErr)
|
||||
} else if ensureErr := reopened.EnsureFolder(dstFolder); ensureErr != nil {
|
||||
log.Printf("%s %s ensure target %s after reconnect failed; continuing archive-only: %v", a.Name, folder.Name, dstFolder, ensureErr)
|
||||
_ = reopened.Close()
|
||||
} else {
|
||||
dst = reopened
|
||||
targetReady = true
|
||||
}
|
||||
} else {
|
||||
targetReady = true
|
||||
}
|
||||
}
|
||||
folderDone, folderErrs := 0, 0
|
||||
folderArchived, folderTargeted, folderPending := 0, 0, 0
|
||||
folderTotal := 0
|
||||
identityFolder := safeMboxName(folder.Name)
|
||||
// After an identity-schema upgrade the indexed byte count is invalidated.
|
||||
// Rebuild from the actual archive before consulting copied, otherwise a
|
||||
// legacy Message-ID row could hide a byte-different message.
|
||||
archivePath := filepath.Join(mboxDir, safeMboxName(folder.Name)+mboxFileExt())
|
||||
if _, statErr := os.Stat(archivePath); statErr == nil {
|
||||
if indexErr := EnsureMboxIndexed(archivePath); indexErr != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s archive identity index error: %v", a.Name, folder.Name, indexErr)
|
||||
continue
|
||||
}
|
||||
} else if !os.IsNotExist(statErr) {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s archive stat error: %v", a.Name, folder.Name, statErr)
|
||||
continue
|
||||
}
|
||||
if err := src.Fetch(folder.Name, func(m RawMessage) error {
|
||||
total++
|
||||
folderTotal++
|
||||
already, err := AlreadyCopied(a.ID, folder.Name, m.MessageID)
|
||||
identity := identityForRawMessage(m)
|
||||
state, err := GetCopyIdentityStateWithFolderAlias(a.ID, identityFolder, folder.Name, identity)
|
||||
if err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s dedup error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if already {
|
||||
if state.MboxDone && state.TargetDone {
|
||||
return nil
|
||||
}
|
||||
if err := dst.Append(dstFolder, m); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s append to %s error %s: %v", a.Name, folder.Name, dstFolder, m.MessageID, err)
|
||||
return nil
|
||||
if !state.MboxDone {
|
||||
appendInfo, err := mbox.Append(folder.Name, m)
|
||||
if err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := SaveMboxIndex(MboxIndexEntry{
|
||||
AccountID: a.ID,
|
||||
Folder: identityFolder,
|
||||
MessageID: identity.MessageID,
|
||||
BodySHA256: identity.BodySHA256,
|
||||
Subject: appendInfo.Subject,
|
||||
From: appendInfo.From,
|
||||
Date: appendInfo.Date,
|
||||
FileOffset: appendInfo.FileOffset,
|
||||
FrameLen: appendInfo.FrameLen,
|
||||
InnerOffset: appendInfo.InnerOffset,
|
||||
InnerLen: appendInfo.InnerLen,
|
||||
}); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox index error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := UpdateMboxIndexState(a.ID, identityFolder, appendInfo.FileOffset+appendInfo.FrameLen); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox index state error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := MarkIdentityMboxCopied(a.ID, identityFolder, identity); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox stage mark error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
state.MboxDone = true
|
||||
archived++
|
||||
folderArchived++
|
||||
}
|
||||
appendInfo, err := mbox.Append(folder.Name, m)
|
||||
if err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := SaveMboxIndex(MboxIndexEntry{
|
||||
AccountID: a.ID,
|
||||
Folder: folder.Name,
|
||||
MessageID: m.MessageID,
|
||||
Subject: appendInfo.Subject,
|
||||
From: appendInfo.From,
|
||||
Date: appendInfo.Date,
|
||||
FileOffset: appendInfo.FileOffset,
|
||||
FrameLen: appendInfo.FrameLen,
|
||||
InnerOffset: appendInfo.InnerOffset,
|
||||
InnerLen: appendInfo.InnerLen,
|
||||
}); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox index error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := UpdateMboxIndexState(a.ID, folder.Name, appendInfo.FileOffset+appendInfo.FrameLen); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mbox index state error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
if err := MarkCopied(a.ID, folder.Name, m.MessageID); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s mark error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
if !state.TargetDone {
|
||||
if dst == nil || !targetReady {
|
||||
errs++
|
||||
folderErrs++
|
||||
targetPending++
|
||||
folderPending++
|
||||
if !targetUnavailableLogged {
|
||||
log.Printf("%s target unavailable; messages remain safely archived with target_done=0", a.Name)
|
||||
targetUnavailableLogged = true
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := dst.Append(dstFolder, m); err != nil {
|
||||
log.Printf("%s %s append to %s error %s; retrying connection: %v", a.Name, folder.Name, dstFolder, m.MessageID, err)
|
||||
_ = dst.Close()
|
||||
dst = nil
|
||||
reopened, reopenErr := OpenIMAPTarget(a)
|
||||
if reopenErr == nil {
|
||||
reopenErr = reopened.EnsureFolder(dstFolder)
|
||||
}
|
||||
if reopenErr == nil {
|
||||
reopenErr = reopened.Append(dstFolder, m)
|
||||
}
|
||||
if reopenErr != nil {
|
||||
if reopened != nil {
|
||||
_ = reopened.Close()
|
||||
}
|
||||
errs++
|
||||
folderErrs++
|
||||
targetPending++
|
||||
folderPending++
|
||||
targetReady = false
|
||||
log.Printf("%s target reconnect/append failed; continuing archive-only: %v", a.Name, reopenErr)
|
||||
return nil
|
||||
}
|
||||
dst = reopened
|
||||
targetReady = true
|
||||
}
|
||||
if err := MarkIdentityTargetCopied(a.ID, identityFolder, identity); err != nil {
|
||||
errs++
|
||||
folderErrs++
|
||||
log.Printf("%s %s target stage mark error %s: %v", a.Name, folder.Name, m.MessageID, err)
|
||||
return nil
|
||||
}
|
||||
targeted++
|
||||
folderTargeted++
|
||||
}
|
||||
done++
|
||||
folderDone++
|
||||
|
|
@ -198,14 +297,16 @@ func migrateAccount(a Account, selectedFolders map[string]bool) error {
|
|||
folderErrs++
|
||||
log.Printf("%s %s fetch error: %v", a.Name, folder.Name, err)
|
||||
}
|
||||
log.Printf("migration %s %s -> %s: total=%d copied=%d errors=%d", a.Name, folder.Name, dstFolder, folderTotal, folderDone, folderErrs)
|
||||
log.Printf("migration %s %s -> %s: total=%d complete=%d archived=%d target=%d pending=%d errors=%d",
|
||||
a.Name, folder.Name, dstFolder, folderTotal, folderDone, folderArchived, folderTargeted, folderPending, folderErrs)
|
||||
}
|
||||
state := "success"
|
||||
if errs > 0 {
|
||||
state = "partial"
|
||||
}
|
||||
_ = finishJob(jobID, total, done, errs, state)
|
||||
log.Printf("migration %s all folders: total=%d copied=%d errors=%d", a.Name, total, done, errs)
|
||||
log.Printf("migration %s all folders: total=%d complete=%d archived=%d target=%d pending=%d errors=%d",
|
||||
a.Name, total, done, archived, targeted, targetPending, errs)
|
||||
if errs > 0 {
|
||||
return fmt.Errorf("migration %s finished with %d errors", a.Name, errs)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -62,7 +62,7 @@ func messageHandler(w http.ResponseWriter, r *http.Request) {
|
|||
decodeHeader(msg.Header.Get("Subject")),
|
||||
decodeHeader(msg.Header.Get("From")),
|
||||
decodeHeader(msg.Header.Get("Date")),
|
||||
messageBody(raw),
|
||||
messageBodyWithContext(raw, fmt.Sprintf("archive account=%q folder=%q seq=%d", account, folder, index)),
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -393,7 +393,7 @@ func renderTargetMatch(b *strings.Builder, match forwardedMatch) {
|
|||
decodeHeader(msg.Header.Get("Subject")),
|
||||
decodeHeader(msg.Header.Get("From")),
|
||||
decodeHeader(msg.Header.Get("Date"))+" | "+match.Location(),
|
||||
messageBody(match.Raw),
|
||||
messageBodyWithContext(match.Raw, "target "+match.Location()),
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -487,7 +487,8 @@ func filterEntries(path string, entries []MboxEntry, q string) []MboxEntry {
|
|||
continue
|
||||
}
|
||||
raw, err := ReadMboxMessage(path, entry.Index)
|
||||
if err == nil && strings.Contains(strings.ToLower(messageBody(raw)), needle) {
|
||||
if err == nil && strings.Contains(strings.ToLower(messageBodyWithContext(raw,
|
||||
fmt.Sprintf("archive search path=%q seq=%d", path, entry.Index))), needle) {
|
||||
out = append(out, entry)
|
||||
}
|
||||
}
|
||||
|
|
@ -523,7 +524,7 @@ func findTargetIMAPCopy(account, targetAccount string, raw []byte) (forwardedMat
|
|||
if err != nil {
|
||||
return forwardedMatch{}, false, err
|
||||
}
|
||||
match, ok, scanErr := scanTargetForMessage(dst, targetUser, id)
|
||||
match, ok, scanErr := scanTargetForMessage(dst, targetUser, id, bodySHA256(raw))
|
||||
closeErr := dst.Close()
|
||||
if scanErr != nil {
|
||||
return forwardedMatch{}, false, scanErr
|
||||
|
|
@ -538,7 +539,7 @@ func findTargetIMAPCopy(account, targetAccount string, raw []byte) (forwardedMat
|
|||
return forwardedMatch{}, false, nil
|
||||
}
|
||||
|
||||
func scanTargetForMessage(dst TargetMailbox, targetUser, messageID string) (forwardedMatch, bool, error) {
|
||||
func scanTargetForMessage(dst TargetMailbox, targetUser, messageID, bodyHash string) (forwardedMatch, bool, error) {
|
||||
messageID = normalizeMessageID(messageID)
|
||||
folders, err := dst.Folders()
|
||||
if err != nil {
|
||||
|
|
@ -559,6 +560,9 @@ func scanTargetForMessage(dst TargetMailbox, targetUser, messageID string) (forw
|
|||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if bodySHA256(msg.Body) != bodyHash {
|
||||
continue
|
||||
}
|
||||
return forwardedMatch{
|
||||
Account: "Ziel: " + targetUser,
|
||||
Folder: folder.Name,
|
||||
|
|
@ -582,6 +586,7 @@ func findForwardedCopy(account, targetAccount string, raw []byte) (forwardedMatc
|
|||
if err != nil {
|
||||
return forwardedMatch{}, false
|
||||
}
|
||||
bodyHash := bodySHA256(raw)
|
||||
for _, entry := range entries {
|
||||
if !entry.IsDir() || entry.Name() == account {
|
||||
continue
|
||||
|
|
@ -600,7 +605,7 @@ func findForwardedCopy(account, targetAccount string, raw []byte) (forwardedMatc
|
|||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if messageIDHeader(candidate) == id {
|
||||
if messageIDHeader(candidate) == id && bodySHA256(candidate) == bodyHash {
|
||||
return forwardedMatch{
|
||||
Account: entry.Name(),
|
||||
Folder: folderNameFromMboxPath(path),
|
||||
|
|
@ -713,7 +718,7 @@ func appendManualTargetCopy(account, folder, targetAccount string, raw []byte) (
|
|||
return "", err
|
||||
}
|
||||
if a.ID != 0 {
|
||||
_ = MarkCopied(a.ID, folder, id)
|
||||
_ = MarkIdentityCopied(a.ID, folder, identityForRawMessage(msg))
|
||||
}
|
||||
return "Manuell kopiert nach " + strings.TrimSpace(a.DstUser) + " / " + dstFolder + ".", nil
|
||||
}
|
||||
|
|
|
|||
|
|
@ -15,21 +15,24 @@ type TargetDedupReport struct {
|
|||
ExtraCopies int
|
||||
Candidates int
|
||||
WithoutID int
|
||||
VariantGroups int
|
||||
Deleted int
|
||||
FolderReports []TargetDedupFolderReport
|
||||
}
|
||||
|
||||
type TargetDedupFolderReport struct {
|
||||
Folder string
|
||||
Groups int
|
||||
ExtraCopies int
|
||||
WithoutID int
|
||||
Deletes []TargetDedupDelete
|
||||
Folder string
|
||||
Groups int
|
||||
ExtraCopies int
|
||||
WithoutID int
|
||||
VariantGroups int
|
||||
Deletes []TargetDedupDelete
|
||||
}
|
||||
|
||||
type TargetDedupDelete struct {
|
||||
UID uint32
|
||||
MessageID string
|
||||
UID uint32
|
||||
MessageID string
|
||||
BodySHA256 string
|
||||
}
|
||||
|
||||
func DedupTarget(name string, folders []string, apply bool) (TargetDedupReport, error) {
|
||||
|
|
@ -61,13 +64,14 @@ func DedupTarget(name string, folders []string, apply bool) (TargetDedupReport,
|
|||
report.ExtraCopies += folderReport.ExtraCopies
|
||||
report.Candidates += len(folderReport.Deletes)
|
||||
report.WithoutID += folderReport.WithoutID
|
||||
report.VariantGroups += folderReport.VariantGroups
|
||||
if apply {
|
||||
report.Deleted += len(folderReport.Deletes)
|
||||
}
|
||||
report.FolderReports = append(report.FolderReports, folderReport)
|
||||
}
|
||||
log.Printf("target dedup %s apply=%v folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d",
|
||||
report.Account, report.Apply, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID)
|
||||
log.Printf("target dedup %s apply=%v folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d protected_variant_groups=%d",
|
||||
report.Account, report.Apply, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID, report.VariantGroups)
|
||||
return report, nil
|
||||
}
|
||||
|
||||
|
|
@ -91,12 +95,35 @@ func dedupTargetFolder(dst TargetMailbox, folder string, apply bool) (TargetDedu
|
|||
if len(uids) < 2 {
|
||||
continue
|
||||
}
|
||||
sort.Slice(uids, func(i, j int) bool { return uids[i] < uids[j] })
|
||||
report.Groups++
|
||||
report.ExtraCopies += len(uids) - 1
|
||||
for _, uid := range uids[1:] {
|
||||
report.Deletes = append(report.Deletes, TargetDedupDelete{UID: uid, MessageID: id})
|
||||
deleteUIDs = append(deleteUIDs, uid)
|
||||
byBody := map[string][]uint32{}
|
||||
for _, uid := range uids {
|
||||
msg, err := dst.FetchOne(folder, uid)
|
||||
if err != nil {
|
||||
return report, fmt.Errorf("%s fetch uid %d for safe dedup: %w", folder, uid, err)
|
||||
}
|
||||
hash := bodySHA256(msg.Body)
|
||||
byBody[hash] = append(byBody[hash], uid)
|
||||
}
|
||||
if len(byBody) > 1 {
|
||||
report.VariantGroups++
|
||||
}
|
||||
hashes := make([]string, 0, len(byBody))
|
||||
for hash := range byBody {
|
||||
hashes = append(hashes, hash)
|
||||
}
|
||||
sort.Strings(hashes)
|
||||
for _, hash := range hashes {
|
||||
bodyUIDs := byBody[hash]
|
||||
if len(bodyUIDs) < 2 {
|
||||
continue
|
||||
}
|
||||
sort.Slice(bodyUIDs, func(i, j int) bool { return bodyUIDs[i] < bodyUIDs[j] })
|
||||
report.Groups++
|
||||
report.ExtraCopies += len(bodyUIDs) - 1
|
||||
for _, uid := range bodyUIDs[1:] {
|
||||
report.Deletes = append(report.Deletes, TargetDedupDelete{UID: uid, MessageID: id, BodySHA256: hash})
|
||||
deleteUIDs = append(deleteUIDs, uid)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Slice(report.Deletes, func(i, j int) bool {
|
||||
|
|
@ -108,14 +135,14 @@ func dedupTargetFolder(dst TargetMailbox, folder string, apply bool) (TargetDedu
|
|||
if apply && len(deleteUIDs) > 0 {
|
||||
sort.Slice(deleteUIDs, func(i, j int) bool { return deleteUIDs[i] < deleteUIDs[j] })
|
||||
for _, del := range report.Deletes {
|
||||
log.Printf("target dedup delete folder=%s uid=%d message_id=%s", folder, del.UID, del.MessageID)
|
||||
log.Printf("target dedup delete folder=%s uid=%d message_id=%s body_sha256=%s", folder, del.UID, del.MessageID, del.BodySHA256)
|
||||
}
|
||||
if err := dst.DeleteUIDs(folder, deleteUIDs); err != nil {
|
||||
return report, fmt.Errorf("%s delete uids: %w", folder, err)
|
||||
}
|
||||
}
|
||||
if report.Groups > 0 || report.WithoutID > 0 {
|
||||
log.Printf("target dedup folder=%s groups=%d extra=%d without_id=%d apply=%v", folder, report.Groups, report.ExtraCopies, report.WithoutID, apply)
|
||||
if report.Groups > 0 || report.WithoutID > 0 || report.VariantGroups > 0 {
|
||||
log.Printf("target dedup folder=%s groups=%d extra=%d without_id=%d protected_variant_groups=%d apply=%v", folder, report.Groups, report.ExtraCopies, report.WithoutID, report.VariantGroups, apply)
|
||||
}
|
||||
return report, nil
|
||||
}
|
||||
|
|
@ -125,13 +152,13 @@ func LogTargetDedupReport(report TargetDedupReport) {
|
|||
if report.Apply {
|
||||
mode = "APPLY"
|
||||
}
|
||||
log.Printf("target dedup report %s account=%s folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d",
|
||||
mode, report.Account, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID)
|
||||
log.Printf("target dedup report %s account=%s folders=%d groups=%d extra=%d candidates=%d deleted=%d without_id=%d protected_variant_groups=%d",
|
||||
mode, report.Account, report.Folders, report.Groups, report.ExtraCopies, report.Candidates, report.Deleted, report.WithoutID, report.VariantGroups)
|
||||
for _, folder := range report.FolderReports {
|
||||
if folder.Groups == 0 && folder.WithoutID == 0 {
|
||||
if folder.Groups == 0 && folder.WithoutID == 0 && folder.VariantGroups == 0 {
|
||||
continue
|
||||
}
|
||||
log.Printf("target dedup report folder=%s groups=%d extra=%d without_id=%d", folder.Folder, folder.Groups, folder.ExtraCopies, folder.WithoutID)
|
||||
log.Printf("target dedup report folder=%s groups=%d extra=%d without_id=%d protected_variant_groups=%d", folder.Folder, folder.Groups, folder.ExtraCopies, folder.WithoutID, folder.VariantGroups)
|
||||
for _, del := range folder.Deletes {
|
||||
log.Printf("target dedup report candidate folder=%s uid=%d message_id=%s", folder.Folder, del.UID, del.MessageID)
|
||||
}
|
||||
|
|
|
|||
70
backend/13-target-dedup_test.go
Normal file
70
backend/13-target-dedup_test.go
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
package backend
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type dedupTargetStub struct {
|
||||
headers []MessageHeader
|
||||
bodies map[uint32][]byte
|
||||
deleted []uint32
|
||||
}
|
||||
|
||||
func (s *dedupTargetStub) Folders() ([]TargetFolder, error) { return nil, nil }
|
||||
func (s *dedupTargetStub) Delim() string { return "/" }
|
||||
func (s *dedupTargetStub) EnsureFolder(string) error { return nil }
|
||||
func (s *dedupTargetStub) Append(string, RawMessage) error { return nil }
|
||||
func (s *dedupTargetStub) Headers(string, int, int) ([]MessageHeader, error) {
|
||||
return s.headers, nil
|
||||
}
|
||||
func (s *dedupTargetStub) AllHeaders(string) ([]MessageHeader, error) { return s.headers, nil }
|
||||
func (s *dedupTargetStub) FetchOne(_ string, uid uint32) (RawMessage, error) {
|
||||
body, ok := s.bodies[uid]
|
||||
if !ok {
|
||||
return RawMessage{}, fmt.Errorf("missing uid %d", uid)
|
||||
}
|
||||
return RawMessage{Body: body}, nil
|
||||
}
|
||||
func (s *dedupTargetStub) DeleteUIDs(_ string, uids []uint32) error {
|
||||
s.deleted = append(s.deleted, uids...)
|
||||
return nil
|
||||
}
|
||||
func (s *dedupTargetStub) Close() error { return nil }
|
||||
|
||||
func TestDedupTargetFolderProtectsDifferentBodiesWithSameMessageID(t *testing.T) {
|
||||
sameA := []byte("Message-ID: <same@example.com>\r\n\r\nA")
|
||||
sameB := []byte("Message-ID: <same@example.com>\r\n\r\nB")
|
||||
dst := &dedupTargetStub{
|
||||
headers: []MessageHeader{
|
||||
{UID: 1, MessageID: "same@example.com"},
|
||||
{UID: 2, MessageID: "same@example.com"},
|
||||
{UID: 3, MessageID: "same@example.com"},
|
||||
},
|
||||
bodies: map[uint32][]byte{1: sameA, 2: sameA, 3: sameB},
|
||||
}
|
||||
report, err := dedupTargetFolder(dst, "INBOX", true)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if report.Groups != 1 || report.ExtraCopies != 1 || report.VariantGroups != 1 {
|
||||
t.Fatalf("unexpected report: %#v", report)
|
||||
}
|
||||
if !reflect.DeepEqual(dst.deleted, []uint32{2}) {
|
||||
t.Fatalf("deleted=%v, want only exact duplicate UID 2", dst.deleted)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDedupTargetFolderAbortsBeforeDeleteWhenBodyCannotBeVerified(t *testing.T) {
|
||||
dst := &dedupTargetStub{
|
||||
headers: []MessageHeader{{UID: 1, MessageID: "same@example.com"}, {UID: 2, MessageID: "same@example.com"}},
|
||||
bodies: map[uint32][]byte{1: []byte("Message-ID: <same@example.com>\r\n\r\nA")},
|
||||
}
|
||||
if _, err := dedupTargetFolder(dst, "INBOX", true); err == nil {
|
||||
t.Fatal("expected safe dedup to fail when a body cannot be fetched")
|
||||
}
|
||||
if len(dst.deleted) != 0 {
|
||||
t.Fatalf("deleted despite incomplete verification: %v", dst.deleted)
|
||||
}
|
||||
}
|
||||
84
bug-hashkey-instabil.md
Normal file
84
bug-hashkey-instabil.md
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
# 🟠 Instabiler Ersatzschlüssel: 4.024 Doppel-Einträge in `copied`
|
||||
|
||||
**Kein Datenverlust, keine Dubletten in Postfächern.** Aber die Buchhaltung
|
||||
stimmt nicht — und beim Aufräumen lauert eine Falle (siehe „⚠️ NICHT TUN").
|
||||
|
||||
## Befund
|
||||
|
||||
```
|
||||
Datei-Records: 58.051 ═ mbox_index: 58.051 ✓ (Drift 0)
|
||||
copied: 62.073 ← 4.022 mehr
|
||||
```
|
||||
`copied`-Zeilen ohne passenden `mbox_index`-Eintrag: **4.024**, davon
|
||||
**4.021 mit `sha256:`-Schlüssel**. Verteilung exakt auf die Konten mit
|
||||
`--reindex --rebuild`: gb +2533, rb +719, byrne +564, crystal +112, roby +92.
|
||||
(gb hatte gemessen 2.537 Mails ohne Message-ID — passt auf ~4 genau.)
|
||||
|
||||
## Ursache
|
||||
|
||||
Mails **ohne `Message-ID`-Header** bekommen `sha256:<hash>` als Ersatzschlüssel.
|
||||
Der Hash wird aber über **unterschiedliche Bytes** gebildet:
|
||||
|
||||
| Pfad | hasht |
|
||||
|---|---|
|
||||
| Migration (`07-migrate.go`) | die **IMAP-Rohbytes** der Mail |
|
||||
| Reindex/Rebuild (`06-mbox.go`) | den **mbox-Record aus der Datei** |
|
||||
|
||||
Die mbox-Fassung ist nicht byte-gleich mit der IMAP-Fassung: Der Writer setzt
|
||||
`>From `-Quoting und normalisiert Zeilenenden. → **anderer Hash → anderer
|
||||
Schlüssel → dieselbe Mail steht zweimal in `copied`.**
|
||||
|
||||
Betroffen sind alle **6.623** Mails ohne Message-ID (so viele hat `mbox_index`
|
||||
mit `sha256:`-Schlüssel).
|
||||
|
||||
## ⚠️ NICHT TUN: die „Waisen" einfach löschen
|
||||
|
||||
Die 4.024 Waisen sind **nicht** der Müll — sie sind die Zeilen, auf die der
|
||||
**Migrationspfad tatsächlich matcht** (er rechnet den IMAP-Hash aus). Die vom
|
||||
Rebuild ergänzten Zeilen (mbox-Hash) sind die redundanten.
|
||||
|
||||
**Werden die Waisen gelöscht, hält die Migration diese 4.021 Mails für
|
||||
unkopiert und schreibt sie erneut ins Ziel-Postfach → echte Dubletten in
|
||||
fremden Postfächern.** Genau der Schaden, den wir vermeiden wollen.
|
||||
|
||||
Aus demselben Grund darf die **Schlüsselberechnung nicht einfach umgestellt**
|
||||
werden: Für Konten **ohne** Rebuild (kolmer, palamari, colak, …) existiert nur
|
||||
der IMAP-Hash. Ein Wechsel auf den mbox-Hash lässt dort **alle** Message-ID-losen
|
||||
Mails als „neu" erscheinen → Massen-Neukopie.
|
||||
|
||||
## Fix (in dieser Reihenfolge, sonst knallt es)
|
||||
|
||||
1. **Schlüssel vereinheitlichen:** Der Ersatzschlüssel muss in **beiden** Pfaden
|
||||
über **dieselben** Bytes gehen. Sauberste Wahl: über die **mbox-normalisierte
|
||||
Fassung** (also das, was tatsächlich im Archiv steht) — dann liefern
|
||||
Migration und Reindex/Rebuild dauerhaft denselben Wert, auch nach jedem
|
||||
künftigen Rebuild.
|
||||
→ In `07-migrate.go` den Hash **nach** der mbox-Normalisierung bilden, nicht
|
||||
über die IMAP-Rohbytes.
|
||||
2. **Einmalige `copied`-Migration** (zwingend zusammen mit 1., nicht danach):
|
||||
Für jedes Konto/Ordner `copied` aus `mbox_index` neu ableiten
|
||||
(`message_id` aus dem Index, `mbox_done=1`), `target_done` aus dem
|
||||
bestehenden Stand übernehmen (die Mails **sind** im Ziel — nicht auf 0
|
||||
zurücksetzen!). Anschließend die alten IMAP-Hash-Zeilen entfernen.
|
||||
**In einer Transaktion**, mit Zählung vorher/nachher.
|
||||
3. **Abnahme:** `copied` == `mbox_index` (58.051 == 58.051), und der erste
|
||||
Watch-/Migrationslauf danach meldet **`archived=0 target=0 pending=0`** —
|
||||
also keine einzige Neukopie. Meldet er mehr, sofort stoppen: dann matcht der
|
||||
Schlüssel nicht und es entstehen gerade Dubletten.
|
||||
|
||||
## Alternativ (wenn 1.+2. zu heikel erscheinen)
|
||||
Zustand **so lassen**. Er ist funktional korrekt: Der Migrationspfad matcht auf
|
||||
die IMAP-Hash-Zeilen, es wird nichts doppelt kopiert (live bestätigt:
|
||||
`complete=0 archived=0 target=0 pending=0 errors=0`). Kosten: `copied` ist um
|
||||
~7 % aufgebläht und die Zahl taugt nicht als Beleg für „so viel ist archiviert".
|
||||
Dann aber **in `INSTALL.md` dokumentieren**, warum die Zahlen auseinandergehen —
|
||||
sonst rätselt in zwei Jahren jemand daran.
|
||||
|
||||
## Hängt zusammen mit
|
||||
Der noch offenen `body_sha256`-Bereinigung (byte-verschiedene Mails mit gleicher
|
||||
Message-ID, ~109 Nachzügler): Beide drehen sich um „was ist die Identität einer
|
||||
Mail". Am besten **in einem Zug** lösen und **eine** Schlüsselregel festlegen:
|
||||
- Message-ID vorhanden → `message_id` **+ `body_sha256`** (fängt die
|
||||
byte-verschiedenen Fälle),
|
||||
- keine Message-ID → nur `body_sha256`,
|
||||
- **`body_sha256` immer über die mbox-normalisierte Fassung.**
|
||||
106
bug-htmltotext-panic.md
Normal file
106
bug-htmltotext-panic.md
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
# 🔴 Vorschau stürzt bei ~0,1 % der Mails ab (htmlToText, Unicode)
|
||||
|
||||
Gefunden beim Verifizieren des Index-Drift-Fixes. **Der Drift-Fix selbst ist
|
||||
sauber** (2000 Stichproben über 8 Ordner, 0 falsche Zuordnungen, Drift 0 über
|
||||
alle 69 Ordner) — das hier ist ein **separater, neuer Befund**.
|
||||
|
||||
## Symptom
|
||||
|
||||
```
|
||||
2026/07/16 09:45:44 http: panic serving: runtime error:
|
||||
slice bounds out of range [:207] with length 175
|
||||
backend.replaceCaseInsensitive 06-mbox.go:770
|
||||
backend.markHTMLBreaks 06-mbox.go:755
|
||||
backend.htmlToText 06-mbox.go:690
|
||||
backend.extractTextBody / messageBody / messageHandler
|
||||
```
|
||||
Die betroffene Mail ist **nicht anzeigbar** — die Verbindung stirbt. Gemessen:
|
||||
**1 von 2000 Stichproben (~0,1 %)**, hochgerechnet ~58 der 57.771 Mails.
|
||||
Beispiel: `kolmer/INBOX seq=2378 "[www.mtp24.pl] Weryfikacja adresu"` — eine
|
||||
polnische Mail. Auslöser ist Nicht-ASCII.
|
||||
|
||||
## Ursache — dieselbe Klasse wie der Buffer-Bug: zwei Koordinatensysteme
|
||||
|
||||
[`06-mbox.go:760`](backend/06-mbox.go):
|
||||
```go
|
||||
func replaceCaseInsensitive(s, old, new string) string {
|
||||
lower := strings.ToLower(s) // <-- kann ANDERE Byte-Laenge haben als s!
|
||||
needle := strings.ToLower(old)
|
||||
for {
|
||||
i := strings.Index(lower, needle) // Index in LOWER
|
||||
...
|
||||
b.WriteString(s[:i]) // auf S angewendet <-- FALSCH
|
||||
cut := i + len(old)
|
||||
s = s[cut:]
|
||||
lower = lower[cut:] // beide mit demselben cut -> driften auseinander
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
`strings.ToLower` ist **nicht byte-laengen-erhaltend**:
|
||||
- `İ` (U+0130, 2 Bytes) → `i̇` (3 Bytes) = **+1**
|
||||
- `K` (U+212A Kelvin, 3 Bytes) → `k` (1 Byte) = **−2**
|
||||
|
||||
Sobald ein solches Zeichen vorkommt, passen die Indizes aus `lower` nicht mehr
|
||||
auf `s`. Mit jedem `cut` driftet es weiter, bis `i > len(s)` → Panic. Das erklaert
|
||||
`[:207] with length 175` exakt.
|
||||
|
||||
**`stripHTMLBlock` direkt darunter hat denselben Fehler** (`lower`-Index auf `s`
|
||||
angewendet) — bitte mitfixen, auch wenn er noch nicht gecrasht ist.
|
||||
|
||||
## Fix
|
||||
|
||||
Die Needles sind **alle ASCII** (`<br>`, `</p>`, `</div>`, …). Also gar keine
|
||||
lowercase-Kopie bauen, sondern direkt auf dem Original suchen — mit
|
||||
ASCII-Case-Folding:
|
||||
|
||||
```go
|
||||
// asciiFoldIndex sucht needle (ASCII, bereits lowercase) case-insensitiv in s
|
||||
// und liefert einen Index, der IMMER fuer s gueltig ist.
|
||||
func asciiFoldIndex(s, needle string) int {
|
||||
n := len(needle)
|
||||
if n == 0 || len(s) < n { return -1 }
|
||||
for i := 0; i+n <= len(s); i++ {
|
||||
ok := true
|
||||
for j := 0; j < n; j++ {
|
||||
c := s[i+j]
|
||||
if 'A' <= c && c <= 'Z' { c += 'a' - 'A' }
|
||||
if c != needle[j] { ok = false; break }
|
||||
}
|
||||
if ok { return i }
|
||||
}
|
||||
return -1
|
||||
}
|
||||
```
|
||||
Damit entfaellt das zweite Koordinatensystem komplett — es gibt nur noch `s`.
|
||||
(Alternative: vorkompilierte `regexp.MustCompile("(?i)"+regexp.QuoteMeta(old))`
|
||||
— ebenfalls korrekt, aber langsamer und schwerer als noetig.)
|
||||
|
||||
## Zusaetzlich: `recover()` im Vorschau-Pfad — Pflicht für ein Beweis-Archiv
|
||||
|
||||
`htmlToText` verarbeitet **beliebiges, fremdes HTML** aus 20 Jahren Mail
|
||||
(Spam, kaputte Mailer, alle Sprachen). Ein Parser-Bug darf **nie** dazu fuehren,
|
||||
dass eine archivierte Mail unsichtbar wird.
|
||||
|
||||
In `messageHandler` (bzw. um `messageBody`) ein `defer recover()`:
|
||||
- bei Panic → **nicht** die Verbindung sterben lassen,
|
||||
- stattdessen den **Rohtext** der Mail anzeigen (bzw. Header + Hinweis
|
||||
„Darstellung fehlgeschlagen, Rohfassung unten"),
|
||||
- und den Vorfall loggen (Konto/Ordner/seq), damit wir solche Mails finden.
|
||||
|
||||
**Grundsatz:** Man muss jede archivierte Mail *irgendwie* ansehen koennen —
|
||||
notfalls roh. Lieber haesslich als unsichtbar.
|
||||
|
||||
## Abnahme
|
||||
|
||||
1. Unit-Test, der den Bug faengt (mit ASCII-only Test wuerde er durchrutschen!):
|
||||
```go
|
||||
in := "<p>Turkish İ test</p><BR>x" // U+0130 aendert die Byte-Laenge
|
||||
out := htmlToText(in) // darf NICHT panicken
|
||||
```
|
||||
Zusaetzlich ein Fall mit `K` (U+212A) und ein gemischter (mehrere Vorkommen,
|
||||
damit die Drift ueber mehrere Schleifendurchlaeufe sichtbar wuerde).
|
||||
2. `go test ./...` gruen.
|
||||
3. Live: `kolmer/INBOX seq=2378` (`[www.mtp24.pl] Weryfikacja adresu`) muss sich
|
||||
oeffnen lassen — heute stirbt die Verbindung.
|
||||
4. Ich (Claude) fahre danach den Sweep erneut: **0 Abstuerze** in 2000+ Proben.
|
||||
88
bug-index-drift.md
Normal file
88
bug-index-drift.md
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# 🔴 KRITISCH — Vorschau/Liste um 1 verschoben (Index-Drift durch SQLITE_BUSY)
|
||||
|
||||
**Der Reader-Aliasing-Fix (6c4073a) war richtig, aber es gibt einen ZWEITEN,
|
||||
unabhaengigen Bug.** Klick auf „iTagPro" zeigt „Orivelle Pen" (die Nachbarzeile).
|
||||
|
||||
## Root Cause — bewiesen, nicht vermutet
|
||||
|
||||
**Zwei Fehler, die sich verketten:**
|
||||
|
||||
1. **6 Mails sind in der mbox-DATEI, aber nicht im `mbox_index`.**
|
||||
Ursache: `PRAGMA busy_timeout` steht immer noch auf **0** (der
|
||||
`rettungslauf-fixes.md`-Brief ist nicht umgesetzt). Beim Rettungslauf gab es
|
||||
pro betroffenem Ordner `SQLITE_BUSY`: `mbox.Append` schrieb den Record in die
|
||||
Datei, der darauffolgende **Index-INSERT scheiterte sofort**. Ergebnis:
|
||||
Datei-Record vorhanden, Index-Zeile fehlt.
|
||||
|
||||
2. **Der plain-Reader liest nach DATEI-POSITION statt nach `file_offset`.**
|
||||
[`06-mbox.go` `readMboxMessageFromIndex`](backend/06-mbox.go): holt zwar die
|
||||
Index-Zeile (mit korrektem `file_offset`), wirft sie fuer plain aber weg
|
||||
(`if !strings.HasSuffix(path, ".zst") { return nil, false, nil }`) und faellt
|
||||
auf `msgs[index]` = Datei-Position zurueck. Ab der fehlenden Index-Zeile ist
|
||||
Position != seq → alles um 1 verschoben.
|
||||
|
||||
**Der Beweis (drift-scan ueber alle 66 Ordner):**
|
||||
```
|
||||
byrne/INBOX Datei 5869 Index 5868 +1 (Rettungslauf errors=1)
|
||||
crystal/INBOX Datei 4874 Index 4873 +1 (errors=2 ->
|
||||
crystal/INBOX.Sent Datei 2146 Index 2145 +1 genau 2 Ordner)
|
||||
gb/INBOX Datei 6675 Index 6674 +1 (errors=1)
|
||||
rb/INBOX Datei 8628 Index 8627 +1 (errors=1)
|
||||
roby/INBOX Datei 2718 Index 2717 +1 (errors=1)
|
||||
```
|
||||
**Drift-Anzahl == SQLITE_BUSY-Fehleranzahl, exakt.** Die 60 fehlerfreien Ordner
|
||||
haben 0 Drift. Und: Lesen ueber `file_offset` liefert fuer gb **4129/4129**
|
||||
Mails korrekt (0 echte Fehler) — die gespeicherten Offsets sind die Wahrheit.
|
||||
|
||||
## Fix (drei Teile, in dieser Reihenfolge)
|
||||
|
||||
### A. `busy_timeout` setzen — PFLICHT, sonst passiert es wieder
|
||||
`ConnectDB` (02-database.go): `PRAGMA busy_timeout = 10000;` neben `journal_mode=WAL`.
|
||||
|
||||
### B. Plain-mbox ueber `file_offset` lesen (der eigentliche Fix)
|
||||
`readMboxMessageFromIndex` **darf fuer plain nicht mehr auf die Positions-
|
||||
Zerlegung zurueckfallen.** Analog zum zstd-Pfad:
|
||||
- Index-Zeile holen → `file_offset` seeken → Record bis zum `file_offset` der
|
||||
**naechsten** Zeile (bzw. EOF) lesen, `TrimRight("\n")`.
|
||||
- Die fragile Neu-Zerlegung `readMboxMessages`+`msgs[index]` **ganz aus dem
|
||||
Vorschau-/Export-/Copy-Pfad entfernen.** Zwei getrennte Zerleger (Writer-
|
||||
Offsets vs. Reader-Resplit), die auseinanderlaufen koennen, sind die Wurzel.
|
||||
- Damit lesen Liste UND Vorschau aus **derselben** Quelle (`mbox_index`) → sie
|
||||
koennen nicht mehr auseinanderlaufen.
|
||||
|
||||
### C. Die 6 verlorenen Index-Zeilen nachziehen — VOLLER Reindex aus der Datei
|
||||
Das bestehende `--reindex` haengt nur ab `indexed_bytes` an — es findet die
|
||||
**mittendrin** fehlenden Mails nicht. Es braucht einen **Full-Rebuild**:
|
||||
- `--reindex --rebuild <konto|all>`: `mbox_index` fuer den Ordner **loeschen**
|
||||
und aus der Datei **komplett neu** aufbauen (jeder Record → seq, file_offset,
|
||||
message_id, subject, date). Danach ist Index-Anzahl == Datei-Records.
|
||||
- **Wichtig — `copied` mitziehen:** Diese 6 Mails stehen auch **nicht** in
|
||||
`copied` (der Busy-Fehler hat auch `MarkCopied` verhindert). Ein Watch-Lauf
|
||||
wuerde sie sonst erneut ins Ziel-Postfach **und** erneut in die mbox kopieren
|
||||
(Dublette in beiden!). Beim Rebuild deshalb pro Record die Message-ID (+ ggf.
|
||||
`body_sha256`) in `copied` mit `mbox_done=1` eintragen.
|
||||
|
||||
## Sekundaerbefunde (mitnehmen, kein Blocker)
|
||||
- **Ungueltiges UTF-8 in `mbox_index.subject`** (z.B. eine Tesla-Newsletter-Mail
|
||||
mit kaputtem Byte) — bricht sogar `sqlite3`-Abfragen. Subject vor dem Speichern
|
||||
nach UTF-8 sanitisieren (ungueltige Bytes ersetzen).
|
||||
- **Envelope-Message-ID vs. Roh-Header:** In ~8 gb-Faellen steht im Index ein
|
||||
`sha256:`-Fallback, obwohl die Datei einen echten `Message-ID`-Header hat.
|
||||
Beim Indizieren zuerst den Roh-Header parsen, bevor auf Hash zurueckgefallen
|
||||
wird — sonst greift Dedup fuer diese Mails nicht.
|
||||
|
||||
## Abnahme
|
||||
1. `go test ./...` gruen; `PRAGMA busy_timeout` = 10000.
|
||||
2. Test, der die Drift faengt: mbox mit N Records schreiben, eine Index-Zeile
|
||||
**in der Mitte** loeschen, dann Vorschau von seq > Luecke abrufen → muss die
|
||||
**richtige** Mail liefern (heute die Nachbarmail).
|
||||
3. `--reindex --rebuild all`, danach **drift-scan == 0** fuer ALLE 66 Ordner
|
||||
(Datei-Records == Index == Listen-Eintraege).
|
||||
4. Watch-Lauf nach dem Rebuild: `copied=0` fuer alle Konten (keine Neukopie der
|
||||
nachgezogenen Mails).
|
||||
5. Voller Sweep (Betreff **und** Datum) ueber gb/INBOX und crystal/INBOX:
|
||||
**0 Abweichungen**.
|
||||
|
||||
Ich (Claude) fahre nach dem Fix den drift-scan ueber alle Ordner UND einen
|
||||
Voll-Sweep gegen `mbox_index` — diesmal ueber ALLE betroffenen Postfaecher,
|
||||
nicht nur eines.
|
||||
137
rettungslauf-fixes.md
Normal file
137
rettungslauf-fixes.md
Normal file
|
|
@ -0,0 +1,137 @@
|
|||
# Codex-Brief — Fixes aus dem echten Rettungslauf (14.07., 15 GB / 58k Mails)
|
||||
|
||||
Der Lauf gegen die 18 echten `@dr-gold.de`-Postfaecher hat vier Schwaechen
|
||||
freigelegt, die im Testbetrieb nie sichtbar waren. **Fix 1 und 2 sind Pflicht,
|
||||
bevor wir den Nachschlag fahren** — sonst reisst es an derselben Stelle wieder.
|
||||
|
||||
Kontext: Der Quell-Hoster verliert derzeit Mails. Das lokale mbox-Archiv ist das
|
||||
Sicherheitsnetz. **Alles, was das Archiv gefaehrdet, ist ein Notfall.**
|
||||
|
||||
---
|
||||
|
||||
## 1. 🔴 `busy_timeout = 0` — SQLite scheitert sofort statt zu warten
|
||||
|
||||
**Befund:** `PRAGMA busy_timeout` steht auf **0**. Bei Lock-Konflikten (Watcher +
|
||||
Web-App + Migration auf derselben DB) bricht die Abfrage **sofort** ab:
|
||||
```
|
||||
mbox index error <id>: database is locked (5) (SQLITE_BUSY)
|
||||
```
|
||||
Sechsmal im Rettungslauf passiert.
|
||||
|
||||
**Fix:** In `ConnectDB` (02-database.go) neben `journal_mode=WAL` setzen:
|
||||
```sql
|
||||
PRAGMA busy_timeout = 10000; -- 10s warten statt sofort scheitern
|
||||
```
|
||||
Einzeiler, grosse Wirkung.
|
||||
|
||||
---
|
||||
|
||||
## 2. 🔴 Das Archiv haengt am Ziel — und das ist verkehrt herum
|
||||
|
||||
**Der schwerste Befund.** Heutige Reihenfolge in `migrateAccount`:
|
||||
```
|
||||
AlreadyCopied -> dst.Append (All-Inkl) -> mbox.Append (lokal) -> MarkCopied
|
||||
^^^^^^^^^^ scheitert das hier, wird per `return nil` abgebrochen
|
||||
-> die Mail landet AUCH NICHT im lokalen Archiv
|
||||
```
|
||||
|
||||
**Real passiert bei `petarus`:** Die Ziel-Verbindung starb mitten im Lauf
|
||||
(`use of closed network connection`). Danach schlug jedes `EnsureFolder`/`Append`
|
||||
fehl → **~620 von 1135 Mails wurden weder ins Ziel noch ins Archiv geschrieben.**
|
||||
Das Sicherheitsnetz hat genau dann gerissen, als es gebraucht wurde.
|
||||
|
||||
**Das Archiv muss zuerst kommen und vom Ziel unabhaengig sein.** Dafuer braucht
|
||||
`copied` getrennte Zustaende — sonst erzeugt ein Retry mbox-Dubletten:
|
||||
|
||||
```sql
|
||||
ALTER TABLE copied ADD COLUMN mbox_done INTEGER NOT NULL DEFAULT 1;
|
||||
ALTER TABLE copied ADD COLUMN target_done INTEGER NOT NULL DEFAULT 1;
|
||||
-- Default 1: Bestandszeilen sind beides-fertig, das stimmt.
|
||||
```
|
||||
|
||||
**Neuer Ablauf:**
|
||||
```
|
||||
1. Zeile vorhanden und mbox_done=1 und target_done=1 -> ueberspringen
|
||||
2. mbox.Append (ZUERST — das Archiv ist das Sicherheitsnetz)
|
||||
scheitert -> Fehler zaehlen, Mail ueberspringen, nichts markieren
|
||||
3. Zeile schreiben/aktualisieren: mbox_done=1, target_done=0
|
||||
4. dst.Append (Ziel)
|
||||
OK -> target_done=1
|
||||
scheitert -> Fehler zaehlen, WEITERMACHEN. Die Mail IST archiviert.
|
||||
```
|
||||
|
||||
**Was das loest:**
|
||||
- Ein sterbendes Ziel kostet nie wieder eine Archiv-Kopie.
|
||||
- Ein zweiter Lauf sieht `mbox_done=1, target_done=0` → **ueberspringt den
|
||||
mbox-Append** (keine Dublette!) und **holt nur das Ziel nach**.
|
||||
|
||||
`MarkCopied` bleibt sinngemaess das Letzte je Stufe. Ein Absturz zwischen
|
||||
mbox-Write und Zeilen-Update kopiert die Mail beim naechsten Lauf erneut ins
|
||||
Archiv — dieselbe bewusste Entscheidung wie bisher, nur jetzt pro Stufe.
|
||||
|
||||
---
|
||||
|
||||
## 3. 🟠 Kein Reconnect zum Ziel
|
||||
|
||||
Bei einem 2-GB-Postfach haelt All-Inkl die IMAP-Sitzung nicht durch. Stirbt sie,
|
||||
scheitert **jede** weitere Operation — der Rest des Kontos faellt weg.
|
||||
|
||||
**Fix:** Verbindungsfehler erkennen (`use of closed network connection`, EOF,
|
||||
`broken pipe`) und **die Ziel-Verbindung neu aufbauen** (1–2 Versuche mit kurzer
|
||||
Pause). Gelingt das nicht:
|
||||
- **nicht abbrechen**, sondern fuer den Rest des Kontos im **Archiv-Only-Modus**
|
||||
weiterlaufen (dank Fix 2 ist das gefahrlos — die Mails landen sicher im mbox,
|
||||
`target_done=0`), und
|
||||
- am Ende **laut melden**: `ZIEL NICHT ERREICHBAR — N Mails nur archiviert,
|
||||
Nachlauf noetig`.
|
||||
|
||||
Lieber ein vollstaendiges Archiv mit offenem Ziel-Nachlauf als beides halb.
|
||||
|
||||
---
|
||||
|
||||
## 4. 🟠 Gleiche Message-ID im selben Ordner → Mail wird still verworfen
|
||||
|
||||
**Befund:** Bei mehreren Konten ist `copied < total` **ohne** Fehler
|
||||
(neubauer 1, simsek 5, palamari 8, gb u. a. — zusammen ~25 Mails). Ursache: Die
|
||||
gleiche Message-ID kommt im selben Ordner zweimal vor (Re-Import, kaputte Mailer),
|
||||
und der Dedup wirft die zweite weg.
|
||||
|
||||
Fuer ein **Beweis-Archiv ist das nicht hinnehmbar**: Zwei Mails mit derselben
|
||||
Message-ID koennen **unterschiedlichen Inhalt** haben. Wir duerfen nie still eine
|
||||
byte-verschiedene Mail verwerfen.
|
||||
|
||||
**Fix:** `copied` um `body_sha256` erweitern, Unique-Key auf
|
||||
`(account_id, folder, message_id, body_sha256)`.
|
||||
- Gleiche ID **und** gleiche Bytes → echte Dublette, ueberspringen (wie bisher).
|
||||
- Gleiche ID, **andere** Bytes → **andere Mail**, archivieren.
|
||||
- Bestandszeilen: `body_sha256` leer lassen und beim Vergleich als Wildcard
|
||||
behandeln (kein Reindex-Zwang, keine Re-Kopie des Bestands).
|
||||
|
||||
---
|
||||
|
||||
## 5. 🟢 Phantom-Ordner tolerieren
|
||||
|
||||
`LIST` meldet `INBOX.Posteingang`, `SELECT` sagt `NO Mailbox doesn't exist`.
|
||||
Kaputte Quelle (passt zum Hoster-Problem). Heute: Fehler. Kuenftig: **Warnung**,
|
||||
Ordner ueberspringen, Lauf sauber weiterfuehren — das ist kein Fehler unseres
|
||||
Tools.
|
||||
|
||||
---
|
||||
|
||||
## Abnahme
|
||||
|
||||
1. `go test ./...` gruen.
|
||||
2. `PRAGMA busy_timeout` → 10000.
|
||||
3. **Der entscheidende Test — Ziel-Ausfall simulieren:** Migration eines
|
||||
Testkontos starten, waehrend des Laufs das Ziel unerreichbar machen
|
||||
(z. B. falscher `dst_host`/Port). Erwartung:
|
||||
- **Alle** Quell-Mails landen trotzdem **vollstaendig im lokalen mbox**.
|
||||
- `target_done=0` fuer diese Mails.
|
||||
- Lauf meldet laut „Ziel nicht erreichbar, Nachlauf noetig".
|
||||
- **Zweiter Lauf mit erreichbarem Ziel:** kopiert **nichts** ins mbox
|
||||
(keine Dublette!) und holt **nur** das Ziel nach.
|
||||
Das ist die Regression, die uns bei `petarus` ~620 Mails gekostet haette.
|
||||
4. Nachschlag `petarus` (`--folders` ordnerweise) → alle 1135 Mails im Archiv.
|
||||
|
||||
Ich (Claude) fahre danach den Verifikationslauf: Quell-Anzahl vs. Archiv-Anzahl
|
||||
je Konto, und ziehe alle Luecken nach.
|
||||
Loading…
Add table
Add a link
Reference in a new issue