feat(backup): add encrypted Google backups
This commit is contained in:
parent
0337dcc572
commit
e49e9f45c3
@ -1,5 +1,10 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
### Added
|
||||
- Backup: add `gog backup` with age-encrypted Git shards, Gmail labels/raw message export, manifest status, full decrypt-and-verify, docs, and security-focused regression coverage.
|
||||
|
||||
## 0.13.0 - 2026-04-20
|
||||
|
||||
### Highlights
|
||||
|
||||
33
README.md
33
README.md
@ -9,6 +9,7 @@ Fast, script-friendly CLI for Gmail, Calendar, Chat, Classroom, Drive, Docs, Sli
|
||||
|
||||
- **Gmail** - search threads/messages, send mail, view attachments, manage labels/drafts/filters/delegation/vacation settings, auto-reply once to matching mail, modify single messages, export filters, inspect history, and run Pub/Sub watch webhooks
|
||||
- **Email tracking** - track opens for `gog gmail send --track` with a small Cloudflare Worker backend
|
||||
- **Encrypted backups** - export Google account data to age-encrypted Git shards (`gog backup`, Gmail first)
|
||||
- **Calendar** - list/create/update/delete events, manage invitations, aliases, subscriptions, team calendars, free/busy/conflicts, propose new times, focus/OOO/working-location events, recurrence, and reminders
|
||||
- **Classroom** - manage courses, roster, coursework/materials, submissions, announcements, topics, invitations, guardians, profiles
|
||||
- **Chat** - list/find/create spaces, list messages/threads, send messages and DMs, and manage emoji reactions (Workspace-only)
|
||||
@ -720,6 +721,38 @@ Gmail watch (Pub/Sub push):
|
||||
- `watch serve --fetch-delay` defaults to `3s` and helps avoid Gmail History indexing races after push delivery.
|
||||
- `watch serve --exclude-labels` defaults to `SPAM,TRASH`; IDs are case-sensitive.
|
||||
|
||||
### Encrypted Backup
|
||||
|
||||
```bash
|
||||
gog backup init --repo ~/Projects/backup-gog --remote https://github.com/steipete/backup-gog.git
|
||||
gog backup push --services gmail --account you@gmail.com
|
||||
gog backup status
|
||||
gog backup verify
|
||||
```
|
||||
|
||||
For a bounded first run:
|
||||
|
||||
```bash
|
||||
gog backup push --services gmail --account you@gmail.com --query 'newer_than:7d' --max 25
|
||||
```
|
||||
|
||||
Backups use age-encrypted JSONL gzip shards under `data/`. `gog` stores the
|
||||
private age identity locally at `~/.gog/age.key`; GitHub only receives public
|
||||
`age1...` recipients, `manifest.json`, and encrypted `*.jsonl.gz.age` payloads.
|
||||
The private `AGE-SECRET-KEY-...` value must stay local or in a password manager.
|
||||
|
||||
`manifest.json` is intentionally cleartext for cheap status and verification.
|
||||
It exposes metadata: export time, service names, account hashes, shard paths,
|
||||
row counts, encrypted byte sizes, plaintext verification hashes, backup cadence,
|
||||
and which shards changed. It does not contain email bodies, subjects, senders,
|
||||
recipients, raw MIME, labels, Drive filenames, contacts, or event titles.
|
||||
|
||||
Security boundary: GitHub cannot read Google content without the age identity.
|
||||
Repository writers can still replace backup contents with different encrypted
|
||||
data, so keep write access tight and review unexpected backup commits. If the
|
||||
age identity leaks, rotate recipients and re-encrypt; old Git history may still
|
||||
contain shards decryptable with the leaked key. See `docs/backup.md`.
|
||||
|
||||
### Email Tracking
|
||||
|
||||
Track when recipients open your emails:
|
||||
|
||||
172
docs/backup.md
Normal file
172
docs/backup.md
Normal file
@ -0,0 +1,172 @@
|
||||
---
|
||||
summary: "Encrypted Google account backups"
|
||||
read_when:
|
||||
- Adding a new gog backup service adapter
|
||||
- Changing encrypted backup layout, manifest fields, or age identity handling
|
||||
- Debugging backup-gog push, status, or verify
|
||||
---
|
||||
|
||||
# Encrypted Backups
|
||||
|
||||
`gog backup` writes Google account data into a Git repository as age-encrypted
|
||||
JSONL gzip shards. The intended repository is private, for example
|
||||
`https://github.com/steipete/backup-gog`, but service payloads are encrypted
|
||||
before Git sees them.
|
||||
|
||||
## Commands
|
||||
|
||||
Initialize local config, create an age identity if needed, seed the backup repo,
|
||||
and print the public recipient:
|
||||
|
||||
```bash
|
||||
gog backup init \
|
||||
--repo ~/Projects/backup-gog \
|
||||
--remote https://github.com/steipete/backup-gog.git
|
||||
```
|
||||
|
||||
Back up Gmail:
|
||||
|
||||
```bash
|
||||
gog backup push --services gmail --account steipete@gmail.com
|
||||
```
|
||||
|
||||
For a bounded smoke run:
|
||||
|
||||
```bash
|
||||
gog backup push --services gmail --account steipete@gmail.com --query 'newer_than:7d' --max 25
|
||||
```
|
||||
|
||||
Inspect cleartext manifest metadata:
|
||||
|
||||
```bash
|
||||
gog backup status
|
||||
```
|
||||
|
||||
Decrypt every shard and verify hashes and row counts:
|
||||
|
||||
```bash
|
||||
gog backup verify
|
||||
```
|
||||
|
||||
Use `--no-push` on `init` or `push` to commit locally without pushing to the
|
||||
remote.
|
||||
|
||||
## Files
|
||||
|
||||
Local config:
|
||||
|
||||
```text
|
||||
~/.gog/backup.json
|
||||
~/.gog/age.key
|
||||
```
|
||||
|
||||
Backup repo:
|
||||
|
||||
```text
|
||||
README.md
|
||||
manifest.json
|
||||
data/gmail/<account-hash>/labels.jsonl.gz.age
|
||||
data/gmail/<account-hash>/messages/YYYY/MM/part-0001.jsonl.gz.age
|
||||
```
|
||||
|
||||
`manifest.json` is intentionally cleartext. It contains format version, export
|
||||
time, public age recipients, service names, account hashes, shard paths, row
|
||||
counts, encrypted byte sizes, and plaintext hashes used for verification. It
|
||||
does not contain email subjects, senders, recipients, bodies, raw message IDs,
|
||||
or labels.
|
||||
|
||||
## Encryption
|
||||
|
||||
Backups use the Go `filippo.io/age` library with X25519 age identities. There
|
||||
is no backup password. The private identity is stored locally:
|
||||
|
||||
```text
|
||||
~/.gog/age.key
|
||||
```
|
||||
|
||||
The matching public recipient starts with `age1...` and is safe to store in
|
||||
`~/.gog/backup.json` and `manifest.json`. The private `AGE-SECRET-KEY-...`
|
||||
value must stay local or in a password manager.
|
||||
|
||||
For each shard, `gog backup push`:
|
||||
|
||||
1. Exports deterministic JSONL rows.
|
||||
2. Gzip-compresses the JSONL with a fixed gzip timestamp.
|
||||
3. Encrypts the compressed bytes with age for every configured recipient.
|
||||
4. Writes only encrypted `*.jsonl.gz.age` files to Git.
|
||||
5. Writes `manifest.json` with cleartext metadata for status and verification.
|
||||
|
||||
`gog backup verify` decrypts each shard with the local age identity, gunzips it,
|
||||
checks the plaintext SHA-256 hash from the manifest, and verifies row counts.
|
||||
|
||||
## Security Boundary
|
||||
|
||||
The encrypted shards protect Google content from GitHub and anyone else without
|
||||
the local age identity. That includes email bodies, subjects, senders,
|
||||
recipients, raw MIME payloads, labels, Drive filenames, contacts, event titles,
|
||||
and similar service data.
|
||||
|
||||
The manifest is not secret. It leaks operational metadata by design:
|
||||
|
||||
- Export time.
|
||||
- Public age recipients.
|
||||
- Service names.
|
||||
- Account hashes.
|
||||
- Shard paths and month buckets.
|
||||
- Row counts.
|
||||
- Encrypted byte sizes.
|
||||
- Plaintext shard hashes used by `verify`.
|
||||
- Backup cadence and which shards changed in Git history.
|
||||
|
||||
The account hash is not anonymity. It is useful to avoid putting the literal
|
||||
email address in paths, but someone who can guess the address can compute and
|
||||
compare the same hash.
|
||||
|
||||
Current trust model:
|
||||
|
||||
- Confidentiality: good for a private GitHub backup repo as long as
|
||||
`~/.gog/age.key` stays private.
|
||||
- Integrity against random corruption: `age` authentication, gzip decoding,
|
||||
plaintext SHA-256, and row-count verification catch damaged shards.
|
||||
- Integrity against repository writers: limited. Anyone with push access can
|
||||
replace encrypted backup data with different data encrypted to the public
|
||||
recipient. Keep repo write access restricted and review unexpected commits.
|
||||
- Key compromise: if `AGE-SECRET-KEY-...` leaks, historical shards in Git
|
||||
history may be readable. Rotate recipients, re-encrypt, and treat old Git
|
||||
history as exposed unless it is rewritten and all copies are removed.
|
||||
|
||||
Future hardening ideas:
|
||||
|
||||
- Store only ciphertext hashes in the public manifest and move plaintext hashes
|
||||
into encrypted shard metadata.
|
||||
- Sign manifests or commits with a local signing key so `verify` can prove who
|
||||
created the backup, not just that the files are internally consistent.
|
||||
- Add optional shard padding or disable gzip for deployments that care more
|
||||
about size side channels than repository size.
|
||||
|
||||
## Gmail Adapter
|
||||
|
||||
The first adapter backs up:
|
||||
|
||||
- Gmail labels.
|
||||
- Raw Gmail messages from `users.messages.get(format=raw)`.
|
||||
|
||||
Raw message payloads stay base64url encoded inside encrypted JSONL. This
|
||||
preserves the RFC 2822 message content while keeping the shard format text
|
||||
friendly.
|
||||
|
||||
`--include-spam-trash` defaults to true. Use `--query` and `--max` for bounded
|
||||
test exports; omit them for a full mailbox scan.
|
||||
|
||||
## Adding Services
|
||||
|
||||
Keep one backup engine and add service adapters. A service adapter should:
|
||||
|
||||
1. Resolve the authenticated account through normal `gog` auth.
|
||||
2. Export stable rows without writing Google data.
|
||||
3. Store sensitive identifiers, titles, names, and content inside encrypted
|
||||
shards only.
|
||||
4. Add cleartext manifest counts and account hashes only.
|
||||
5. Support bounded smoke flags when the service can be huge.
|
||||
|
||||
Good next adapters: Calendar, Contacts/People, Tasks, then Drive.
|
||||
@ -48,6 +48,13 @@ Generated from `gog schema --json`.
|
||||
- `gog auth tokens export <email> [flags]` - Export a refresh token to a file (contains secrets)
|
||||
- `gog auth tokens import <inPath>` - Import a refresh token file into keyring (contains secrets)
|
||||
- `gog auth tokens list` - List stored tokens (by key only)
|
||||
- `gog backup <command> [flags]` - Encrypted Google account backups
|
||||
- `gog backup gmail <command>` - Gmail backup operations
|
||||
- `gog backup gmail push [flags]` - Export Gmail into encrypted backup shards
|
||||
- `gog backup init [flags]` - Initialize encrypted backup config and repository
|
||||
- `gog backup push [flags]` - Export services into encrypted backup shards
|
||||
- `gog backup status [flags]` - Inspect backup manifest without decrypting shards
|
||||
- `gog backup verify [flags]` - Decrypt and verify all backup shards
|
||||
- `gog calendar (cal) <command> [flags]` - Google Calendar
|
||||
- `gog calendar (cal) acl (permissions,perms) <calendarId> [flags]` - List calendar ACL
|
||||
- `gog calendar (cal) alias <command>` - Manage calendar aliases
|
||||
@ -357,6 +364,12 @@ Generated from `gog schema --json`.
|
||||
- `gog sheets (sheet) <command> [flags]` - Google Sheets
|
||||
- `gog sheets (sheet) add-tab (add-sheet) <spreadsheetId> <tabName> [flags]` - Add a new tab/sheet to a spreadsheet
|
||||
- `gog sheets (sheet) append (add) <spreadsheetId> <range> [<values> ...] [flags]` - Append values to a range
|
||||
- `gog sheets (sheet) chart (charts) <command>` - Manage spreadsheet charts
|
||||
- `gog sheets (sheet) chart (charts) create (add,new) --spec-json=STRING <spreadsheetId> [flags]` - Create a chart from a JSON spec
|
||||
- `gog sheets (sheet) chart (charts) delete (rm,remove,del) <spreadsheetId> <chartId>` - Delete a chart
|
||||
- `gog sheets (sheet) chart (charts) get (show,info) <spreadsheetId> <chartId>` - Get full chart definition (spec + position)
|
||||
- `gog sheets (sheet) chart (charts) list <spreadsheetId>` - List charts in a spreadsheet
|
||||
- `gog sheets (sheet) chart (charts) update (edit,set) --spec-json=STRING <spreadsheetId> <chartId>` - Update a chart spec
|
||||
- `gog sheets (sheet) clear <spreadsheetId> <range>` - Clear values in a range
|
||||
- `gog sheets (sheet) copy (cp,duplicate) <spreadsheetId> <title> [flags]` - Copy a Google Sheet
|
||||
- `gog sheets (sheet) create (new) <title> [flags]` - Create a new spreadsheet
|
||||
|
||||
2
go.mod
2
go.mod
@ -3,6 +3,7 @@ module github.com/steipete/gogcli
|
||||
go 1.26.2
|
||||
|
||||
require (
|
||||
filippo.io/age v1.3.1
|
||||
github.com/99designs/keyring v1.2.2
|
||||
github.com/alecthomas/kong v1.15.0
|
||||
github.com/muesli/termenv v0.16.0
|
||||
@ -19,6 +20,7 @@ require (
|
||||
cloud.google.com/go/auth v0.20.0 // indirect
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
|
||||
cloud.google.com/go/compute/metadata v0.9.0 // indirect
|
||||
filippo.io/hpke v0.4.0 // indirect
|
||||
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
|
||||
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
|
||||
6
go.sum
6
go.sum
@ -1,9 +1,15 @@
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
|
||||
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
|
||||
cloud.google.com/go/auth v0.20.0 h1:kXTssoVb4azsVDoUiF8KvxAqrsQcQtB53DcSgta74CA=
|
||||
cloud.google.com/go/auth v0.20.0/go.mod h1:942/yi/itH1SsmpyrbnTMDgGfdy2BUqIKyd0cyYLc5Q=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
|
||||
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
|
||||
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
|
||||
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
|
||||
filippo.io/age v1.3.1 h1:hbzdQOJkuaMEpRCLSN1/C5DX74RPcNCk6oqhKMXmZi0=
|
||||
filippo.io/age v1.3.1/go.mod h1:EZorDTYUxt836i3zdori5IJX/v2Lj6kWFU0cfh6C0D4=
|
||||
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
|
||||
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
|
||||
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 h1:/vQbFIOMbk2FiG/kXiLl8BRyzTWDw7gX/Hz7Dd5eDMs=
|
||||
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4/go.mod h1:hN7oaIRCjzsZ2dE+yG5k+rsdt3qcwykqK6HVGcKwsw4=
|
||||
github.com/99designs/keyring v1.2.2 h1:pZd3neh/EmUzWONb35LxQfvuY7kiSXAq3HQd97+XBn0=
|
||||
|
||||
558
internal/backup/backup.go
Normal file
558
internal/backup/backup.go
Normal file
@ -0,0 +1,558 @@
|
||||
//nolint:err113,govet,revive,wrapcheck,wsl_v5 // Contextual errors keep backup call sites readable.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
const formatVersion = 1
|
||||
|
||||
type Manifest struct {
|
||||
Format int `json:"format"`
|
||||
App string `json:"app"`
|
||||
Encrypted bool `json:"encrypted"`
|
||||
Exported time.Time `json:"exported"`
|
||||
Recipients []string `json:"recipients,omitempty"`
|
||||
Services []string `json:"services,omitempty"`
|
||||
Accounts []string `json:"accounts,omitempty"`
|
||||
Counts map[string]int `json:"counts,omitempty"`
|
||||
Shards []ShardEntry `json:"shards"`
|
||||
}
|
||||
|
||||
type ShardEntry struct {
|
||||
Service string `json:"service"`
|
||||
Kind string `json:"kind"`
|
||||
Account string `json:"account,omitempty"`
|
||||
Path string `json:"path"`
|
||||
Rows int `json:"rows"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Bytes int64 `json:"bytes"`
|
||||
}
|
||||
|
||||
type PlainShard struct {
|
||||
Service string
|
||||
Kind string
|
||||
Account string
|
||||
Path string
|
||||
Rows int
|
||||
Plaintext []byte
|
||||
}
|
||||
|
||||
type Snapshot struct {
|
||||
Services []string
|
||||
Accounts []string
|
||||
Counts map[string]int
|
||||
Shards []PlainShard
|
||||
}
|
||||
|
||||
type Result struct {
|
||||
Repo string `json:"repo"`
|
||||
Changed bool `json:"changed"`
|
||||
Encrypted bool `json:"encrypted"`
|
||||
Shards int `json:"shards"`
|
||||
Counts map[string]int `json:"counts,omitempty"`
|
||||
}
|
||||
|
||||
func Init(ctx context.Context, opts Options) (Config, string, error) {
|
||||
cfg, err := ResolveOptions(opts)
|
||||
if err != nil {
|
||||
return Config{}, "", err
|
||||
}
|
||||
recipient, err := EnsureIdentity(cfg.Identity)
|
||||
if err != nil {
|
||||
return Config{}, "", err
|
||||
}
|
||||
if len(cfg.Recipients) == 0 {
|
||||
cfg.Recipients = []string{recipient}
|
||||
}
|
||||
if err := SaveConfig(opts.ConfigPath, cfg); err != nil {
|
||||
return Config{}, "", err
|
||||
}
|
||||
if err := ensureRepo(ctx, cfg); err != nil {
|
||||
return Config{}, "", err
|
||||
}
|
||||
if err := writeBackupReadme(cfg.Repo); err != nil {
|
||||
return Config{}, "", err
|
||||
}
|
||||
_, err = commitAndPush(ctx, cfg, "docs: describe encrypted gog backup", opts.Push)
|
||||
return cfg, recipient, err
|
||||
}
|
||||
|
||||
func PushSnapshot(ctx context.Context, snapshot Snapshot, opts Options) (Result, error) {
|
||||
cfg, err := ResolveOptions(opts)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if len(cfg.Recipients) == 0 {
|
||||
recipient, err := RecipientFromIdentity(cfg.Identity)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
cfg.Recipients = []string{recipient}
|
||||
}
|
||||
if err := ensureRepo(ctx, cfg); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if err := writeBackupReadme(cfg.Repo); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
oldManifest, _ := readManifest(cfg.Repo)
|
||||
manifest, err := writeSnapshot(ctx, cfg, snapshot, oldManifest)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
changed, err := commitAndPush(ctx, cfg, "sync: update encrypted gog backup", opts.Push)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
return Result{Repo: cfg.Repo, Changed: changed, Encrypted: true, Shards: len(manifest.Shards), Counts: manifest.Counts}, nil
|
||||
}
|
||||
|
||||
func Verify(ctx context.Context, opts Options) (Result, error) {
|
||||
cfg, err := ResolveOptions(opts)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if err := ensureRepo(ctx, cfg); err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
manifest, err := readManifest(cfg.Repo)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if manifest.Format != formatVersion {
|
||||
return Result{}, fmt.Errorf("unsupported backup format %d", manifest.Format)
|
||||
}
|
||||
counts := map[string]int{}
|
||||
for _, shard := range manifest.Shards {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return Result{}, ctx.Err()
|
||||
default:
|
||||
}
|
||||
plaintext, err := decryptShardFile(cfg, shard)
|
||||
if err != nil {
|
||||
return Result{}, err
|
||||
}
|
||||
if got := sha256Hex(plaintext); got != shard.SHA256 {
|
||||
return Result{}, fmt.Errorf("backup shard hash mismatch for %s", shard.Path)
|
||||
}
|
||||
rows, err := countJSONLLines(plaintext)
|
||||
if err != nil {
|
||||
return Result{}, fmt.Errorf("count rows in %s: %w", shard.Path, err)
|
||||
}
|
||||
if rows != shard.Rows {
|
||||
return Result{}, fmt.Errorf("backup shard row count mismatch for %s: got %d, want %d", shard.Path, rows, shard.Rows)
|
||||
}
|
||||
key := shard.Service
|
||||
if strings.TrimSpace(shard.Kind) != "" {
|
||||
key += "." + shard.Kind
|
||||
}
|
||||
counts[key] += rows
|
||||
}
|
||||
return Result{Repo: cfg.Repo, Changed: false, Encrypted: manifest.Encrypted, Shards: len(manifest.Shards), Counts: counts}, nil
|
||||
}
|
||||
|
||||
func Status(ctx context.Context, opts Options) (Manifest, string, error) {
|
||||
cfg, err := ResolveOptions(opts)
|
||||
if err != nil {
|
||||
return Manifest{}, "", err
|
||||
}
|
||||
if err := ensureRepo(ctx, cfg); err != nil {
|
||||
return Manifest{}, "", err
|
||||
}
|
||||
manifest, err := readManifest(cfg.Repo)
|
||||
if err != nil {
|
||||
return Manifest{}, "", err
|
||||
}
|
||||
return manifest, cfg.Repo, nil
|
||||
}
|
||||
|
||||
func NewJSONLShard(service, kind, account, rel string, rows any) (PlainShard, error) {
|
||||
plaintext, count, err := encodeJSONL(rows)
|
||||
if err != nil {
|
||||
return PlainShard{}, err
|
||||
}
|
||||
return PlainShard{
|
||||
Service: strings.TrimSpace(service),
|
||||
Kind: strings.TrimSpace(kind),
|
||||
Account: strings.TrimSpace(account),
|
||||
Path: filepath.ToSlash(rel),
|
||||
Rows: count,
|
||||
Plaintext: plaintext,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func writeSnapshot(ctx context.Context, cfg Config, snapshot Snapshot, old Manifest) (Manifest, error) {
|
||||
recipients := normalizedStrings(cfg.Recipients)
|
||||
reuseEncrypted := sameStrings(old.Recipients, recipients)
|
||||
shards := make([]ShardEntry, 0, len(snapshot.Shards))
|
||||
for _, shard := range snapshot.Shards {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return Manifest{}, ctx.Err()
|
||||
default:
|
||||
}
|
||||
entry, err := writeShard(cfg, old, shard, reuseEncrypted)
|
||||
if err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
shards = append(shards, entry)
|
||||
}
|
||||
sort.Slice(shards, func(i, j int) bool { return shards[i].Path < shards[j].Path })
|
||||
manifest := Manifest{
|
||||
Format: formatVersion,
|
||||
App: "gog",
|
||||
Encrypted: true,
|
||||
Exported: time.Now().UTC(),
|
||||
Recipients: recipients,
|
||||
Services: normalizedStrings(snapshot.Services),
|
||||
Accounts: normalizedStrings(snapshot.Accounts),
|
||||
Counts: cloneCounts(snapshot.Counts),
|
||||
Shards: shards,
|
||||
}
|
||||
if manifest.Counts == nil {
|
||||
manifest.Counts = map[string]int{}
|
||||
for _, shard := range shards {
|
||||
key := shard.Service
|
||||
if shard.Kind != "" {
|
||||
key += "." + shard.Kind
|
||||
}
|
||||
manifest.Counts[key] += shard.Rows
|
||||
}
|
||||
}
|
||||
if equivalentManifest(old, manifest) {
|
||||
return old, nil
|
||||
}
|
||||
if err := removeStaleShards(cfg.Repo, shards); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
if err := writeManifest(cfg.Repo, manifest); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
return manifest, nil
|
||||
}
|
||||
|
||||
func writeShard(cfg Config, old Manifest, shard PlainShard, reuseEncrypted bool) (ShardEntry, error) {
|
||||
if strings.TrimSpace(shard.Service) == "" {
|
||||
return ShardEntry{}, fmt.Errorf("backup shard service is required")
|
||||
}
|
||||
hash := sha256Hex(shard.Plaintext)
|
||||
path, err := resolveShardPath(cfg.Repo, shard.Path)
|
||||
if err != nil {
|
||||
return ShardEntry{}, err
|
||||
}
|
||||
if oldEntry, ok := old.entry(shard.Path); reuseEncrypted && ok && oldEntry.SHA256 == hash {
|
||||
if info, err := os.Stat(path); err == nil {
|
||||
oldEntry.Bytes = info.Size()
|
||||
return oldEntry, nil
|
||||
}
|
||||
}
|
||||
encrypted, _, err := encryptShard(shard.Plaintext, cfg.Recipients)
|
||||
if err != nil {
|
||||
return ShardEntry{}, err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
return ShardEntry{}, err
|
||||
}
|
||||
if err := os.WriteFile(path, encrypted, 0o600); err != nil {
|
||||
return ShardEntry{}, err
|
||||
}
|
||||
return ShardEntry{
|
||||
Service: shard.Service,
|
||||
Kind: shard.Kind,
|
||||
Account: shard.Account,
|
||||
Path: shard.Path,
|
||||
Rows: shard.Rows,
|
||||
SHA256: hash,
|
||||
Bytes: int64(len(encrypted)),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func decryptShardFile(cfg Config, shard ShardEntry) ([]byte, error) {
|
||||
path, err := resolveShardPath(cfg.Repo, shard.Path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
ciphertext, err := os.ReadFile(path) // #nosec G304 -- resolveShardPath confines manifest-controlled shard paths to data/*.age inside the backup repo.
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return decryptShard(ciphertext, cfg.Identity)
|
||||
}
|
||||
|
||||
func resolveShardPath(repo, rel string) (string, error) {
|
||||
clean := path.Clean(strings.TrimSpace(rel))
|
||||
if clean == "." || clean == ".." || strings.HasPrefix(clean, "../") || path.IsAbs(clean) {
|
||||
return "", fmt.Errorf("backup shard path escapes backup root: %s", rel)
|
||||
}
|
||||
if !strings.HasPrefix(clean, "data/") || !strings.HasSuffix(clean, ".age") {
|
||||
return "", fmt.Errorf("invalid backup shard path: %s", rel)
|
||||
}
|
||||
full := filepath.Join(repo, filepath.FromSlash(clean))
|
||||
root := filepath.Clean(filepath.Join(repo, "data"))
|
||||
parent := filepath.Clean(filepath.Dir(full))
|
||||
if parent != root && !strings.HasPrefix(parent, root+string(filepath.Separator)) {
|
||||
return "", fmt.Errorf("backup shard path escapes backup root: %s", rel)
|
||||
}
|
||||
return full, nil
|
||||
}
|
||||
|
||||
func encodeJSONL(rows any) ([]byte, int, error) {
|
||||
value := reflect.ValueOf(rows)
|
||||
if value.Kind() != reflect.Slice {
|
||||
return nil, 0, fmt.Errorf("unsupported JSONL rows %T", rows)
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
enc := json.NewEncoder(&buf)
|
||||
for i := 0; i < value.Len(); i++ {
|
||||
if err := enc.Encode(value.Index(i).Interface()); err != nil {
|
||||
return nil, 0, err
|
||||
}
|
||||
}
|
||||
return buf.Bytes(), value.Len(), nil
|
||||
}
|
||||
|
||||
func DecodeJSONL[T any](plaintext []byte, out *[]T) error {
|
||||
scanner := bufio.NewScanner(bytes.NewReader(plaintext))
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
|
||||
for scanner.Scan() {
|
||||
var value T
|
||||
if err := json.Unmarshal(scanner.Bytes(), &value); err != nil {
|
||||
return err
|
||||
}
|
||||
*out = append(*out, value)
|
||||
}
|
||||
return scanner.Err()
|
||||
}
|
||||
|
||||
func countJSONLLines(plaintext []byte) (int, error) {
|
||||
scanner := bufio.NewScanner(bytes.NewReader(plaintext))
|
||||
scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
|
||||
count := 0
|
||||
for scanner.Scan() {
|
||||
if len(bytes.TrimSpace(scanner.Bytes())) > 0 {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count, scanner.Err()
|
||||
}
|
||||
|
||||
func readManifest(repo string) (Manifest, error) {
|
||||
data, err := os.ReadFile(filepath.Join(repo, "manifest.json")) // #nosec G304 -- repo is the configured local backup repository.
|
||||
if err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
var manifest Manifest
|
||||
if err := json.Unmarshal(data, &manifest); err != nil {
|
||||
return Manifest{}, err
|
||||
}
|
||||
return manifest, nil
|
||||
}
|
||||
|
||||
func writeManifest(repo string, manifest Manifest) error {
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data = append(data, '\n')
|
||||
return os.WriteFile(filepath.Join(repo, "manifest.json"), data, 0o600)
|
||||
}
|
||||
|
||||
func (m Manifest) entry(path string) (ShardEntry, bool) {
|
||||
for _, shard := range m.Shards {
|
||||
if shard.Path == path {
|
||||
return shard, true
|
||||
}
|
||||
}
|
||||
return ShardEntry{}, false
|
||||
}
|
||||
|
||||
func equivalentManifest(a, b Manifest) bool {
|
||||
if a.Format != b.Format ||
|
||||
a.App != b.App ||
|
||||
a.Encrypted != b.Encrypted ||
|
||||
!sameStrings(a.Recipients, b.Recipients) ||
|
||||
!sameStrings(a.Services, b.Services) ||
|
||||
!sameStrings(a.Accounts, b.Accounts) ||
|
||||
!reflect.DeepEqual(a.Counts, b.Counts) ||
|
||||
len(a.Shards) != len(b.Shards) {
|
||||
return false
|
||||
}
|
||||
for i := range a.Shards {
|
||||
left, right := a.Shards[i], b.Shards[i]
|
||||
left.Bytes, right.Bytes = 0, 0
|
||||
if left != right {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func normalizedStrings(values []string) []string {
|
||||
seen := map[string]struct{}{}
|
||||
out := make([]string, 0, len(values))
|
||||
for _, value := range values {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
if _, ok := seen[value]; ok {
|
||||
continue
|
||||
}
|
||||
seen[value] = struct{}{}
|
||||
out = append(out, value)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func sameStrings(a, b []string) bool {
|
||||
a, b = normalizedStrings(a), normalizedStrings(b)
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for i := range a {
|
||||
if a[i] != b[i] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func cloneCounts(in map[string]int) map[string]int {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]int, len(in))
|
||||
for key, value := range in {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func removeStaleShards(repo string, shards []ShardEntry) error {
|
||||
keep := map[string]struct{}{}
|
||||
for _, shard := range shards {
|
||||
keep[filepath.Clean(filepath.Join(repo, filepath.FromSlash(shard.Path)))] = struct{}{}
|
||||
}
|
||||
root := filepath.Join(repo, "data")
|
||||
if _, err := os.Stat(root); os.IsNotExist(err) {
|
||||
return nil
|
||||
}
|
||||
var stale []string
|
||||
if err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
|
||||
if err != nil || d == nil || d.IsDir() {
|
||||
return err
|
||||
}
|
||||
if !strings.HasSuffix(path, ".age") {
|
||||
return nil
|
||||
}
|
||||
clean := filepath.Clean(path)
|
||||
if _, ok := keep[clean]; ok {
|
||||
return nil
|
||||
}
|
||||
stale = append(stale, clean)
|
||||
return nil
|
||||
}); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, path := range stale {
|
||||
rel, err := filepath.Rel(root, path)
|
||||
if err != nil || rel == "." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
|
||||
return fmt.Errorf("stale shard path escapes backup root: %s", path)
|
||||
}
|
||||
if err := os.Remove(path); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func writeBackupReadme(repo string) error {
|
||||
path := filepath.Join(repo, "README.md")
|
||||
if _, err := os.Stat(path); err == nil {
|
||||
return nil
|
||||
}
|
||||
const body = `# backup-gog
|
||||
|
||||
Encrypted Git backup for Google account data exported by gog.
|
||||
|
||||
This repository is written by ` + "`gog backup push`" + `. It is safe to keep on
|
||||
GitHub because service payloads are encrypted before Git sees them.
|
||||
|
||||
## Layout
|
||||
|
||||
` + "```text" + `
|
||||
README.md
|
||||
manifest.json
|
||||
data/<service>/<account-hash>/...
|
||||
` + "```" + `
|
||||
|
||||
` + "`manifest.json`" + ` is cleartext and contains format version, export time,
|
||||
public age recipients, service names, account hashes, shard paths, row counts,
|
||||
encrypted byte sizes, and plaintext hashes used for verification. Email bodies,
|
||||
subjects, senders, Drive filenames, contacts, event titles, and other private
|
||||
Google data stay inside encrypted ` + "`*.jsonl.gz.age`" + ` shards.
|
||||
|
||||
## Security Model
|
||||
|
||||
Shard contents are deterministic JSONL, gzip-compressed with a fixed timestamp,
|
||||
and encrypted with age for every configured public recipient. The local
|
||||
` + "`~/.gog/age.key`" + ` identity is required to decrypt.
|
||||
|
||||
Git can still see manifest metadata: export time, public recipients, service
|
||||
names, account hashes, shard paths, encrypted byte sizes, plaintext shard
|
||||
hashes, backup cadence, and which encrypted shards changed. Git cannot read
|
||||
Google content without an age identity.
|
||||
|
||||
Anyone who can push to this repository can replace encrypted backup data with
|
||||
different data encrypted to your public recipient. Keep repository write access
|
||||
restricted and review unexpected backup commits. If an age identity is
|
||||
compromised, remove its public recipient and push a new backup; old Git history
|
||||
may still contain shards decryptable by the compromised key.
|
||||
|
||||
## Push
|
||||
|
||||
` + "```bash" + `
|
||||
gog backup push --services gmail
|
||||
` + "```" + `
|
||||
|
||||
The command pulls/rebases this checkout, exports selected Google services,
|
||||
writes encrypted shards, updates the manifest, commits, and pushes this
|
||||
repository.
|
||||
|
||||
## Verify
|
||||
|
||||
` + "```bash" + `
|
||||
gog backup verify
|
||||
` + "```" + `
|
||||
|
||||
` + "`verify`" + ` decrypts every shard with the local age identity and verifies the
|
||||
manifest hashes and row counts. It does not restore or write Google data.
|
||||
|
||||
## Recovery
|
||||
|
||||
Install gog, clone this repo to the path in ` + "`~/.gog/backup.json`" + `,
|
||||
restore the local age identity file, then run:
|
||||
|
||||
` + "```bash" + `
|
||||
gog backup verify
|
||||
` + "```" + `
|
||||
|
||||
Do not commit the age identity. Only public ` + "`age1...`" + ` recipients belong in
|
||||
config; ` + "`AGE-SECRET-KEY-...`" + ` values must stay local or in a password manager.
|
||||
`
|
||||
return os.WriteFile(path, []byte(body), 0o600)
|
||||
}
|
||||
367
internal/backup/backup_test.go
Normal file
367
internal/backup/backup_test.go
Normal file
@ -0,0 +1,367 @@
|
||||
//nolint:wsl_v5 // Tests stay compact around setup/action/assert blocks.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPushSnapshotAndVerify(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
|
||||
shard, err := NewJSONLShard("gmail", "messages", "acct", "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{
|
||||
{"id": "m1", "raw": "private email body"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("NewJSONLShard: %v", err)
|
||||
}
|
||||
result, err := PushSnapshot(ctx, Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{"acct"},
|
||||
Counts: map[string]int{"gmail.messages": 1},
|
||||
Shards: []PlainShard{shard},
|
||||
}, Options{ConfigPath: config, Push: false})
|
||||
if err != nil {
|
||||
t.Fatalf("PushSnapshot: %v", err)
|
||||
}
|
||||
if !result.Changed || result.Shards != 1 || result.Counts["gmail.messages"] != 1 {
|
||||
t.Fatalf("unexpected push result: %+v", result)
|
||||
}
|
||||
|
||||
ciphertext, err := os.ReadFile(filepath.Join(repo, "data", "gmail", "acct", "messages", "2026", "04", "part-0001.jsonl.gz.age"))
|
||||
if err != nil {
|
||||
t.Fatalf("read ciphertext: %v", err)
|
||||
}
|
||||
if strings.Contains(string(ciphertext), "private email body") {
|
||||
t.Fatal("encrypted shard contains plaintext")
|
||||
}
|
||||
|
||||
verify, err := Verify(ctx, Options{ConfigPath: config})
|
||||
if err != nil {
|
||||
t.Fatalf("Verify: %v", err)
|
||||
}
|
||||
if verify.Shards != 1 || verify.Counts["gmail.messages"] != 1 {
|
||||
t.Fatalf("unexpected verify result: %+v", verify)
|
||||
}
|
||||
|
||||
status, statusRepo, err := Status(ctx, Options{ConfigPath: config})
|
||||
if err != nil {
|
||||
t.Fatalf("Status: %v", err)
|
||||
}
|
||||
if statusRepo != repo || !status.Encrypted || status.Counts["gmail.messages"] != 1 {
|
||||
t.Fatalf("unexpected status repo=%s manifest=%+v", statusRepo, status)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInitCreatesPrivateIdentityAndConfig(t *testing.T) {
|
||||
_, _, config, identity := initTestBackup(t)
|
||||
|
||||
for _, path := range []string{config, identity} {
|
||||
info, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatalf("stat %s: %v", path, err)
|
||||
}
|
||||
if got := info.Mode().Perm(); got != 0o600 {
|
||||
t.Fatalf("%s mode = %v, want 0600", path, got)
|
||||
}
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(identity)
|
||||
if err != nil {
|
||||
t.Fatalf("read identity: %v", err)
|
||||
}
|
||||
if !strings.HasPrefix(strings.TrimSpace(string(data)), "AGE-SECRET-KEY-") {
|
||||
t.Fatalf("identity does not look like an age secret key")
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestDoesNotContainPayloadPlaintext(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
shard := mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{
|
||||
"id": "msg-plain-marker",
|
||||
"subject": "very secret subject marker",
|
||||
"raw": "private raw mime marker",
|
||||
}})
|
||||
|
||||
if _, err := PushSnapshot(ctx, Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{"acct"},
|
||||
Counts: map[string]int{"gmail.messages": 1},
|
||||
Shards: []PlainShard{shard},
|
||||
}, Options{ConfigPath: config, Push: false}); err != nil {
|
||||
t.Fatalf("PushSnapshot: %v", err)
|
||||
}
|
||||
|
||||
for _, name := range []string{"manifest.json", "README.md"} {
|
||||
data, err := os.ReadFile(filepath.Join(repo, name))
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", name, err)
|
||||
}
|
||||
text := string(data)
|
||||
for _, marker := range []string{"msg-plain-marker", "very secret subject marker", "private raw mime marker"} {
|
||||
if strings.Contains(text, marker) {
|
||||
t.Fatalf("%s contains private payload marker %q", name, marker)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyDetectsTamperedCiphertext(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
|
||||
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}}))
|
||||
|
||||
path := filepath.Join(repo, filepath.FromSlash(shardPath))
|
||||
ciphertext, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read ciphertext: %v", err)
|
||||
}
|
||||
ciphertext[len(ciphertext)-1] ^= 0xff
|
||||
if err := os.WriteFile(path, ciphertext, 0o600); err != nil {
|
||||
t.Fatalf("write tampered ciphertext: %v", err)
|
||||
}
|
||||
|
||||
if _, err := Verify(ctx, Options{ConfigPath: config}); err == nil {
|
||||
t.Fatal("expected verify to reject tampered ciphertext")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyDetectsManifestHashMismatch(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{"id": "m1", "raw": "body"}}))
|
||||
|
||||
manifest := readTestManifest(t, repo)
|
||||
manifest.Shards[0].SHA256 = strings.Repeat("0", 64)
|
||||
writeTestManifest(t, repo, manifest)
|
||||
|
||||
_, err := Verify(ctx, Options{ConfigPath: config})
|
||||
if err == nil || !strings.Contains(err.Error(), "hash mismatch") {
|
||||
t.Fatalf("Verify error = %v, want hash mismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyDetectsManifestRowCountMismatch(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{"id": "m1", "raw": "body"}}))
|
||||
|
||||
manifest := readTestManifest(t, repo)
|
||||
manifest.Shards[0].Rows = 2
|
||||
writeTestManifest(t, repo, manifest)
|
||||
|
||||
_, err := Verify(ctx, Options{ConfigPath: config})
|
||||
if err == nil || !strings.Contains(err.Error(), "row count mismatch") {
|
||||
t.Fatalf("Verify error = %v, want row count mismatch", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushReusesEncryptedShardWhenPlaintextAndRecipientsMatch(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
|
||||
shard := mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}})
|
||||
|
||||
first := pushSingleShard(t, ctx, config, shard)
|
||||
firstCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
|
||||
second := pushSingleShard(t, ctx, config, shard)
|
||||
secondCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
|
||||
|
||||
if !first.Changed {
|
||||
t.Fatalf("first push changed = false, want true")
|
||||
}
|
||||
if second.Changed {
|
||||
t.Fatalf("second push changed = true, want false")
|
||||
}
|
||||
if string(firstCiphertext) != string(secondCiphertext) {
|
||||
t.Fatalf("ciphertext changed even though plaintext and recipients matched")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushReencryptsShardWhenRecipientChanges(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
|
||||
shard := mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}})
|
||||
pushSingleShard(t, ctx, config, shard)
|
||||
firstCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
|
||||
|
||||
secondIdentity := filepath.Join(t.TempDir(), "age.key")
|
||||
secondRecipient, err := EnsureIdentity(secondIdentity)
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureIdentity second: %v", err)
|
||||
}
|
||||
if _, err := PushSnapshot(ctx, Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{"acct"},
|
||||
Counts: map[string]int{"gmail.messages": 1},
|
||||
Shards: []PlainShard{shard},
|
||||
}, Options{ConfigPath: config, Identity: secondIdentity, Recipients: []string{secondRecipient}, Push: false}); err != nil {
|
||||
t.Fatalf("PushSnapshot second recipient: %v", err)
|
||||
}
|
||||
secondCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
|
||||
if string(firstCiphertext) == string(secondCiphertext) {
|
||||
t.Fatal("ciphertext did not change after recipient rotation")
|
||||
}
|
||||
if _, err := Verify(ctx, Options{ConfigPath: config, Identity: secondIdentity}); err != nil {
|
||||
t.Fatalf("Verify with rotated identity: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPushRemovesStaleEncryptedShards(t *testing.T) {
|
||||
ctx, repo, config, _ := initTestBackup(t)
|
||||
oldPath := "data/gmail/acct/messages/2026/03/part-0001.jsonl.gz.age"
|
||||
keepPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
|
||||
oldShard := mustGmailMessageShard(t, oldPath, []map[string]string{{"id": "old"}})
|
||||
keepShard := mustGmailMessageShard(t, keepPath, []map[string]string{{"id": "keep"}})
|
||||
|
||||
if _, err := PushSnapshot(ctx, Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{"acct"},
|
||||
Counts: map[string]int{"gmail.messages": 2},
|
||||
Shards: []PlainShard{oldShard, keepShard},
|
||||
}, Options{ConfigPath: config, Push: false}); err != nil {
|
||||
t.Fatalf("initial PushSnapshot: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(oldPath))); err != nil {
|
||||
t.Fatalf("old shard should exist before pruning: %v", err)
|
||||
}
|
||||
|
||||
pushSingleShard(t, ctx, config, keepShard)
|
||||
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(oldPath))); !os.IsNotExist(err) {
|
||||
t.Fatalf("old shard still exists after pruning: %v", err)
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(keepPath))); err != nil {
|
||||
t.Fatalf("kept shard missing after pruning: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRejectsInvalidShardPaths(t *testing.T) {
|
||||
_, _, config, _ := initTestBackup(t)
|
||||
for _, rel := range []string{
|
||||
"../nope.age",
|
||||
"/tmp/nope.age",
|
||||
"manifest.age",
|
||||
"data/gmail/acct/plain.jsonl",
|
||||
"data/../nope.age",
|
||||
} {
|
||||
t.Run(rel, func(t *testing.T) {
|
||||
shard := mustGmailMessageShard(t, rel, []map[string]string{{"id": "m1"}})
|
||||
_, err := PushSnapshot(context.Background(), Snapshot{Shards: []PlainShard{shard}}, Options{
|
||||
ConfigPath: config,
|
||||
Push: false,
|
||||
})
|
||||
if err == nil {
|
||||
t.Fatal("expected invalid shard path error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncryptDecryptRoundTripMultipleRecipients(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
firstIdentity := filepath.Join(dir, "first.age")
|
||||
firstRecipient, err := EnsureIdentity(firstIdentity)
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureIdentity first: %v", err)
|
||||
}
|
||||
secondIdentity := filepath.Join(dir, "second.age")
|
||||
secondRecipient, err := EnsureIdentity(secondIdentity)
|
||||
if err != nil {
|
||||
t.Fatalf("EnsureIdentity second: %v", err)
|
||||
}
|
||||
|
||||
encrypted, hash, err := encryptShard([]byte("secret jsonl\n"), []string{firstRecipient, secondRecipient})
|
||||
if err != nil {
|
||||
t.Fatalf("encryptShard: %v", err)
|
||||
}
|
||||
if hash != sha256Hex([]byte("secret jsonl\n")) {
|
||||
t.Fatalf("hash = %s, want plaintext sha256", hash)
|
||||
}
|
||||
for _, identity := range []string{firstIdentity, secondIdentity} {
|
||||
plaintext, err := decryptShard(encrypted, identity)
|
||||
if err != nil {
|
||||
t.Fatalf("decryptShard %s: %v", identity, err)
|
||||
}
|
||||
if string(plaintext) != "secret jsonl\n" {
|
||||
t.Fatalf("plaintext = %q", plaintext)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func initTestBackup(t *testing.T) (context.Context, string, string, string) {
|
||||
t.Helper()
|
||||
ctx := context.Background()
|
||||
dir := t.TempDir()
|
||||
repo := filepath.Join(dir, "repo")
|
||||
identity := filepath.Join(dir, "age.key")
|
||||
config := filepath.Join(dir, "backup.json")
|
||||
|
||||
cfg, recipient, err := Init(ctx, Options{
|
||||
ConfigPath: config,
|
||||
Repo: repo,
|
||||
Identity: identity,
|
||||
Push: false,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Init: %v", err)
|
||||
}
|
||||
if cfg.Repo != repo || !strings.HasPrefix(recipient, "age1") {
|
||||
t.Fatalf("unexpected init cfg=%+v recipient=%q", cfg, recipient)
|
||||
}
|
||||
return ctx, repo, config, identity
|
||||
}
|
||||
|
||||
func mustGmailMessageShard(t *testing.T, rel string, rows any) PlainShard {
|
||||
t.Helper()
|
||||
shard, err := NewJSONLShard("gmail", "messages", "acct", rel, rows)
|
||||
if err != nil {
|
||||
t.Fatalf("NewJSONLShard: %v", err)
|
||||
}
|
||||
return shard
|
||||
}
|
||||
|
||||
func pushSingleShard(t *testing.T, ctx context.Context, config string, shard PlainShard) Result {
|
||||
t.Helper()
|
||||
result, err := PushSnapshot(ctx, Snapshot{
|
||||
Services: []string{shard.Service},
|
||||
Accounts: []string{shard.Account},
|
||||
Counts: map[string]int{shard.Service + "." + shard.Kind: shard.Rows},
|
||||
Shards: []PlainShard{shard},
|
||||
}, Options{ConfigPath: config, Push: false})
|
||||
if err != nil {
|
||||
t.Fatalf("PushSnapshot: %v", err)
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
func readTestManifest(t *testing.T, repo string) Manifest {
|
||||
t.Helper()
|
||||
data := readFile(t, filepath.Join(repo, "manifest.json"))
|
||||
var manifest Manifest
|
||||
if err := json.Unmarshal(data, &manifest); err != nil {
|
||||
t.Fatalf("unmarshal manifest: %v", err)
|
||||
}
|
||||
return manifest
|
||||
}
|
||||
|
||||
func writeTestManifest(t *testing.T, repo string, manifest Manifest) {
|
||||
t.Helper()
|
||||
data, err := json.MarshalIndent(manifest, "", " ")
|
||||
if err != nil {
|
||||
t.Fatalf("marshal manifest: %v", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
if err := os.WriteFile(filepath.Join(repo, "manifest.json"), data, 0o600); err != nil {
|
||||
t.Fatalf("write manifest: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func readFile(t *testing.T, path string) []byte {
|
||||
t.Helper()
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatalf("read %s: %v", path, err)
|
||||
}
|
||||
return data
|
||||
}
|
||||
117
internal/backup/config.go
Normal file
117
internal/backup/config.go
Normal file
@ -0,0 +1,117 @@
|
||||
//nolint:wrapcheck,wsl_v5 // Backup config errors are surfaced directly to the CLI.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const (
|
||||
defaultRemote = "https://github.com/steipete/backup-gog.git"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Repo string `json:"repo"`
|
||||
Remote string `json:"remote"`
|
||||
Identity string `json:"identity"`
|
||||
Recipients []string `json:"recipients"`
|
||||
}
|
||||
|
||||
type Options struct {
|
||||
ConfigPath string
|
||||
Repo string
|
||||
Remote string
|
||||
Identity string
|
||||
Recipients []string
|
||||
Push bool
|
||||
}
|
||||
|
||||
func DefaultConfig() Config {
|
||||
return Config{
|
||||
Repo: "~/Projects/backup-gog",
|
||||
Remote: defaultRemote,
|
||||
Identity: "~/.gog/age.key",
|
||||
}
|
||||
}
|
||||
|
||||
func DefaultConfigPath() string {
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return "backup.json"
|
||||
}
|
||||
return filepath.Join(home, ".gog", "backup.json")
|
||||
}
|
||||
|
||||
func LoadConfig(path string) (Config, error) {
|
||||
if strings.TrimSpace(path) == "" {
|
||||
path = DefaultConfigPath()
|
||||
}
|
||||
cfg := DefaultConfig()
|
||||
data, err := os.ReadFile(expandHome(path))
|
||||
if err != nil {
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return cfg, nil
|
||||
}
|
||||
return Config{}, err
|
||||
}
|
||||
if err := json.Unmarshal(data, &cfg); err != nil {
|
||||
return Config{}, fmt.Errorf("read backup config: %w", err)
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func SaveConfig(path string, cfg Config) error {
|
||||
if strings.TrimSpace(path) == "" {
|
||||
path = DefaultConfigPath()
|
||||
}
|
||||
path = expandHome(path)
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
data, err := json.MarshalIndent(cfg, "", " ")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data = append(data, '\n')
|
||||
return os.WriteFile(path, data, 0o600)
|
||||
}
|
||||
|
||||
func ResolveOptions(opts Options) (Config, error) {
|
||||
cfg, err := LoadConfig(opts.ConfigPath)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if strings.TrimSpace(opts.Repo) != "" {
|
||||
cfg.Repo = opts.Repo
|
||||
}
|
||||
if strings.TrimSpace(opts.Remote) != "" {
|
||||
cfg.Remote = opts.Remote
|
||||
}
|
||||
if strings.TrimSpace(opts.Identity) != "" {
|
||||
cfg.Identity = opts.Identity
|
||||
}
|
||||
if len(opts.Recipients) > 0 {
|
||||
cfg.Recipients = opts.Recipients
|
||||
}
|
||||
cfg.Repo = expandHome(cfg.Repo)
|
||||
cfg.Identity = expandHome(cfg.Identity)
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func expandHome(path string) string {
|
||||
if path == "~" {
|
||||
if home, err := os.UserHomeDir(); err == nil {
|
||||
return home
|
||||
}
|
||||
}
|
||||
if after, ok := strings.CutPrefix(path, "~/"); ok {
|
||||
if home, err := os.UserHomeDir(); err == nil {
|
||||
return filepath.Join(home, after)
|
||||
}
|
||||
}
|
||||
return path
|
||||
}
|
||||
142
internal/backup/crypto.go
Normal file
142
internal/backup/crypto.go
Normal file
@ -0,0 +1,142 @@
|
||||
//nolint:err113,govet,wrapcheck,wsl_v5 // Crypto helpers keep age/gzip errors intact for diagnosis.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"filippo.io/age"
|
||||
)
|
||||
|
||||
func EnsureIdentity(path string) (string, error) {
|
||||
path = expandHome(path)
|
||||
if data, err := os.ReadFile(path); err == nil { // #nosec G304 -- path is the configured local age identity file.
|
||||
identity, err := parseIdentity(data)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return identity.Recipient().String(), nil
|
||||
} else if !os.IsNotExist(err) {
|
||||
return "", err
|
||||
}
|
||||
|
||||
identity, err := age.GenerateX25519Identity()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
|
||||
return "", err
|
||||
}
|
||||
data := []byte(identity.String() + "\n")
|
||||
if err := os.WriteFile(path, data, 0o600); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return identity.Recipient().String(), nil
|
||||
}
|
||||
|
||||
func RecipientFromIdentity(path string) (string, error) {
|
||||
data, err := os.ReadFile(expandHome(path))
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
identity, err := parseIdentity(data)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return identity.Recipient().String(), nil
|
||||
}
|
||||
|
||||
func encryptShard(plaintext []byte, recipientStrings []string) ([]byte, string, error) {
|
||||
recipients, err := parseRecipients(recipientStrings)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
var compressed bytes.Buffer
|
||||
gz := gzip.NewWriter(&compressed)
|
||||
gz.ModTime = time.Unix(0, 0).UTC()
|
||||
_, _ = gz.Write(plaintext)
|
||||
_ = gz.Close()
|
||||
|
||||
var encrypted bytes.Buffer
|
||||
w, err := age.Encrypt(&encrypted, recipients...)
|
||||
if err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
_, _ = w.Write(compressed.Bytes())
|
||||
if err := w.Close(); err != nil {
|
||||
return nil, "", err
|
||||
}
|
||||
return encrypted.Bytes(), sha256Hex(plaintext), nil
|
||||
}
|
||||
|
||||
func decryptShard(ciphertext []byte, identityPath string) ([]byte, error) {
|
||||
data, err := os.ReadFile(expandHome(identityPath)) // #nosec G304 -- path is the configured local age identity file.
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
identity, err := parseIdentity(data)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
gz, err := gzip.NewReader(r)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer func() { _ = gz.Close() }()
|
||||
plaintext, err := io.ReadAll(gz)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return plaintext, nil
|
||||
}
|
||||
|
||||
func parseRecipients(values []string) ([]age.Recipient, error) {
|
||||
var out []age.Recipient
|
||||
for _, value := range values {
|
||||
value = strings.TrimSpace(value)
|
||||
if value == "" {
|
||||
continue
|
||||
}
|
||||
recipient, err := age.ParseX25519Recipient(value)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse age recipient: %w", err)
|
||||
}
|
||||
out = append(out, recipient)
|
||||
}
|
||||
if len(out) == 0 {
|
||||
return nil, fmt.Errorf("at least one age recipient is required")
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func parseIdentity(data []byte) (*age.X25519Identity, error) {
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
line = strings.TrimSpace(line)
|
||||
if line == "" || strings.HasPrefix(line, "#") {
|
||||
continue
|
||||
}
|
||||
identity, err := age.ParseX25519Identity(line)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse age identity: %w", err)
|
||||
}
|
||||
return identity, nil
|
||||
}
|
||||
return nil, fmt.Errorf("age identity file is empty")
|
||||
}
|
||||
|
||||
func sha256Hex(data []byte) string {
|
||||
sum := sha256.Sum256(data)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
92
internal/backup/git.go
Normal file
92
internal/backup/git.go
Normal file
@ -0,0 +1,92 @@
|
||||
//nolint:err113,wrapcheck,wsl_v5 // Git helper returns command-context errors.
|
||||
package backup
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func ensureRepo(ctx context.Context, cfg Config) error {
|
||||
if strings.TrimSpace(cfg.Repo) == "" {
|
||||
return fmt.Errorf("backup repo path is required")
|
||||
}
|
||||
if _, err := os.Stat(filepath.Join(cfg.Repo, ".git")); err == nil {
|
||||
pullErr := git(ctx, cfg.Repo, "pull", "--rebase")
|
||||
if pullErr != nil {
|
||||
hasHead := git(ctx, cfg.Repo, "rev-parse", "--verify", "HEAD") == nil
|
||||
if !hasHead {
|
||||
return nil
|
||||
}
|
||||
if strings.Contains(pullErr.Error(), "no tracking information") ||
|
||||
strings.Contains(pullErr.Error(), "No remote repository specified") ||
|
||||
strings.Contains(pullErr.Error(), "no such ref was fetched") {
|
||||
return nil
|
||||
}
|
||||
return pullErr
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if strings.TrimSpace(cfg.Remote) != "" {
|
||||
if err := os.MkdirAll(filepath.Dir(cfg.Repo), 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := git(ctx, "", "clone", cfg.Remote, cfg.Repo); err == nil {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
if err := os.MkdirAll(cfg.Repo, 0o700); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := git(ctx, cfg.Repo, "init"); err != nil {
|
||||
return err
|
||||
}
|
||||
if strings.TrimSpace(cfg.Remote) != "" {
|
||||
if err := git(ctx, cfg.Repo, "remote", "add", "origin", cfg.Remote); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func commitAndPush(ctx context.Context, cfg Config, message string, push bool) (bool, error) {
|
||||
if err := git(ctx, cfg.Repo, "add", "."); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if err := git(ctx, cfg.Repo, "diff", "--cached", "--quiet"); err == nil {
|
||||
return false, nil
|
||||
}
|
||||
if err := git(ctx, cfg.Repo, "commit", "-m", message); err != nil {
|
||||
return false, err
|
||||
}
|
||||
if push {
|
||||
if err := git(ctx, cfg.Repo, "push", "-u", "origin", "HEAD"); err != nil {
|
||||
return true, err
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func git(ctx context.Context, dir string, args ...string) error {
|
||||
cmd := exec.CommandContext(ctx, "git", args...) // #nosec G204 -- callers pass fixed git subcommands plus configured repo paths.
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(),
|
||||
"GIT_AUTHOR_NAME=gog",
|
||||
"GIT_AUTHOR_EMAIL=gog@example.invalid",
|
||||
"GIT_COMMITTER_NAME=gog",
|
||||
"GIT_COMMITTER_EMAIL=gog@example.invalid",
|
||||
)
|
||||
var stderr bytes.Buffer
|
||||
cmd.Stderr = &stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
if stderr.Len() > 0 {
|
||||
return fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
|
||||
}
|
||||
return fmt.Errorf("git %s: %w", strings.Join(args, " "), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
471
internal/cmd/backup.go
Normal file
471
internal/cmd/backup.go
Normal file
@ -0,0 +1,471 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"google.golang.org/api/gmail/v1"
|
||||
|
||||
"github.com/steipete/gogcli/internal/backup"
|
||||
"github.com/steipete/gogcli/internal/outfmt"
|
||||
"github.com/steipete/gogcli/internal/ui"
|
||||
)
|
||||
|
||||
type BackupCmd struct {
|
||||
Init BackupInitCmd `cmd:"" name:"init" help:"Initialize encrypted backup config and repository"`
|
||||
Push BackupPushCmd `cmd:"" name:"push" help:"Export services into encrypted backup shards"`
|
||||
Status BackupStatusCmd `cmd:"" name:"status" help:"Inspect backup manifest without decrypting shards"`
|
||||
Verify BackupVerifyCmd `cmd:"" name:"verify" help:"Decrypt and verify all backup shards"`
|
||||
Gmail BackupGmailCmd `cmd:"" name:"gmail" help:"Gmail backup operations"`
|
||||
}
|
||||
|
||||
type BackupGmailCmd struct {
|
||||
Push BackupGmailPushCmd `cmd:"" name:"push" help:"Export Gmail into encrypted backup shards"`
|
||||
}
|
||||
|
||||
type backupFlags struct {
|
||||
Config string `name:"config" help:"Backup config path" default:""`
|
||||
Repo string `name:"repo" help:"Local backup repository path"`
|
||||
Remote string `name:"remote" help:"Backup Git remote URL"`
|
||||
Identity string `name:"identity" help:"Local age identity path"`
|
||||
Recipients []string `name:"recipient" help:"Public age recipient (repeatable)"`
|
||||
NoPush bool `name:"no-push" help:"Commit locally but do not push to the remote"`
|
||||
}
|
||||
|
||||
func (f backupFlags) options() backup.Options {
|
||||
return backup.Options{
|
||||
ConfigPath: f.Config,
|
||||
Repo: f.Repo,
|
||||
Remote: f.Remote,
|
||||
Identity: f.Identity,
|
||||
Recipients: f.Recipients,
|
||||
Push: !f.NoPush,
|
||||
}
|
||||
}
|
||||
|
||||
type BackupInitCmd struct {
|
||||
backupFlags
|
||||
}
|
||||
|
||||
func (c *BackupInitCmd) Run(ctx context.Context) error {
|
||||
cfg, recipient, err := backup.Init(ctx, c.options())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if outfmt.IsJSON(ctx) {
|
||||
return outfmt.WriteJSON(ctx, os.Stdout, map[string]any{
|
||||
"repo": cfg.Repo,
|
||||
"remote": cfg.Remote,
|
||||
"identity": cfg.Identity,
|
||||
"recipient": recipient,
|
||||
})
|
||||
}
|
||||
u := ui.FromContext(ctx)
|
||||
u.Out().Printf("repo\t%s", cfg.Repo)
|
||||
u.Out().Printf("remote\t%s", cfg.Remote)
|
||||
u.Out().Printf("identity\t%s", cfg.Identity)
|
||||
u.Out().Printf("recipient\t%s", recipient)
|
||||
return nil
|
||||
}
|
||||
|
||||
type BackupPushCmd struct {
|
||||
backupFlags
|
||||
Services string `name:"services" help:"Comma-separated services to back up (currently: gmail)" default:"gmail"`
|
||||
Query string `name:"query" help:"Gmail query for bounded/test backups"`
|
||||
Max int64 `name:"max" aliases:"limit" help:"Max Gmail messages to export; 0 means all" default:"0"`
|
||||
IncludeSpamTrash bool `name:"include-spam-trash" help:"Include Gmail spam and trash" default:"true"`
|
||||
ShardMaxRows int `name:"shard-max-rows" help:"Max Gmail messages per encrypted shard" default:"1000"`
|
||||
}
|
||||
|
||||
func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
|
||||
services := splitCSV(c.Services)
|
||||
if len(services) == 0 {
|
||||
return usage("at least one --services value is required")
|
||||
}
|
||||
var snapshots []backup.Snapshot
|
||||
for _, service := range services {
|
||||
switch strings.ToLower(strings.TrimSpace(service)) {
|
||||
case "gmail":
|
||||
snapshot, err := buildGmailBackupSnapshot(ctx, flags, gmailBackupOptions{
|
||||
Query: c.Query,
|
||||
Max: c.Max,
|
||||
IncludeSpamTrash: c.IncludeSpamTrash,
|
||||
ShardMaxRows: c.ShardMaxRows,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
snapshots = append(snapshots, snapshot)
|
||||
default:
|
||||
return fmt.Errorf("unsupported backup service %q (currently: gmail)", service)
|
||||
}
|
||||
}
|
||||
result, err := backup.PushSnapshot(ctx, mergeBackupSnapshots(snapshots...), c.options())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return writeBackupResult(ctx, result)
|
||||
}
|
||||
|
||||
type BackupGmailPushCmd struct {
|
||||
backupFlags
|
||||
Query string `name:"query" help:"Gmail query for bounded/test backups"`
|
||||
Max int64 `name:"max" aliases:"limit" help:"Max Gmail messages to export; 0 means all" default:"0"`
|
||||
IncludeSpamTrash bool `name:"include-spam-trash" help:"Include spam and trash" default:"true"`
|
||||
ShardMaxRows int `name:"shard-max-rows" help:"Max messages per encrypted shard" default:"1000"`
|
||||
}
|
||||
|
||||
func (c *BackupGmailPushCmd) Run(ctx context.Context, flags *RootFlags) error {
|
||||
snapshot, err := buildGmailBackupSnapshot(ctx, flags, gmailBackupOptions{
|
||||
Query: c.Query,
|
||||
Max: c.Max,
|
||||
IncludeSpamTrash: c.IncludeSpamTrash,
|
||||
ShardMaxRows: c.ShardMaxRows,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result, err := backup.PushSnapshot(ctx, snapshot, c.options())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return writeBackupResult(ctx, result)
|
||||
}
|
||||
|
||||
type BackupStatusCmd struct {
|
||||
backupFlags
|
||||
}
|
||||
|
||||
func (c *BackupStatusCmd) Run(ctx context.Context) error {
|
||||
manifest, repo, err := backup.Status(ctx, c.options())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if outfmt.IsJSON(ctx) {
|
||||
return outfmt.WriteJSON(ctx, os.Stdout, map[string]any{"repo": repo, "manifest": manifest})
|
||||
}
|
||||
u := ui.FromContext(ctx)
|
||||
u.Out().Printf("repo\t%s", repo)
|
||||
u.Out().Printf("encrypted\t%t", manifest.Encrypted)
|
||||
u.Out().Printf("exported\t%s", manifest.Exported.Format(time.RFC3339))
|
||||
u.Out().Printf("services\t%s", strings.Join(manifest.Services, ","))
|
||||
u.Out().Printf("accounts\t%s", strings.Join(manifest.Accounts, ","))
|
||||
u.Out().Printf("shards\t%d", len(manifest.Shards))
|
||||
keys := make([]string, 0, len(manifest.Counts))
|
||||
for key := range manifest.Counts {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
u.Out().Printf("count.%s\t%d", key, manifest.Counts[key])
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type BackupVerifyCmd struct {
|
||||
backupFlags
|
||||
}
|
||||
|
||||
func (c *BackupVerifyCmd) Run(ctx context.Context) error {
|
||||
result, err := backup.Verify(ctx, c.options())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return writeBackupResult(ctx, result)
|
||||
}
|
||||
|
||||
type gmailBackupOptions struct {
|
||||
Query string
|
||||
Max int64
|
||||
IncludeSpamTrash bool
|
||||
ShardMaxRows int
|
||||
}
|
||||
|
||||
type gmailBackupMessage struct {
|
||||
ID string `json:"id"`
|
||||
ThreadID string `json:"threadId,omitempty"`
|
||||
HistoryID string `json:"historyId,omitempty"`
|
||||
InternalDate int64 `json:"internalDate,omitempty"`
|
||||
LabelIDs []string `json:"labelIds,omitempty"`
|
||||
SizeEstimate int64 `json:"sizeEstimate,omitempty"`
|
||||
Raw string `json:"raw"`
|
||||
}
|
||||
|
||||
type gmailBackupLabel struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
Type string `json:"type,omitempty"`
|
||||
MessageListVisibility string `json:"messageListVisibility,omitempty"`
|
||||
LabelListVisibility string `json:"labelListVisibility,omitempty"`
|
||||
MessagesTotal int64 `json:"messagesTotal,omitempty"`
|
||||
MessagesUnread int64 `json:"messagesUnread,omitempty"`
|
||||
ThreadsTotal int64 `json:"threadsTotal,omitempty"`
|
||||
ThreadsUnread int64 `json:"threadsUnread,omitempty"`
|
||||
}
|
||||
|
||||
func buildGmailBackupSnapshot(ctx context.Context, flags *RootFlags, opts gmailBackupOptions) (backup.Snapshot, error) {
|
||||
if opts.ShardMaxRows <= 0 {
|
||||
opts.ShardMaxRows = 1000
|
||||
}
|
||||
account, err := requireAccount(flags)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
svc, err := newGmailService(ctx, account)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
accountHash := backupAccountHash(account)
|
||||
labels, err := fetchGmailBackupLabels(ctx, svc)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
messages, err := fetchGmailBackupMessages(ctx, svc, opts)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
shards := make([]backup.PlainShard, 0, 1)
|
||||
labelShard, err := backup.NewJSONLShard("gmail", "labels", accountHash, fmt.Sprintf("data/gmail/%s/labels.jsonl.gz.age", accountHash), labels)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
shards = append(shards, labelShard)
|
||||
messageShards, err := buildGmailMessageShards(accountHash, messages, opts.ShardMaxRows)
|
||||
if err != nil {
|
||||
return backup.Snapshot{}, err
|
||||
}
|
||||
shards = append(shards, messageShards...)
|
||||
return backup.Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{accountHash},
|
||||
Counts: map[string]int{
|
||||
"gmail.labels": len(labels),
|
||||
"gmail.messages": len(messages),
|
||||
},
|
||||
Shards: shards,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func fetchGmailBackupLabels(ctx context.Context, svc *gmail.Service) ([]gmailBackupLabel, error) {
|
||||
resp, err := svc.Users.Labels.List("me").Context(ctx).Do()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out := make([]gmailBackupLabel, 0, len(resp.Labels))
|
||||
for _, label := range resp.Labels {
|
||||
if label == nil {
|
||||
continue
|
||||
}
|
||||
out = append(out, gmailBackupLabel{
|
||||
ID: label.Id,
|
||||
Name: label.Name,
|
||||
Type: label.Type,
|
||||
MessageListVisibility: label.MessageListVisibility,
|
||||
LabelListVisibility: label.LabelListVisibility,
|
||||
MessagesTotal: label.MessagesTotal,
|
||||
MessagesUnread: label.MessagesUnread,
|
||||
ThreadsTotal: label.ThreadsTotal,
|
||||
ThreadsUnread: label.ThreadsUnread,
|
||||
})
|
||||
}
|
||||
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func fetchGmailBackupMessages(ctx context.Context, svc *gmail.Service, opts gmailBackupOptions) ([]gmailBackupMessage, error) {
|
||||
ids, err := listGmailBackupMessageIDs(ctx, svc, opts)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
const maxConcurrency = 8
|
||||
sem := make(chan struct{}, maxConcurrency)
|
||||
type result struct {
|
||||
index int
|
||||
msg gmailBackupMessage
|
||||
err error
|
||||
}
|
||||
results := make(chan result, len(ids))
|
||||
var wg sync.WaitGroup
|
||||
for i, id := range ids {
|
||||
wg.Add(1)
|
||||
go func(index int, messageID string) {
|
||||
defer wg.Done()
|
||||
select {
|
||||
case sem <- struct{}{}:
|
||||
defer func() { <-sem }()
|
||||
case <-ctx.Done():
|
||||
results <- result{index: index, err: ctx.Err()}
|
||||
return
|
||||
}
|
||||
msg, err := svc.Users.Messages.Get("me", messageID).
|
||||
Format(gmailFormatRaw).
|
||||
Fields("id,threadId,historyId,internalDate,labelIds,sizeEstimate,raw").
|
||||
Context(ctx).
|
||||
Do()
|
||||
if err != nil {
|
||||
results <- result{index: index, err: fmt.Errorf("gmail message %s: %w", messageID, err)}
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(msg.Raw) == "" {
|
||||
results <- result{index: index, err: fmt.Errorf("gmail message %s returned empty raw payload", messageID)}
|
||||
return
|
||||
}
|
||||
results <- result{index: index, msg: gmailBackupMessage{
|
||||
ID: msg.Id,
|
||||
ThreadID: msg.ThreadId,
|
||||
HistoryID: formatHistoryID(msg.HistoryId),
|
||||
InternalDate: msg.InternalDate,
|
||||
LabelIDs: append([]string(nil), msg.LabelIds...),
|
||||
SizeEstimate: msg.SizeEstimate,
|
||||
Raw: msg.Raw,
|
||||
}}
|
||||
}(i, id)
|
||||
}
|
||||
go func() {
|
||||
wg.Wait()
|
||||
close(results)
|
||||
}()
|
||||
ordered := make([]gmailBackupMessage, len(ids))
|
||||
var firstErr error
|
||||
for res := range results {
|
||||
if res.err != nil && firstErr == nil {
|
||||
firstErr = res.err
|
||||
}
|
||||
ordered[res.index] = res.msg
|
||||
}
|
||||
if firstErr != nil {
|
||||
return nil, firstErr
|
||||
}
|
||||
return ordered, nil
|
||||
}
|
||||
|
||||
func listGmailBackupMessageIDs(ctx context.Context, svc *gmail.Service, opts gmailBackupOptions) ([]string, error) {
|
||||
var ids []string
|
||||
pageToken := ""
|
||||
for {
|
||||
maxResults := int64(500)
|
||||
if opts.Max > 0 {
|
||||
remaining := opts.Max - int64(len(ids))
|
||||
if remaining <= 0 {
|
||||
break
|
||||
}
|
||||
if remaining < maxResults {
|
||||
maxResults = remaining
|
||||
}
|
||||
}
|
||||
call := svc.Users.Messages.List("me").
|
||||
MaxResults(maxResults).
|
||||
IncludeSpamTrash(opts.IncludeSpamTrash).
|
||||
Fields("messages(id),nextPageToken").
|
||||
Context(ctx)
|
||||
if strings.TrimSpace(opts.Query) != "" {
|
||||
call = call.Q(opts.Query)
|
||||
}
|
||||
if pageToken != "" {
|
||||
call = call.PageToken(pageToken)
|
||||
}
|
||||
resp, err := call.Do()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
for _, message := range resp.Messages {
|
||||
if message != nil && strings.TrimSpace(message.Id) != "" {
|
||||
ids = append(ids, message.Id)
|
||||
}
|
||||
}
|
||||
if resp.NextPageToken == "" {
|
||||
break
|
||||
}
|
||||
pageToken = resp.NextPageToken
|
||||
}
|
||||
return ids, nil
|
||||
}
|
||||
|
||||
func buildGmailMessageShards(accountHash string, messages []gmailBackupMessage, shardMaxRows int) ([]backup.PlainShard, error) {
|
||||
if shardMaxRows <= 0 {
|
||||
shardMaxRows = 1000
|
||||
}
|
||||
buckets := map[string][]gmailBackupMessage{}
|
||||
for _, message := range messages {
|
||||
t := time.UnixMilli(message.InternalDate).UTC()
|
||||
if message.InternalDate <= 0 {
|
||||
t = time.Unix(0, 0).UTC()
|
||||
}
|
||||
key := fmt.Sprintf("%04d/%02d", t.Year(), int(t.Month()))
|
||||
buckets[key] = append(buckets[key], message)
|
||||
}
|
||||
keys := make([]string, 0, len(buckets))
|
||||
for key := range buckets {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
shards := make([]backup.PlainShard, 0, len(keys))
|
||||
for _, key := range keys {
|
||||
values := buckets[key]
|
||||
sort.Slice(values, func(i, j int) bool {
|
||||
if values[i].InternalDate == values[j].InternalDate {
|
||||
return values[i].ID < values[j].ID
|
||||
}
|
||||
return values[i].InternalDate < values[j].InternalDate
|
||||
})
|
||||
for part, start := 1, 0; start < len(values); part, start = part+1, start+shardMaxRows {
|
||||
end := start + shardMaxRows
|
||||
if end > len(values) {
|
||||
end = len(values)
|
||||
}
|
||||
rel := fmt.Sprintf("data/gmail/%s/messages/%s/part-%04d.jsonl.gz.age", accountHash, key, part)
|
||||
shard, err := backup.NewJSONLShard("gmail", "messages", accountHash, rel, values[start:end])
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
shards = append(shards, shard)
|
||||
}
|
||||
}
|
||||
return shards, nil
|
||||
}
|
||||
|
||||
func backupAccountHash(account string) string {
|
||||
sum := sha256.Sum256([]byte(strings.ToLower(strings.TrimSpace(account))))
|
||||
return hex.EncodeToString(sum[:12])
|
||||
}
|
||||
|
||||
func mergeBackupSnapshots(snapshots ...backup.Snapshot) backup.Snapshot {
|
||||
out := backup.Snapshot{Counts: map[string]int{}}
|
||||
for _, snapshot := range snapshots {
|
||||
out.Services = append(out.Services, snapshot.Services...)
|
||||
out.Accounts = append(out.Accounts, snapshot.Accounts...)
|
||||
out.Shards = append(out.Shards, snapshot.Shards...)
|
||||
for key, value := range snapshot.Counts {
|
||||
out.Counts[key] += value
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
func writeBackupResult(ctx context.Context, result backup.Result) error {
|
||||
if outfmt.IsJSON(ctx) {
|
||||
return outfmt.WriteJSON(ctx, os.Stdout, result)
|
||||
}
|
||||
u := ui.FromContext(ctx)
|
||||
u.Out().Printf("repo\t%s", result.Repo)
|
||||
u.Out().Printf("changed\t%t", result.Changed)
|
||||
u.Out().Printf("encrypted\t%t", result.Encrypted)
|
||||
u.Out().Printf("shards\t%d", result.Shards)
|
||||
keys := make([]string, 0, len(result.Counts))
|
||||
for key := range result.Counts {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
for _, key := range keys {
|
||||
u.Out().Printf("count.%s\t%s", key, strconv.Itoa(result.Counts[key]))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
98
internal/cmd/backup_test.go
Normal file
98
internal/cmd/backup_test.go
Normal file
@ -0,0 +1,98 @@
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/steipete/gogcli/internal/backup"
|
||||
)
|
||||
|
||||
func TestBackupAccountHashStableAndOpaque(t *testing.T) {
|
||||
got := backupAccountHash(" User@Example.COM ")
|
||||
want := backupAccountHash("user@example.com")
|
||||
if got != want {
|
||||
t.Fatalf("hash not normalized: got %s want %s", got, want)
|
||||
}
|
||||
if len(got) != 24 {
|
||||
t.Fatalf("hash length = %d, want 24 hex chars", len(got))
|
||||
}
|
||||
if strings.Contains(got, "user") || strings.Contains(got, "example") {
|
||||
t.Fatalf("hash leaks account text: %s", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildGmailMessageShardsBucketsSortsAndChunks(t *testing.T) {
|
||||
accountHash := "accthash"
|
||||
messages := []gmailBackupMessage{
|
||||
{ID: "march-new", InternalDate: mustUnixMilli(t, "2026-03-02T10:00:00Z"), Raw: "raw-3"},
|
||||
{ID: "april-later", InternalDate: mustUnixMilli(t, "2026-04-02T10:00:00Z"), Raw: "raw-2"},
|
||||
{ID: "april-earlier-b", InternalDate: mustUnixMilli(t, "2026-04-01T10:00:00Z"), Raw: "raw-1b"},
|
||||
{ID: "april-earlier-a", InternalDate: mustUnixMilli(t, "2026-04-01T10:00:00Z"), Raw: "raw-1a"},
|
||||
}
|
||||
|
||||
shards, err := buildGmailMessageShards(accountHash, messages, 2)
|
||||
if err != nil {
|
||||
t.Fatalf("buildGmailMessageShards: %v", err)
|
||||
}
|
||||
if len(shards) != 3 {
|
||||
t.Fatalf("len(shards) = %d, want 3", len(shards))
|
||||
}
|
||||
wantPaths := []string{
|
||||
"data/gmail/accthash/messages/2026/03/part-0001.jsonl.gz.age",
|
||||
"data/gmail/accthash/messages/2026/04/part-0001.jsonl.gz.age",
|
||||
"data/gmail/accthash/messages/2026/04/part-0002.jsonl.gz.age",
|
||||
}
|
||||
for i, want := range wantPaths {
|
||||
if shards[i].Path != want {
|
||||
t.Fatalf("shards[%d].Path = %q, want %q", i, shards[i].Path, want)
|
||||
}
|
||||
}
|
||||
if shards[0].Rows != 1 || shards[1].Rows != 2 || shards[2].Rows != 1 {
|
||||
t.Fatalf("unexpected row counts: %d %d %d", shards[0].Rows, shards[1].Rows, shards[2].Rows)
|
||||
}
|
||||
|
||||
var aprilFirst []gmailBackupMessage
|
||||
if err := backup.DecodeJSONL(shards[1].Plaintext, &aprilFirst); err != nil {
|
||||
t.Fatalf("DecodeJSONL: %v", err)
|
||||
}
|
||||
gotIDs := []string{aprilFirst[0].ID, aprilFirst[1].ID}
|
||||
wantIDs := []string{"april-earlier-a", "april-earlier-b"}
|
||||
for i := range wantIDs {
|
||||
if gotIDs[i] != wantIDs[i] {
|
||||
t.Fatalf("april shard IDs = %v, want %v", gotIDs, wantIDs)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMergeBackupSnapshotsKeepsCountsAndShardOrder(t *testing.T) {
|
||||
left := backup.Snapshot{
|
||||
Services: []string{"gmail"},
|
||||
Accounts: []string{"acct1"},
|
||||
Counts: map[string]int{"gmail.messages": 2},
|
||||
Shards: []backup.PlainShard{{Path: "data/gmail/acct1/messages/2026/04/part-0001.jsonl.gz.age"}},
|
||||
}
|
||||
right := backup.Snapshot{
|
||||
Services: []string{"calendar"},
|
||||
Accounts: []string{"acct1"},
|
||||
Counts: map[string]int{"calendar.events": 3},
|
||||
Shards: []backup.PlainShard{{Path: "data/calendar/acct1/events.jsonl.gz.age"}},
|
||||
}
|
||||
|
||||
merged := mergeBackupSnapshots(left, right)
|
||||
if merged.Counts["gmail.messages"] != 2 || merged.Counts["calendar.events"] != 3 {
|
||||
t.Fatalf("unexpected counts: %+v", merged.Counts)
|
||||
}
|
||||
if len(merged.Shards) != 2 || merged.Shards[0].Path != left.Shards[0].Path || merged.Shards[1].Path != right.Shards[0].Path {
|
||||
t.Fatalf("unexpected shard order: %+v", merged.Shards)
|
||||
}
|
||||
}
|
||||
|
||||
func mustUnixMilli(t *testing.T, value string) int64 {
|
||||
t.Helper()
|
||||
parsed, err := time.Parse(time.RFC3339, value)
|
||||
if err != nil {
|
||||
t.Fatalf("parse time %q: %v", value, err)
|
||||
}
|
||||
return parsed.UnixMilli()
|
||||
}
|
||||
@ -64,6 +64,7 @@ type CLI struct {
|
||||
Whoami PeopleMeCmd `cmd:"" name:"whoami" aliases:"who-am-i" help:"Show your profile (alias for 'people me')"`
|
||||
|
||||
Auth AuthCmd `cmd:"" help:"Auth and credentials"`
|
||||
Backup BackupCmd `cmd:"" help:"Encrypted Google account backups"`
|
||||
Groups GroupsCmd `cmd:"" aliases:"group" help:"Google Groups"`
|
||||
Admin AdminCmd `cmd:"" help:"Google Workspace Admin (Directory API) - requires domain-wide delegation"`
|
||||
Drive DriveCmd `cmd:"" aliases:"drv" help:"Google Drive"`
|
||||
|
||||
Loading…
Reference in New Issue
Block a user