feat(backup): add encrypted Google backups

This commit is contained in:
Peter Steinberger 2026-04-27 09:19:06 +01:00
parent 0337dcc572
commit e49e9f45c3
No known key found for this signature in database
14 changed files with 2077 additions and 0 deletions

View File

@ -1,5 +1,10 @@
# Changelog
## Unreleased
### Added
- Backup: add `gog backup` with age-encrypted Git shards, Gmail labels/raw message export, manifest status, full decrypt-and-verify, docs, and security-focused regression coverage.
## 0.13.0 - 2026-04-20
### Highlights

View File

@ -9,6 +9,7 @@ Fast, script-friendly CLI for Gmail, Calendar, Chat, Classroom, Drive, Docs, Sli
- **Gmail** - search threads/messages, send mail, view attachments, manage labels/drafts/filters/delegation/vacation settings, auto-reply once to matching mail, modify single messages, export filters, inspect history, and run Pub/Sub watch webhooks
- **Email tracking** - track opens for `gog gmail send --track` with a small Cloudflare Worker backend
- **Encrypted backups** - export Google account data to age-encrypted Git shards (`gog backup`, Gmail first)
- **Calendar** - list/create/update/delete events, manage invitations, aliases, subscriptions, team calendars, free/busy/conflicts, propose new times, focus/OOO/working-location events, recurrence, and reminders
- **Classroom** - manage courses, roster, coursework/materials, submissions, announcements, topics, invitations, guardians, profiles
- **Chat** - list/find/create spaces, list messages/threads, send messages and DMs, and manage emoji reactions (Workspace-only)
@ -720,6 +721,38 @@ Gmail watch (Pub/Sub push):
- `watch serve --fetch-delay` defaults to `3s` and helps avoid Gmail History indexing races after push delivery.
- `watch serve --exclude-labels` defaults to `SPAM,TRASH`; IDs are case-sensitive.
### Encrypted Backup
```bash
gog backup init --repo ~/Projects/backup-gog --remote https://github.com/steipete/backup-gog.git
gog backup push --services gmail --account you@gmail.com
gog backup status
gog backup verify
```
For a bounded first run:
```bash
gog backup push --services gmail --account you@gmail.com --query 'newer_than:7d' --max 25
```
Backups use age-encrypted JSONL gzip shards under `data/`. `gog` stores the
private age identity locally at `~/.gog/age.key`; GitHub only receives public
`age1...` recipients, `manifest.json`, and encrypted `*.jsonl.gz.age` payloads.
The private `AGE-SECRET-KEY-...` value must stay local or in a password manager.
`manifest.json` is intentionally cleartext for cheap status and verification.
It exposes metadata: export time, service names, account hashes, shard paths,
row counts, encrypted byte sizes, plaintext verification hashes, backup cadence,
and which shards changed. It does not contain email bodies, subjects, senders,
recipients, raw MIME, labels, Drive filenames, contacts, or event titles.
Security boundary: GitHub cannot read Google content without the age identity.
Repository writers can still replace backup contents with different encrypted
data, so keep write access tight and review unexpected backup commits. If the
age identity leaks, rotate recipients and re-encrypt; old Git history may still
contain shards decryptable with the leaked key. See `docs/backup.md`.
### Email Tracking
Track when recipients open your emails:

172
docs/backup.md Normal file
View File

@ -0,0 +1,172 @@
---
summary: "Encrypted Google account backups"
read_when:
- Adding a new gog backup service adapter
- Changing encrypted backup layout, manifest fields, or age identity handling
- Debugging backup-gog push, status, or verify
---
# Encrypted Backups
`gog backup` writes Google account data into a Git repository as age-encrypted
JSONL gzip shards. The intended repository is private, for example
`https://github.com/steipete/backup-gog`, but service payloads are encrypted
before Git sees them.
## Commands
Initialize local config, create an age identity if needed, seed the backup repo,
and print the public recipient:
```bash
gog backup init \
--repo ~/Projects/backup-gog \
--remote https://github.com/steipete/backup-gog.git
```
Back up Gmail:
```bash
gog backup push --services gmail --account steipete@gmail.com
```
For a bounded smoke run:
```bash
gog backup push --services gmail --account steipete@gmail.com --query 'newer_than:7d' --max 25
```
Inspect cleartext manifest metadata:
```bash
gog backup status
```
Decrypt every shard and verify hashes and row counts:
```bash
gog backup verify
```
Use `--no-push` on `init` or `push` to commit locally without pushing to the
remote.
## Files
Local config:
```text
~/.gog/backup.json
~/.gog/age.key
```
Backup repo:
```text
README.md
manifest.json
data/gmail/<account-hash>/labels.jsonl.gz.age
data/gmail/<account-hash>/messages/YYYY/MM/part-0001.jsonl.gz.age
```
`manifest.json` is intentionally cleartext. It contains format version, export
time, public age recipients, service names, account hashes, shard paths, row
counts, encrypted byte sizes, and plaintext hashes used for verification. It
does not contain email subjects, senders, recipients, bodies, raw message IDs,
or labels.
## Encryption
Backups use the Go `filippo.io/age` library with X25519 age identities. There
is no backup password. The private identity is stored locally:
```text
~/.gog/age.key
```
The matching public recipient starts with `age1...` and is safe to store in
`~/.gog/backup.json` and `manifest.json`. The private `AGE-SECRET-KEY-...`
value must stay local or in a password manager.
For each shard, `gog backup push`:
1. Exports deterministic JSONL rows.
2. Gzip-compresses the JSONL with a fixed gzip timestamp.
3. Encrypts the compressed bytes with age for every configured recipient.
4. Writes only encrypted `*.jsonl.gz.age` files to Git.
5. Writes `manifest.json` with cleartext metadata for status and verification.
`gog backup verify` decrypts each shard with the local age identity, gunzips it,
checks the plaintext SHA-256 hash from the manifest, and verifies row counts.
## Security Boundary
The encrypted shards protect Google content from GitHub and anyone else without
the local age identity. That includes email bodies, subjects, senders,
recipients, raw MIME payloads, labels, Drive filenames, contacts, event titles,
and similar service data.
The manifest is not secret. It leaks operational metadata by design:
- Export time.
- Public age recipients.
- Service names.
- Account hashes.
- Shard paths and month buckets.
- Row counts.
- Encrypted byte sizes.
- Plaintext shard hashes used by `verify`.
- Backup cadence and which shards changed in Git history.
The account hash is not anonymity. It is useful to avoid putting the literal
email address in paths, but someone who can guess the address can compute and
compare the same hash.
Current trust model:
- Confidentiality: good for a private GitHub backup repo as long as
`~/.gog/age.key` stays private.
- Integrity against random corruption: `age` authentication, gzip decoding,
plaintext SHA-256, and row-count verification catch damaged shards.
- Integrity against repository writers: limited. Anyone with push access can
replace encrypted backup data with different data encrypted to the public
recipient. Keep repo write access restricted and review unexpected commits.
- Key compromise: if `AGE-SECRET-KEY-...` leaks, historical shards in Git
history may be readable. Rotate recipients, re-encrypt, and treat old Git
history as exposed unless it is rewritten and all copies are removed.
Future hardening ideas:
- Store only ciphertext hashes in the public manifest and move plaintext hashes
into encrypted shard metadata.
- Sign manifests or commits with a local signing key so `verify` can prove who
created the backup, not just that the files are internally consistent.
- Add optional shard padding or disable gzip for deployments that care more
about size side channels than repository size.
## Gmail Adapter
The first adapter backs up:
- Gmail labels.
- Raw Gmail messages from `users.messages.get(format=raw)`.
Raw message payloads stay base64url encoded inside encrypted JSONL. This
preserves the RFC 2822 message content while keeping the shard format text
friendly.
`--include-spam-trash` defaults to true. Use `--query` and `--max` for bounded
test exports; omit them for a full mailbox scan.
## Adding Services
Keep one backup engine and add service adapters. A service adapter should:
1. Resolve the authenticated account through normal `gog` auth.
2. Export stable rows without writing Google data.
3. Store sensitive identifiers, titles, names, and content inside encrypted
shards only.
4. Add cleartext manifest counts and account hashes only.
5. Support bounded smoke flags when the service can be huge.
Good next adapters: Calendar, Contacts/People, Tasks, then Drive.

View File

@ -48,6 +48,13 @@ Generated from `gog schema --json`.
- `gog auth tokens export <email> [flags]` - Export a refresh token to a file (contains secrets)
- `gog auth tokens import <inPath>` - Import a refresh token file into keyring (contains secrets)
- `gog auth tokens list` - List stored tokens (by key only)
- `gog backup <command> [flags]` - Encrypted Google account backups
- `gog backup gmail <command>` - Gmail backup operations
- `gog backup gmail push [flags]` - Export Gmail into encrypted backup shards
- `gog backup init [flags]` - Initialize encrypted backup config and repository
- `gog backup push [flags]` - Export services into encrypted backup shards
- `gog backup status [flags]` - Inspect backup manifest without decrypting shards
- `gog backup verify [flags]` - Decrypt and verify all backup shards
- `gog calendar (cal) <command> [flags]` - Google Calendar
- `gog calendar (cal) acl (permissions,perms) <calendarId> [flags]` - List calendar ACL
- `gog calendar (cal) alias <command>` - Manage calendar aliases
@ -357,6 +364,12 @@ Generated from `gog schema --json`.
- `gog sheets (sheet) <command> [flags]` - Google Sheets
- `gog sheets (sheet) add-tab (add-sheet) <spreadsheetId> <tabName> [flags]` - Add a new tab/sheet to a spreadsheet
- `gog sheets (sheet) append (add) <spreadsheetId> <range> [<values> ...] [flags]` - Append values to a range
- `gog sheets (sheet) chart (charts) <command>` - Manage spreadsheet charts
- `gog sheets (sheet) chart (charts) create (add,new) --spec-json=STRING <spreadsheetId> [flags]` - Create a chart from a JSON spec
- `gog sheets (sheet) chart (charts) delete (rm,remove,del) <spreadsheetId> <chartId>` - Delete a chart
- `gog sheets (sheet) chart (charts) get (show,info) <spreadsheetId> <chartId>` - Get full chart definition (spec + position)
- `gog sheets (sheet) chart (charts) list <spreadsheetId>` - List charts in a spreadsheet
- `gog sheets (sheet) chart (charts) update (edit,set) --spec-json=STRING <spreadsheetId> <chartId>` - Update a chart spec
- `gog sheets (sheet) clear <spreadsheetId> <range>` - Clear values in a range
- `gog sheets (sheet) copy (cp,duplicate) <spreadsheetId> <title> [flags]` - Copy a Google Sheet
- `gog sheets (sheet) create (new) <title> [flags]` - Create a new spreadsheet

2
go.mod
View File

@ -3,6 +3,7 @@ module github.com/steipete/gogcli
go 1.26.2
require (
filippo.io/age v1.3.1
github.com/99designs/keyring v1.2.2
github.com/alecthomas/kong v1.15.0
github.com/muesli/termenv v0.16.0
@ -19,6 +20,7 @@ require (
cloud.google.com/go/auth v0.20.0 // indirect
cloud.google.com/go/auth/oauth2adapt v0.2.8 // indirect
cloud.google.com/go/compute/metadata v0.9.0 // indirect
filippo.io/hpke v0.4.0 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect

6
go.sum
View File

@ -1,9 +1,15 @@
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd h1:ZLsPO6WdZ5zatV4UfVpr7oAwLGRZ+sebTUruuM4Ra3M=
c2sp.org/CCTV/age v0.0.0-20251208015420-e9274a7bdbfd/go.mod h1:SrHC2C7r5GkDk8R+NFVzYy/sdj0Ypg9htaPXQq5Cqeo=
cloud.google.com/go/auth v0.20.0 h1:kXTssoVb4azsVDoUiF8KvxAqrsQcQtB53DcSgta74CA=
cloud.google.com/go/auth v0.20.0/go.mod h1:942/yi/itH1SsmpyrbnTMDgGfdy2BUqIKyd0cyYLc5Q=
cloud.google.com/go/auth/oauth2adapt v0.2.8 h1:keo8NaayQZ6wimpNSmW5OPc283g65QNIiLpZnkHRbnc=
cloud.google.com/go/auth/oauth2adapt v0.2.8/go.mod h1:XQ9y31RkqZCcwJWNSx2Xvric3RrU88hAYYbjDWYDL+c=
cloud.google.com/go/compute/metadata v0.9.0 h1:pDUj4QMoPejqq20dK0Pg2N4yG9zIkYGdBtwLoEkH9Zs=
cloud.google.com/go/compute/metadata v0.9.0/go.mod h1:E0bWwX5wTnLPedCKqk3pJmVgCBSM6qQI1yTBdEb3C10=
filippo.io/age v1.3.1 h1:hbzdQOJkuaMEpRCLSN1/C5DX74RPcNCk6oqhKMXmZi0=
filippo.io/age v1.3.1/go.mod h1:EZorDTYUxt836i3zdori5IJX/v2Lj6kWFU0cfh6C0D4=
filippo.io/hpke v0.4.0 h1:p575VVQ6ted4pL+it6M00V/f2qTZITO0zgmdKCkd5+A=
filippo.io/hpke v0.4.0/go.mod h1:EmAN849/P3qdeK+PCMkDpDm83vRHM5cDipBJ8xbQLVY=
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 h1:/vQbFIOMbk2FiG/kXiLl8BRyzTWDw7gX/Hz7Dd5eDMs=
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4/go.mod h1:hN7oaIRCjzsZ2dE+yG5k+rsdt3qcwykqK6HVGcKwsw4=
github.com/99designs/keyring v1.2.2 h1:pZd3neh/EmUzWONb35LxQfvuY7kiSXAq3HQd97+XBn0=

558
internal/backup/backup.go Normal file
View File

@ -0,0 +1,558 @@
//nolint:err113,govet,revive,wrapcheck,wsl_v5 // Contextual errors keep backup call sites readable.
package backup
import (
"bufio"
"bytes"
"context"
"encoding/json"
"fmt"
"os"
"path"
"path/filepath"
"reflect"
"sort"
"strings"
"time"
)
const formatVersion = 1
type Manifest struct {
Format int `json:"format"`
App string `json:"app"`
Encrypted bool `json:"encrypted"`
Exported time.Time `json:"exported"`
Recipients []string `json:"recipients,omitempty"`
Services []string `json:"services,omitempty"`
Accounts []string `json:"accounts,omitempty"`
Counts map[string]int `json:"counts,omitempty"`
Shards []ShardEntry `json:"shards"`
}
type ShardEntry struct {
Service string `json:"service"`
Kind string `json:"kind"`
Account string `json:"account,omitempty"`
Path string `json:"path"`
Rows int `json:"rows"`
SHA256 string `json:"sha256"`
Bytes int64 `json:"bytes"`
}
type PlainShard struct {
Service string
Kind string
Account string
Path string
Rows int
Plaintext []byte
}
type Snapshot struct {
Services []string
Accounts []string
Counts map[string]int
Shards []PlainShard
}
type Result struct {
Repo string `json:"repo"`
Changed bool `json:"changed"`
Encrypted bool `json:"encrypted"`
Shards int `json:"shards"`
Counts map[string]int `json:"counts,omitempty"`
}
func Init(ctx context.Context, opts Options) (Config, string, error) {
cfg, err := ResolveOptions(opts)
if err != nil {
return Config{}, "", err
}
recipient, err := EnsureIdentity(cfg.Identity)
if err != nil {
return Config{}, "", err
}
if len(cfg.Recipients) == 0 {
cfg.Recipients = []string{recipient}
}
if err := SaveConfig(opts.ConfigPath, cfg); err != nil {
return Config{}, "", err
}
if err := ensureRepo(ctx, cfg); err != nil {
return Config{}, "", err
}
if err := writeBackupReadme(cfg.Repo); err != nil {
return Config{}, "", err
}
_, err = commitAndPush(ctx, cfg, "docs: describe encrypted gog backup", opts.Push)
return cfg, recipient, err
}
func PushSnapshot(ctx context.Context, snapshot Snapshot, opts Options) (Result, error) {
cfg, err := ResolveOptions(opts)
if err != nil {
return Result{}, err
}
if len(cfg.Recipients) == 0 {
recipient, err := RecipientFromIdentity(cfg.Identity)
if err != nil {
return Result{}, err
}
cfg.Recipients = []string{recipient}
}
if err := ensureRepo(ctx, cfg); err != nil {
return Result{}, err
}
if err := writeBackupReadme(cfg.Repo); err != nil {
return Result{}, err
}
oldManifest, _ := readManifest(cfg.Repo)
manifest, err := writeSnapshot(ctx, cfg, snapshot, oldManifest)
if err != nil {
return Result{}, err
}
changed, err := commitAndPush(ctx, cfg, "sync: update encrypted gog backup", opts.Push)
if err != nil {
return Result{}, err
}
return Result{Repo: cfg.Repo, Changed: changed, Encrypted: true, Shards: len(manifest.Shards), Counts: manifest.Counts}, nil
}
func Verify(ctx context.Context, opts Options) (Result, error) {
cfg, err := ResolveOptions(opts)
if err != nil {
return Result{}, err
}
if err := ensureRepo(ctx, cfg); err != nil {
return Result{}, err
}
manifest, err := readManifest(cfg.Repo)
if err != nil {
return Result{}, err
}
if manifest.Format != formatVersion {
return Result{}, fmt.Errorf("unsupported backup format %d", manifest.Format)
}
counts := map[string]int{}
for _, shard := range manifest.Shards {
select {
case <-ctx.Done():
return Result{}, ctx.Err()
default:
}
plaintext, err := decryptShardFile(cfg, shard)
if err != nil {
return Result{}, err
}
if got := sha256Hex(plaintext); got != shard.SHA256 {
return Result{}, fmt.Errorf("backup shard hash mismatch for %s", shard.Path)
}
rows, err := countJSONLLines(plaintext)
if err != nil {
return Result{}, fmt.Errorf("count rows in %s: %w", shard.Path, err)
}
if rows != shard.Rows {
return Result{}, fmt.Errorf("backup shard row count mismatch for %s: got %d, want %d", shard.Path, rows, shard.Rows)
}
key := shard.Service
if strings.TrimSpace(shard.Kind) != "" {
key += "." + shard.Kind
}
counts[key] += rows
}
return Result{Repo: cfg.Repo, Changed: false, Encrypted: manifest.Encrypted, Shards: len(manifest.Shards), Counts: counts}, nil
}
func Status(ctx context.Context, opts Options) (Manifest, string, error) {
cfg, err := ResolveOptions(opts)
if err != nil {
return Manifest{}, "", err
}
if err := ensureRepo(ctx, cfg); err != nil {
return Manifest{}, "", err
}
manifest, err := readManifest(cfg.Repo)
if err != nil {
return Manifest{}, "", err
}
return manifest, cfg.Repo, nil
}
func NewJSONLShard(service, kind, account, rel string, rows any) (PlainShard, error) {
plaintext, count, err := encodeJSONL(rows)
if err != nil {
return PlainShard{}, err
}
return PlainShard{
Service: strings.TrimSpace(service),
Kind: strings.TrimSpace(kind),
Account: strings.TrimSpace(account),
Path: filepath.ToSlash(rel),
Rows: count,
Plaintext: plaintext,
}, nil
}
func writeSnapshot(ctx context.Context, cfg Config, snapshot Snapshot, old Manifest) (Manifest, error) {
recipients := normalizedStrings(cfg.Recipients)
reuseEncrypted := sameStrings(old.Recipients, recipients)
shards := make([]ShardEntry, 0, len(snapshot.Shards))
for _, shard := range snapshot.Shards {
select {
case <-ctx.Done():
return Manifest{}, ctx.Err()
default:
}
entry, err := writeShard(cfg, old, shard, reuseEncrypted)
if err != nil {
return Manifest{}, err
}
shards = append(shards, entry)
}
sort.Slice(shards, func(i, j int) bool { return shards[i].Path < shards[j].Path })
manifest := Manifest{
Format: formatVersion,
App: "gog",
Encrypted: true,
Exported: time.Now().UTC(),
Recipients: recipients,
Services: normalizedStrings(snapshot.Services),
Accounts: normalizedStrings(snapshot.Accounts),
Counts: cloneCounts(snapshot.Counts),
Shards: shards,
}
if manifest.Counts == nil {
manifest.Counts = map[string]int{}
for _, shard := range shards {
key := shard.Service
if shard.Kind != "" {
key += "." + shard.Kind
}
manifest.Counts[key] += shard.Rows
}
}
if equivalentManifest(old, manifest) {
return old, nil
}
if err := removeStaleShards(cfg.Repo, shards); err != nil {
return Manifest{}, err
}
if err := writeManifest(cfg.Repo, manifest); err != nil {
return Manifest{}, err
}
return manifest, nil
}
func writeShard(cfg Config, old Manifest, shard PlainShard, reuseEncrypted bool) (ShardEntry, error) {
if strings.TrimSpace(shard.Service) == "" {
return ShardEntry{}, fmt.Errorf("backup shard service is required")
}
hash := sha256Hex(shard.Plaintext)
path, err := resolveShardPath(cfg.Repo, shard.Path)
if err != nil {
return ShardEntry{}, err
}
if oldEntry, ok := old.entry(shard.Path); reuseEncrypted && ok && oldEntry.SHA256 == hash {
if info, err := os.Stat(path); err == nil {
oldEntry.Bytes = info.Size()
return oldEntry, nil
}
}
encrypted, _, err := encryptShard(shard.Plaintext, cfg.Recipients)
if err != nil {
return ShardEntry{}, err
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return ShardEntry{}, err
}
if err := os.WriteFile(path, encrypted, 0o600); err != nil {
return ShardEntry{}, err
}
return ShardEntry{
Service: shard.Service,
Kind: shard.Kind,
Account: shard.Account,
Path: shard.Path,
Rows: shard.Rows,
SHA256: hash,
Bytes: int64(len(encrypted)),
}, nil
}
func decryptShardFile(cfg Config, shard ShardEntry) ([]byte, error) {
path, err := resolveShardPath(cfg.Repo, shard.Path)
if err != nil {
return nil, err
}
ciphertext, err := os.ReadFile(path) // #nosec G304 -- resolveShardPath confines manifest-controlled shard paths to data/*.age inside the backup repo.
if err != nil {
return nil, err
}
return decryptShard(ciphertext, cfg.Identity)
}
func resolveShardPath(repo, rel string) (string, error) {
clean := path.Clean(strings.TrimSpace(rel))
if clean == "." || clean == ".." || strings.HasPrefix(clean, "../") || path.IsAbs(clean) {
return "", fmt.Errorf("backup shard path escapes backup root: %s", rel)
}
if !strings.HasPrefix(clean, "data/") || !strings.HasSuffix(clean, ".age") {
return "", fmt.Errorf("invalid backup shard path: %s", rel)
}
full := filepath.Join(repo, filepath.FromSlash(clean))
root := filepath.Clean(filepath.Join(repo, "data"))
parent := filepath.Clean(filepath.Dir(full))
if parent != root && !strings.HasPrefix(parent, root+string(filepath.Separator)) {
return "", fmt.Errorf("backup shard path escapes backup root: %s", rel)
}
return full, nil
}
func encodeJSONL(rows any) ([]byte, int, error) {
value := reflect.ValueOf(rows)
if value.Kind() != reflect.Slice {
return nil, 0, fmt.Errorf("unsupported JSONL rows %T", rows)
}
var buf bytes.Buffer
enc := json.NewEncoder(&buf)
for i := 0; i < value.Len(); i++ {
if err := enc.Encode(value.Index(i).Interface()); err != nil {
return nil, 0, err
}
}
return buf.Bytes(), value.Len(), nil
}
func DecodeJSONL[T any](plaintext []byte, out *[]T) error {
scanner := bufio.NewScanner(bytes.NewReader(plaintext))
scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
for scanner.Scan() {
var value T
if err := json.Unmarshal(scanner.Bytes(), &value); err != nil {
return err
}
*out = append(*out, value)
}
return scanner.Err()
}
func countJSONLLines(plaintext []byte) (int, error) {
scanner := bufio.NewScanner(bytes.NewReader(plaintext))
scanner.Buffer(make([]byte, 0, 64*1024), 16*1024*1024)
count := 0
for scanner.Scan() {
if len(bytes.TrimSpace(scanner.Bytes())) > 0 {
count++
}
}
return count, scanner.Err()
}
func readManifest(repo string) (Manifest, error) {
data, err := os.ReadFile(filepath.Join(repo, "manifest.json")) // #nosec G304 -- repo is the configured local backup repository.
if err != nil {
return Manifest{}, err
}
var manifest Manifest
if err := json.Unmarshal(data, &manifest); err != nil {
return Manifest{}, err
}
return manifest, nil
}
func writeManifest(repo string, manifest Manifest) error {
data, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
return err
}
data = append(data, '\n')
return os.WriteFile(filepath.Join(repo, "manifest.json"), data, 0o600)
}
func (m Manifest) entry(path string) (ShardEntry, bool) {
for _, shard := range m.Shards {
if shard.Path == path {
return shard, true
}
}
return ShardEntry{}, false
}
func equivalentManifest(a, b Manifest) bool {
if a.Format != b.Format ||
a.App != b.App ||
a.Encrypted != b.Encrypted ||
!sameStrings(a.Recipients, b.Recipients) ||
!sameStrings(a.Services, b.Services) ||
!sameStrings(a.Accounts, b.Accounts) ||
!reflect.DeepEqual(a.Counts, b.Counts) ||
len(a.Shards) != len(b.Shards) {
return false
}
for i := range a.Shards {
left, right := a.Shards[i], b.Shards[i]
left.Bytes, right.Bytes = 0, 0
if left != right {
return false
}
}
return true
}
func normalizedStrings(values []string) []string {
seen := map[string]struct{}{}
out := make([]string, 0, len(values))
for _, value := range values {
value = strings.TrimSpace(value)
if value == "" {
continue
}
if _, ok := seen[value]; ok {
continue
}
seen[value] = struct{}{}
out = append(out, value)
}
sort.Strings(out)
return out
}
func sameStrings(a, b []string) bool {
a, b = normalizedStrings(a), normalizedStrings(b)
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
func cloneCounts(in map[string]int) map[string]int {
if len(in) == 0 {
return nil
}
out := make(map[string]int, len(in))
for key, value := range in {
out[key] = value
}
return out
}
func removeStaleShards(repo string, shards []ShardEntry) error {
keep := map[string]struct{}{}
for _, shard := range shards {
keep[filepath.Clean(filepath.Join(repo, filepath.FromSlash(shard.Path)))] = struct{}{}
}
root := filepath.Join(repo, "data")
if _, err := os.Stat(root); os.IsNotExist(err) {
return nil
}
var stale []string
if err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error {
if err != nil || d == nil || d.IsDir() {
return err
}
if !strings.HasSuffix(path, ".age") {
return nil
}
clean := filepath.Clean(path)
if _, ok := keep[clean]; ok {
return nil
}
stale = append(stale, clean)
return nil
}); err != nil {
return err
}
for _, path := range stale {
rel, err := filepath.Rel(root, path)
if err != nil || rel == "." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) || filepath.IsAbs(rel) {
return fmt.Errorf("stale shard path escapes backup root: %s", path)
}
if err := os.Remove(path); err != nil {
return err
}
}
return nil
}
func writeBackupReadme(repo string) error {
path := filepath.Join(repo, "README.md")
if _, err := os.Stat(path); err == nil {
return nil
}
const body = `# backup-gog
Encrypted Git backup for Google account data exported by gog.
This repository is written by ` + "`gog backup push`" + `. It is safe to keep on
GitHub because service payloads are encrypted before Git sees them.
## Layout
` + "```text" + `
README.md
manifest.json
data/<service>/<account-hash>/...
` + "```" + `
` + "`manifest.json`" + ` is cleartext and contains format version, export time,
public age recipients, service names, account hashes, shard paths, row counts,
encrypted byte sizes, and plaintext hashes used for verification. Email bodies,
subjects, senders, Drive filenames, contacts, event titles, and other private
Google data stay inside encrypted ` + "`*.jsonl.gz.age`" + ` shards.
## Security Model
Shard contents are deterministic JSONL, gzip-compressed with a fixed timestamp,
and encrypted with age for every configured public recipient. The local
` + "`~/.gog/age.key`" + ` identity is required to decrypt.
Git can still see manifest metadata: export time, public recipients, service
names, account hashes, shard paths, encrypted byte sizes, plaintext shard
hashes, backup cadence, and which encrypted shards changed. Git cannot read
Google content without an age identity.
Anyone who can push to this repository can replace encrypted backup data with
different data encrypted to your public recipient. Keep repository write access
restricted and review unexpected backup commits. If an age identity is
compromised, remove its public recipient and push a new backup; old Git history
may still contain shards decryptable by the compromised key.
## Push
` + "```bash" + `
gog backup push --services gmail
` + "```" + `
The command pulls/rebases this checkout, exports selected Google services,
writes encrypted shards, updates the manifest, commits, and pushes this
repository.
## Verify
` + "```bash" + `
gog backup verify
` + "```" + `
` + "`verify`" + ` decrypts every shard with the local age identity and verifies the
manifest hashes and row counts. It does not restore or write Google data.
## Recovery
Install gog, clone this repo to the path in ` + "`~/.gog/backup.json`" + `,
restore the local age identity file, then run:
` + "```bash" + `
gog backup verify
` + "```" + `
Do not commit the age identity. Only public ` + "`age1...`" + ` recipients belong in
config; ` + "`AGE-SECRET-KEY-...`" + ` values must stay local or in a password manager.
`
return os.WriteFile(path, []byte(body), 0o600)
}

View File

@ -0,0 +1,367 @@
//nolint:wsl_v5 // Tests stay compact around setup/action/assert blocks.
package backup
import (
"context"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
func TestPushSnapshotAndVerify(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
shard, err := NewJSONLShard("gmail", "messages", "acct", "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{
{"id": "m1", "raw": "private email body"},
})
if err != nil {
t.Fatalf("NewJSONLShard: %v", err)
}
result, err := PushSnapshot(ctx, Snapshot{
Services: []string{"gmail"},
Accounts: []string{"acct"},
Counts: map[string]int{"gmail.messages": 1},
Shards: []PlainShard{shard},
}, Options{ConfigPath: config, Push: false})
if err != nil {
t.Fatalf("PushSnapshot: %v", err)
}
if !result.Changed || result.Shards != 1 || result.Counts["gmail.messages"] != 1 {
t.Fatalf("unexpected push result: %+v", result)
}
ciphertext, err := os.ReadFile(filepath.Join(repo, "data", "gmail", "acct", "messages", "2026", "04", "part-0001.jsonl.gz.age"))
if err != nil {
t.Fatalf("read ciphertext: %v", err)
}
if strings.Contains(string(ciphertext), "private email body") {
t.Fatal("encrypted shard contains plaintext")
}
verify, err := Verify(ctx, Options{ConfigPath: config})
if err != nil {
t.Fatalf("Verify: %v", err)
}
if verify.Shards != 1 || verify.Counts["gmail.messages"] != 1 {
t.Fatalf("unexpected verify result: %+v", verify)
}
status, statusRepo, err := Status(ctx, Options{ConfigPath: config})
if err != nil {
t.Fatalf("Status: %v", err)
}
if statusRepo != repo || !status.Encrypted || status.Counts["gmail.messages"] != 1 {
t.Fatalf("unexpected status repo=%s manifest=%+v", statusRepo, status)
}
}
func TestInitCreatesPrivateIdentityAndConfig(t *testing.T) {
_, _, config, identity := initTestBackup(t)
for _, path := range []string{config, identity} {
info, err := os.Stat(path)
if err != nil {
t.Fatalf("stat %s: %v", path, err)
}
if got := info.Mode().Perm(); got != 0o600 {
t.Fatalf("%s mode = %v, want 0600", path, got)
}
}
data, err := os.ReadFile(identity)
if err != nil {
t.Fatalf("read identity: %v", err)
}
if !strings.HasPrefix(strings.TrimSpace(string(data)), "AGE-SECRET-KEY-") {
t.Fatalf("identity does not look like an age secret key")
}
}
func TestManifestDoesNotContainPayloadPlaintext(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
shard := mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{
"id": "msg-plain-marker",
"subject": "very secret subject marker",
"raw": "private raw mime marker",
}})
if _, err := PushSnapshot(ctx, Snapshot{
Services: []string{"gmail"},
Accounts: []string{"acct"},
Counts: map[string]int{"gmail.messages": 1},
Shards: []PlainShard{shard},
}, Options{ConfigPath: config, Push: false}); err != nil {
t.Fatalf("PushSnapshot: %v", err)
}
for _, name := range []string{"manifest.json", "README.md"} {
data, err := os.ReadFile(filepath.Join(repo, name))
if err != nil {
t.Fatalf("read %s: %v", name, err)
}
text := string(data)
for _, marker := range []string{"msg-plain-marker", "very secret subject marker", "private raw mime marker"} {
if strings.Contains(text, marker) {
t.Fatalf("%s contains private payload marker %q", name, marker)
}
}
}
}
func TestVerifyDetectsTamperedCiphertext(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}}))
path := filepath.Join(repo, filepath.FromSlash(shardPath))
ciphertext, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read ciphertext: %v", err)
}
ciphertext[len(ciphertext)-1] ^= 0xff
if err := os.WriteFile(path, ciphertext, 0o600); err != nil {
t.Fatalf("write tampered ciphertext: %v", err)
}
if _, err := Verify(ctx, Options{ConfigPath: config}); err == nil {
t.Fatal("expected verify to reject tampered ciphertext")
}
}
func TestVerifyDetectsManifestHashMismatch(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{"id": "m1", "raw": "body"}}))
manifest := readTestManifest(t, repo)
manifest.Shards[0].SHA256 = strings.Repeat("0", 64)
writeTestManifest(t, repo, manifest)
_, err := Verify(ctx, Options{ConfigPath: config})
if err == nil || !strings.Contains(err.Error(), "hash mismatch") {
t.Fatalf("Verify error = %v, want hash mismatch", err)
}
}
func TestVerifyDetectsManifestRowCountMismatch(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
pushSingleShard(t, ctx, config, mustGmailMessageShard(t, "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age", []map[string]string{{"id": "m1", "raw": "body"}}))
manifest := readTestManifest(t, repo)
manifest.Shards[0].Rows = 2
writeTestManifest(t, repo, manifest)
_, err := Verify(ctx, Options{ConfigPath: config})
if err == nil || !strings.Contains(err.Error(), "row count mismatch") {
t.Fatalf("Verify error = %v, want row count mismatch", err)
}
}
func TestPushReusesEncryptedShardWhenPlaintextAndRecipientsMatch(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
shard := mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}})
first := pushSingleShard(t, ctx, config, shard)
firstCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
second := pushSingleShard(t, ctx, config, shard)
secondCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
if !first.Changed {
t.Fatalf("first push changed = false, want true")
}
if second.Changed {
t.Fatalf("second push changed = true, want false")
}
if string(firstCiphertext) != string(secondCiphertext) {
t.Fatalf("ciphertext changed even though plaintext and recipients matched")
}
}
func TestPushReencryptsShardWhenRecipientChanges(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
shardPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
shard := mustGmailMessageShard(t, shardPath, []map[string]string{{"id": "m1", "raw": "body"}})
pushSingleShard(t, ctx, config, shard)
firstCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
secondIdentity := filepath.Join(t.TempDir(), "age.key")
secondRecipient, err := EnsureIdentity(secondIdentity)
if err != nil {
t.Fatalf("EnsureIdentity second: %v", err)
}
if _, err := PushSnapshot(ctx, Snapshot{
Services: []string{"gmail"},
Accounts: []string{"acct"},
Counts: map[string]int{"gmail.messages": 1},
Shards: []PlainShard{shard},
}, Options{ConfigPath: config, Identity: secondIdentity, Recipients: []string{secondRecipient}, Push: false}); err != nil {
t.Fatalf("PushSnapshot second recipient: %v", err)
}
secondCiphertext := readFile(t, filepath.Join(repo, filepath.FromSlash(shardPath)))
if string(firstCiphertext) == string(secondCiphertext) {
t.Fatal("ciphertext did not change after recipient rotation")
}
if _, err := Verify(ctx, Options{ConfigPath: config, Identity: secondIdentity}); err != nil {
t.Fatalf("Verify with rotated identity: %v", err)
}
}
func TestPushRemovesStaleEncryptedShards(t *testing.T) {
ctx, repo, config, _ := initTestBackup(t)
oldPath := "data/gmail/acct/messages/2026/03/part-0001.jsonl.gz.age"
keepPath := "data/gmail/acct/messages/2026/04/part-0001.jsonl.gz.age"
oldShard := mustGmailMessageShard(t, oldPath, []map[string]string{{"id": "old"}})
keepShard := mustGmailMessageShard(t, keepPath, []map[string]string{{"id": "keep"}})
if _, err := PushSnapshot(ctx, Snapshot{
Services: []string{"gmail"},
Accounts: []string{"acct"},
Counts: map[string]int{"gmail.messages": 2},
Shards: []PlainShard{oldShard, keepShard},
}, Options{ConfigPath: config, Push: false}); err != nil {
t.Fatalf("initial PushSnapshot: %v", err)
}
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(oldPath))); err != nil {
t.Fatalf("old shard should exist before pruning: %v", err)
}
pushSingleShard(t, ctx, config, keepShard)
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(oldPath))); !os.IsNotExist(err) {
t.Fatalf("old shard still exists after pruning: %v", err)
}
if _, err := os.Stat(filepath.Join(repo, filepath.FromSlash(keepPath))); err != nil {
t.Fatalf("kept shard missing after pruning: %v", err)
}
}
func TestRejectsInvalidShardPaths(t *testing.T) {
_, _, config, _ := initTestBackup(t)
for _, rel := range []string{
"../nope.age",
"/tmp/nope.age",
"manifest.age",
"data/gmail/acct/plain.jsonl",
"data/../nope.age",
} {
t.Run(rel, func(t *testing.T) {
shard := mustGmailMessageShard(t, rel, []map[string]string{{"id": "m1"}})
_, err := PushSnapshot(context.Background(), Snapshot{Shards: []PlainShard{shard}}, Options{
ConfigPath: config,
Push: false,
})
if err == nil {
t.Fatal("expected invalid shard path error")
}
})
}
}
func TestEncryptDecryptRoundTripMultipleRecipients(t *testing.T) {
dir := t.TempDir()
firstIdentity := filepath.Join(dir, "first.age")
firstRecipient, err := EnsureIdentity(firstIdentity)
if err != nil {
t.Fatalf("EnsureIdentity first: %v", err)
}
secondIdentity := filepath.Join(dir, "second.age")
secondRecipient, err := EnsureIdentity(secondIdentity)
if err != nil {
t.Fatalf("EnsureIdentity second: %v", err)
}
encrypted, hash, err := encryptShard([]byte("secret jsonl\n"), []string{firstRecipient, secondRecipient})
if err != nil {
t.Fatalf("encryptShard: %v", err)
}
if hash != sha256Hex([]byte("secret jsonl\n")) {
t.Fatalf("hash = %s, want plaintext sha256", hash)
}
for _, identity := range []string{firstIdentity, secondIdentity} {
plaintext, err := decryptShard(encrypted, identity)
if err != nil {
t.Fatalf("decryptShard %s: %v", identity, err)
}
if string(plaintext) != "secret jsonl\n" {
t.Fatalf("plaintext = %q", plaintext)
}
}
}
func initTestBackup(t *testing.T) (context.Context, string, string, string) {
t.Helper()
ctx := context.Background()
dir := t.TempDir()
repo := filepath.Join(dir, "repo")
identity := filepath.Join(dir, "age.key")
config := filepath.Join(dir, "backup.json")
cfg, recipient, err := Init(ctx, Options{
ConfigPath: config,
Repo: repo,
Identity: identity,
Push: false,
})
if err != nil {
t.Fatalf("Init: %v", err)
}
if cfg.Repo != repo || !strings.HasPrefix(recipient, "age1") {
t.Fatalf("unexpected init cfg=%+v recipient=%q", cfg, recipient)
}
return ctx, repo, config, identity
}
func mustGmailMessageShard(t *testing.T, rel string, rows any) PlainShard {
t.Helper()
shard, err := NewJSONLShard("gmail", "messages", "acct", rel, rows)
if err != nil {
t.Fatalf("NewJSONLShard: %v", err)
}
return shard
}
func pushSingleShard(t *testing.T, ctx context.Context, config string, shard PlainShard) Result {
t.Helper()
result, err := PushSnapshot(ctx, Snapshot{
Services: []string{shard.Service},
Accounts: []string{shard.Account},
Counts: map[string]int{shard.Service + "." + shard.Kind: shard.Rows},
Shards: []PlainShard{shard},
}, Options{ConfigPath: config, Push: false})
if err != nil {
t.Fatalf("PushSnapshot: %v", err)
}
return result
}
func readTestManifest(t *testing.T, repo string) Manifest {
t.Helper()
data := readFile(t, filepath.Join(repo, "manifest.json"))
var manifest Manifest
if err := json.Unmarshal(data, &manifest); err != nil {
t.Fatalf("unmarshal manifest: %v", err)
}
return manifest
}
func writeTestManifest(t *testing.T, repo string, manifest Manifest) {
t.Helper()
data, err := json.MarshalIndent(manifest, "", " ")
if err != nil {
t.Fatalf("marshal manifest: %v", err)
}
data = append(data, '\n')
if err := os.WriteFile(filepath.Join(repo, "manifest.json"), data, 0o600); err != nil {
t.Fatalf("write manifest: %v", err)
}
}
func readFile(t *testing.T, path string) []byte {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
return data
}

117
internal/backup/config.go Normal file
View File

@ -0,0 +1,117 @@
//nolint:wrapcheck,wsl_v5 // Backup config errors are surfaced directly to the CLI.
package backup
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
)
const (
defaultRemote = "https://github.com/steipete/backup-gog.git"
)
type Config struct {
Repo string `json:"repo"`
Remote string `json:"remote"`
Identity string `json:"identity"`
Recipients []string `json:"recipients"`
}
type Options struct {
ConfigPath string
Repo string
Remote string
Identity string
Recipients []string
Push bool
}
func DefaultConfig() Config {
return Config{
Repo: "~/Projects/backup-gog",
Remote: defaultRemote,
Identity: "~/.gog/age.key",
}
}
func DefaultConfigPath() string {
home, err := os.UserHomeDir()
if err != nil {
return "backup.json"
}
return filepath.Join(home, ".gog", "backup.json")
}
func LoadConfig(path string) (Config, error) {
if strings.TrimSpace(path) == "" {
path = DefaultConfigPath()
}
cfg := DefaultConfig()
data, err := os.ReadFile(expandHome(path))
if err != nil {
if errors.Is(err, os.ErrNotExist) {
return cfg, nil
}
return Config{}, err
}
if err := json.Unmarshal(data, &cfg); err != nil {
return Config{}, fmt.Errorf("read backup config: %w", err)
}
return cfg, nil
}
func SaveConfig(path string, cfg Config) error {
if strings.TrimSpace(path) == "" {
path = DefaultConfigPath()
}
path = expandHome(path)
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return err
}
data, err := json.MarshalIndent(cfg, "", " ")
if err != nil {
return err
}
data = append(data, '\n')
return os.WriteFile(path, data, 0o600)
}
func ResolveOptions(opts Options) (Config, error) {
cfg, err := LoadConfig(opts.ConfigPath)
if err != nil {
return Config{}, err
}
if strings.TrimSpace(opts.Repo) != "" {
cfg.Repo = opts.Repo
}
if strings.TrimSpace(opts.Remote) != "" {
cfg.Remote = opts.Remote
}
if strings.TrimSpace(opts.Identity) != "" {
cfg.Identity = opts.Identity
}
if len(opts.Recipients) > 0 {
cfg.Recipients = opts.Recipients
}
cfg.Repo = expandHome(cfg.Repo)
cfg.Identity = expandHome(cfg.Identity)
return cfg, nil
}
func expandHome(path string) string {
if path == "~" {
if home, err := os.UserHomeDir(); err == nil {
return home
}
}
if after, ok := strings.CutPrefix(path, "~/"); ok {
if home, err := os.UserHomeDir(); err == nil {
return filepath.Join(home, after)
}
}
return path
}

142
internal/backup/crypto.go Normal file
View File

@ -0,0 +1,142 @@
//nolint:err113,govet,wrapcheck,wsl_v5 // Crypto helpers keep age/gzip errors intact for diagnosis.
package backup
import (
"bytes"
"compress/gzip"
"crypto/sha256"
"encoding/hex"
"fmt"
"io"
"os"
"path/filepath"
"strings"
"time"
"filippo.io/age"
)
func EnsureIdentity(path string) (string, error) {
path = expandHome(path)
if data, err := os.ReadFile(path); err == nil { // #nosec G304 -- path is the configured local age identity file.
identity, err := parseIdentity(data)
if err != nil {
return "", err
}
return identity.Recipient().String(), nil
} else if !os.IsNotExist(err) {
return "", err
}
identity, err := age.GenerateX25519Identity()
if err != nil {
return "", err
}
if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
return "", err
}
data := []byte(identity.String() + "\n")
if err := os.WriteFile(path, data, 0o600); err != nil {
return "", err
}
return identity.Recipient().String(), nil
}
func RecipientFromIdentity(path string) (string, error) {
data, err := os.ReadFile(expandHome(path))
if err != nil {
return "", err
}
identity, err := parseIdentity(data)
if err != nil {
return "", err
}
return identity.Recipient().String(), nil
}
func encryptShard(plaintext []byte, recipientStrings []string) ([]byte, string, error) {
recipients, err := parseRecipients(recipientStrings)
if err != nil {
return nil, "", err
}
var compressed bytes.Buffer
gz := gzip.NewWriter(&compressed)
gz.ModTime = time.Unix(0, 0).UTC()
_, _ = gz.Write(plaintext)
_ = gz.Close()
var encrypted bytes.Buffer
w, err := age.Encrypt(&encrypted, recipients...)
if err != nil {
return nil, "", err
}
_, _ = w.Write(compressed.Bytes())
if err := w.Close(); err != nil {
return nil, "", err
}
return encrypted.Bytes(), sha256Hex(plaintext), nil
}
func decryptShard(ciphertext []byte, identityPath string) ([]byte, error) {
data, err := os.ReadFile(expandHome(identityPath)) // #nosec G304 -- path is the configured local age identity file.
if err != nil {
return nil, err
}
identity, err := parseIdentity(data)
if err != nil {
return nil, err
}
r, err := age.Decrypt(bytes.NewReader(ciphertext), identity)
if err != nil {
return nil, err
}
gz, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
defer func() { _ = gz.Close() }()
plaintext, err := io.ReadAll(gz)
if err != nil {
return nil, err
}
return plaintext, nil
}
func parseRecipients(values []string) ([]age.Recipient, error) {
var out []age.Recipient
for _, value := range values {
value = strings.TrimSpace(value)
if value == "" {
continue
}
recipient, err := age.ParseX25519Recipient(value)
if err != nil {
return nil, fmt.Errorf("parse age recipient: %w", err)
}
out = append(out, recipient)
}
if len(out) == 0 {
return nil, fmt.Errorf("at least one age recipient is required")
}
return out, nil
}
func parseIdentity(data []byte) (*age.X25519Identity, error) {
for _, line := range strings.Split(string(data), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
identity, err := age.ParseX25519Identity(line)
if err != nil {
return nil, fmt.Errorf("parse age identity: %w", err)
}
return identity, nil
}
return nil, fmt.Errorf("age identity file is empty")
}
func sha256Hex(data []byte) string {
sum := sha256.Sum256(data)
return hex.EncodeToString(sum[:])
}

92
internal/backup/git.go Normal file
View File

@ -0,0 +1,92 @@
//nolint:err113,wrapcheck,wsl_v5 // Git helper returns command-context errors.
package backup
import (
"bytes"
"context"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
)
func ensureRepo(ctx context.Context, cfg Config) error {
if strings.TrimSpace(cfg.Repo) == "" {
return fmt.Errorf("backup repo path is required")
}
if _, err := os.Stat(filepath.Join(cfg.Repo, ".git")); err == nil {
pullErr := git(ctx, cfg.Repo, "pull", "--rebase")
if pullErr != nil {
hasHead := git(ctx, cfg.Repo, "rev-parse", "--verify", "HEAD") == nil
if !hasHead {
return nil
}
if strings.Contains(pullErr.Error(), "no tracking information") ||
strings.Contains(pullErr.Error(), "No remote repository specified") ||
strings.Contains(pullErr.Error(), "no such ref was fetched") {
return nil
}
return pullErr
}
return nil
}
if strings.TrimSpace(cfg.Remote) != "" {
if err := os.MkdirAll(filepath.Dir(cfg.Repo), 0o700); err != nil {
return err
}
if err := git(ctx, "", "clone", cfg.Remote, cfg.Repo); err == nil {
return nil
}
}
if err := os.MkdirAll(cfg.Repo, 0o700); err != nil {
return err
}
if err := git(ctx, cfg.Repo, "init"); err != nil {
return err
}
if strings.TrimSpace(cfg.Remote) != "" {
if err := git(ctx, cfg.Repo, "remote", "add", "origin", cfg.Remote); err != nil {
return err
}
}
return nil
}
func commitAndPush(ctx context.Context, cfg Config, message string, push bool) (bool, error) {
if err := git(ctx, cfg.Repo, "add", "."); err != nil {
return false, err
}
if err := git(ctx, cfg.Repo, "diff", "--cached", "--quiet"); err == nil {
return false, nil
}
if err := git(ctx, cfg.Repo, "commit", "-m", message); err != nil {
return false, err
}
if push {
if err := git(ctx, cfg.Repo, "push", "-u", "origin", "HEAD"); err != nil {
return true, err
}
}
return true, nil
}
func git(ctx context.Context, dir string, args ...string) error {
cmd := exec.CommandContext(ctx, "git", args...) // #nosec G204 -- callers pass fixed git subcommands plus configured repo paths.
cmd.Dir = dir
cmd.Env = append(os.Environ(),
"GIT_AUTHOR_NAME=gog",
"GIT_AUTHOR_EMAIL=gog@example.invalid",
"GIT_COMMITTER_NAME=gog",
"GIT_COMMITTER_EMAIL=gog@example.invalid",
)
var stderr bytes.Buffer
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
if stderr.Len() > 0 {
return fmt.Errorf("git %s: %w: %s", strings.Join(args, " "), err, strings.TrimSpace(stderr.String()))
}
return fmt.Errorf("git %s: %w", strings.Join(args, " "), err)
}
return nil
}

471
internal/cmd/backup.go Normal file
View File

@ -0,0 +1,471 @@
package cmd
import (
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"os"
"sort"
"strconv"
"strings"
"sync"
"time"
"google.golang.org/api/gmail/v1"
"github.com/steipete/gogcli/internal/backup"
"github.com/steipete/gogcli/internal/outfmt"
"github.com/steipete/gogcli/internal/ui"
)
type BackupCmd struct {
Init BackupInitCmd `cmd:"" name:"init" help:"Initialize encrypted backup config and repository"`
Push BackupPushCmd `cmd:"" name:"push" help:"Export services into encrypted backup shards"`
Status BackupStatusCmd `cmd:"" name:"status" help:"Inspect backup manifest without decrypting shards"`
Verify BackupVerifyCmd `cmd:"" name:"verify" help:"Decrypt and verify all backup shards"`
Gmail BackupGmailCmd `cmd:"" name:"gmail" help:"Gmail backup operations"`
}
type BackupGmailCmd struct {
Push BackupGmailPushCmd `cmd:"" name:"push" help:"Export Gmail into encrypted backup shards"`
}
type backupFlags struct {
Config string `name:"config" help:"Backup config path" default:""`
Repo string `name:"repo" help:"Local backup repository path"`
Remote string `name:"remote" help:"Backup Git remote URL"`
Identity string `name:"identity" help:"Local age identity path"`
Recipients []string `name:"recipient" help:"Public age recipient (repeatable)"`
NoPush bool `name:"no-push" help:"Commit locally but do not push to the remote"`
}
func (f backupFlags) options() backup.Options {
return backup.Options{
ConfigPath: f.Config,
Repo: f.Repo,
Remote: f.Remote,
Identity: f.Identity,
Recipients: f.Recipients,
Push: !f.NoPush,
}
}
type BackupInitCmd struct {
backupFlags
}
func (c *BackupInitCmd) Run(ctx context.Context) error {
cfg, recipient, err := backup.Init(ctx, c.options())
if err != nil {
return err
}
if outfmt.IsJSON(ctx) {
return outfmt.WriteJSON(ctx, os.Stdout, map[string]any{
"repo": cfg.Repo,
"remote": cfg.Remote,
"identity": cfg.Identity,
"recipient": recipient,
})
}
u := ui.FromContext(ctx)
u.Out().Printf("repo\t%s", cfg.Repo)
u.Out().Printf("remote\t%s", cfg.Remote)
u.Out().Printf("identity\t%s", cfg.Identity)
u.Out().Printf("recipient\t%s", recipient)
return nil
}
type BackupPushCmd struct {
backupFlags
Services string `name:"services" help:"Comma-separated services to back up (currently: gmail)" default:"gmail"`
Query string `name:"query" help:"Gmail query for bounded/test backups"`
Max int64 `name:"max" aliases:"limit" help:"Max Gmail messages to export; 0 means all" default:"0"`
IncludeSpamTrash bool `name:"include-spam-trash" help:"Include Gmail spam and trash" default:"true"`
ShardMaxRows int `name:"shard-max-rows" help:"Max Gmail messages per encrypted shard" default:"1000"`
}
func (c *BackupPushCmd) Run(ctx context.Context, flags *RootFlags) error {
services := splitCSV(c.Services)
if len(services) == 0 {
return usage("at least one --services value is required")
}
var snapshots []backup.Snapshot
for _, service := range services {
switch strings.ToLower(strings.TrimSpace(service)) {
case "gmail":
snapshot, err := buildGmailBackupSnapshot(ctx, flags, gmailBackupOptions{
Query: c.Query,
Max: c.Max,
IncludeSpamTrash: c.IncludeSpamTrash,
ShardMaxRows: c.ShardMaxRows,
})
if err != nil {
return err
}
snapshots = append(snapshots, snapshot)
default:
return fmt.Errorf("unsupported backup service %q (currently: gmail)", service)
}
}
result, err := backup.PushSnapshot(ctx, mergeBackupSnapshots(snapshots...), c.options())
if err != nil {
return err
}
return writeBackupResult(ctx, result)
}
type BackupGmailPushCmd struct {
backupFlags
Query string `name:"query" help:"Gmail query for bounded/test backups"`
Max int64 `name:"max" aliases:"limit" help:"Max Gmail messages to export; 0 means all" default:"0"`
IncludeSpamTrash bool `name:"include-spam-trash" help:"Include spam and trash" default:"true"`
ShardMaxRows int `name:"shard-max-rows" help:"Max messages per encrypted shard" default:"1000"`
}
func (c *BackupGmailPushCmd) Run(ctx context.Context, flags *RootFlags) error {
snapshot, err := buildGmailBackupSnapshot(ctx, flags, gmailBackupOptions{
Query: c.Query,
Max: c.Max,
IncludeSpamTrash: c.IncludeSpamTrash,
ShardMaxRows: c.ShardMaxRows,
})
if err != nil {
return err
}
result, err := backup.PushSnapshot(ctx, snapshot, c.options())
if err != nil {
return err
}
return writeBackupResult(ctx, result)
}
type BackupStatusCmd struct {
backupFlags
}
func (c *BackupStatusCmd) Run(ctx context.Context) error {
manifest, repo, err := backup.Status(ctx, c.options())
if err != nil {
return err
}
if outfmt.IsJSON(ctx) {
return outfmt.WriteJSON(ctx, os.Stdout, map[string]any{"repo": repo, "manifest": manifest})
}
u := ui.FromContext(ctx)
u.Out().Printf("repo\t%s", repo)
u.Out().Printf("encrypted\t%t", manifest.Encrypted)
u.Out().Printf("exported\t%s", manifest.Exported.Format(time.RFC3339))
u.Out().Printf("services\t%s", strings.Join(manifest.Services, ","))
u.Out().Printf("accounts\t%s", strings.Join(manifest.Accounts, ","))
u.Out().Printf("shards\t%d", len(manifest.Shards))
keys := make([]string, 0, len(manifest.Counts))
for key := range manifest.Counts {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
u.Out().Printf("count.%s\t%d", key, manifest.Counts[key])
}
return nil
}
type BackupVerifyCmd struct {
backupFlags
}
func (c *BackupVerifyCmd) Run(ctx context.Context) error {
result, err := backup.Verify(ctx, c.options())
if err != nil {
return err
}
return writeBackupResult(ctx, result)
}
type gmailBackupOptions struct {
Query string
Max int64
IncludeSpamTrash bool
ShardMaxRows int
}
type gmailBackupMessage struct {
ID string `json:"id"`
ThreadID string `json:"threadId,omitempty"`
HistoryID string `json:"historyId,omitempty"`
InternalDate int64 `json:"internalDate,omitempty"`
LabelIDs []string `json:"labelIds,omitempty"`
SizeEstimate int64 `json:"sizeEstimate,omitempty"`
Raw string `json:"raw"`
}
type gmailBackupLabel struct {
ID string `json:"id"`
Name string `json:"name"`
Type string `json:"type,omitempty"`
MessageListVisibility string `json:"messageListVisibility,omitempty"`
LabelListVisibility string `json:"labelListVisibility,omitempty"`
MessagesTotal int64 `json:"messagesTotal,omitempty"`
MessagesUnread int64 `json:"messagesUnread,omitempty"`
ThreadsTotal int64 `json:"threadsTotal,omitempty"`
ThreadsUnread int64 `json:"threadsUnread,omitempty"`
}
func buildGmailBackupSnapshot(ctx context.Context, flags *RootFlags, opts gmailBackupOptions) (backup.Snapshot, error) {
if opts.ShardMaxRows <= 0 {
opts.ShardMaxRows = 1000
}
account, err := requireAccount(flags)
if err != nil {
return backup.Snapshot{}, err
}
svc, err := newGmailService(ctx, account)
if err != nil {
return backup.Snapshot{}, err
}
accountHash := backupAccountHash(account)
labels, err := fetchGmailBackupLabels(ctx, svc)
if err != nil {
return backup.Snapshot{}, err
}
messages, err := fetchGmailBackupMessages(ctx, svc, opts)
if err != nil {
return backup.Snapshot{}, err
}
shards := make([]backup.PlainShard, 0, 1)
labelShard, err := backup.NewJSONLShard("gmail", "labels", accountHash, fmt.Sprintf("data/gmail/%s/labels.jsonl.gz.age", accountHash), labels)
if err != nil {
return backup.Snapshot{}, err
}
shards = append(shards, labelShard)
messageShards, err := buildGmailMessageShards(accountHash, messages, opts.ShardMaxRows)
if err != nil {
return backup.Snapshot{}, err
}
shards = append(shards, messageShards...)
return backup.Snapshot{
Services: []string{"gmail"},
Accounts: []string{accountHash},
Counts: map[string]int{
"gmail.labels": len(labels),
"gmail.messages": len(messages),
},
Shards: shards,
}, nil
}
func fetchGmailBackupLabels(ctx context.Context, svc *gmail.Service) ([]gmailBackupLabel, error) {
resp, err := svc.Users.Labels.List("me").Context(ctx).Do()
if err != nil {
return nil, err
}
out := make([]gmailBackupLabel, 0, len(resp.Labels))
for _, label := range resp.Labels {
if label == nil {
continue
}
out = append(out, gmailBackupLabel{
ID: label.Id,
Name: label.Name,
Type: label.Type,
MessageListVisibility: label.MessageListVisibility,
LabelListVisibility: label.LabelListVisibility,
MessagesTotal: label.MessagesTotal,
MessagesUnread: label.MessagesUnread,
ThreadsTotal: label.ThreadsTotal,
ThreadsUnread: label.ThreadsUnread,
})
}
sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
return out, nil
}
func fetchGmailBackupMessages(ctx context.Context, svc *gmail.Service, opts gmailBackupOptions) ([]gmailBackupMessage, error) {
ids, err := listGmailBackupMessageIDs(ctx, svc, opts)
if err != nil {
return nil, err
}
const maxConcurrency = 8
sem := make(chan struct{}, maxConcurrency)
type result struct {
index int
msg gmailBackupMessage
err error
}
results := make(chan result, len(ids))
var wg sync.WaitGroup
for i, id := range ids {
wg.Add(1)
go func(index int, messageID string) {
defer wg.Done()
select {
case sem <- struct{}{}:
defer func() { <-sem }()
case <-ctx.Done():
results <- result{index: index, err: ctx.Err()}
return
}
msg, err := svc.Users.Messages.Get("me", messageID).
Format(gmailFormatRaw).
Fields("id,threadId,historyId,internalDate,labelIds,sizeEstimate,raw").
Context(ctx).
Do()
if err != nil {
results <- result{index: index, err: fmt.Errorf("gmail message %s: %w", messageID, err)}
return
}
if strings.TrimSpace(msg.Raw) == "" {
results <- result{index: index, err: fmt.Errorf("gmail message %s returned empty raw payload", messageID)}
return
}
results <- result{index: index, msg: gmailBackupMessage{
ID: msg.Id,
ThreadID: msg.ThreadId,
HistoryID: formatHistoryID(msg.HistoryId),
InternalDate: msg.InternalDate,
LabelIDs: append([]string(nil), msg.LabelIds...),
SizeEstimate: msg.SizeEstimate,
Raw: msg.Raw,
}}
}(i, id)
}
go func() {
wg.Wait()
close(results)
}()
ordered := make([]gmailBackupMessage, len(ids))
var firstErr error
for res := range results {
if res.err != nil && firstErr == nil {
firstErr = res.err
}
ordered[res.index] = res.msg
}
if firstErr != nil {
return nil, firstErr
}
return ordered, nil
}
func listGmailBackupMessageIDs(ctx context.Context, svc *gmail.Service, opts gmailBackupOptions) ([]string, error) {
var ids []string
pageToken := ""
for {
maxResults := int64(500)
if opts.Max > 0 {
remaining := opts.Max - int64(len(ids))
if remaining <= 0 {
break
}
if remaining < maxResults {
maxResults = remaining
}
}
call := svc.Users.Messages.List("me").
MaxResults(maxResults).
IncludeSpamTrash(opts.IncludeSpamTrash).
Fields("messages(id),nextPageToken").
Context(ctx)
if strings.TrimSpace(opts.Query) != "" {
call = call.Q(opts.Query)
}
if pageToken != "" {
call = call.PageToken(pageToken)
}
resp, err := call.Do()
if err != nil {
return nil, err
}
for _, message := range resp.Messages {
if message != nil && strings.TrimSpace(message.Id) != "" {
ids = append(ids, message.Id)
}
}
if resp.NextPageToken == "" {
break
}
pageToken = resp.NextPageToken
}
return ids, nil
}
func buildGmailMessageShards(accountHash string, messages []gmailBackupMessage, shardMaxRows int) ([]backup.PlainShard, error) {
if shardMaxRows <= 0 {
shardMaxRows = 1000
}
buckets := map[string][]gmailBackupMessage{}
for _, message := range messages {
t := time.UnixMilli(message.InternalDate).UTC()
if message.InternalDate <= 0 {
t = time.Unix(0, 0).UTC()
}
key := fmt.Sprintf("%04d/%02d", t.Year(), int(t.Month()))
buckets[key] = append(buckets[key], message)
}
keys := make([]string, 0, len(buckets))
for key := range buckets {
keys = append(keys, key)
}
sort.Strings(keys)
shards := make([]backup.PlainShard, 0, len(keys))
for _, key := range keys {
values := buckets[key]
sort.Slice(values, func(i, j int) bool {
if values[i].InternalDate == values[j].InternalDate {
return values[i].ID < values[j].ID
}
return values[i].InternalDate < values[j].InternalDate
})
for part, start := 1, 0; start < len(values); part, start = part+1, start+shardMaxRows {
end := start + shardMaxRows
if end > len(values) {
end = len(values)
}
rel := fmt.Sprintf("data/gmail/%s/messages/%s/part-%04d.jsonl.gz.age", accountHash, key, part)
shard, err := backup.NewJSONLShard("gmail", "messages", accountHash, rel, values[start:end])
if err != nil {
return nil, err
}
shards = append(shards, shard)
}
}
return shards, nil
}
func backupAccountHash(account string) string {
sum := sha256.Sum256([]byte(strings.ToLower(strings.TrimSpace(account))))
return hex.EncodeToString(sum[:12])
}
func mergeBackupSnapshots(snapshots ...backup.Snapshot) backup.Snapshot {
out := backup.Snapshot{Counts: map[string]int{}}
for _, snapshot := range snapshots {
out.Services = append(out.Services, snapshot.Services...)
out.Accounts = append(out.Accounts, snapshot.Accounts...)
out.Shards = append(out.Shards, snapshot.Shards...)
for key, value := range snapshot.Counts {
out.Counts[key] += value
}
}
return out
}
func writeBackupResult(ctx context.Context, result backup.Result) error {
if outfmt.IsJSON(ctx) {
return outfmt.WriteJSON(ctx, os.Stdout, result)
}
u := ui.FromContext(ctx)
u.Out().Printf("repo\t%s", result.Repo)
u.Out().Printf("changed\t%t", result.Changed)
u.Out().Printf("encrypted\t%t", result.Encrypted)
u.Out().Printf("shards\t%d", result.Shards)
keys := make([]string, 0, len(result.Counts))
for key := range result.Counts {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
u.Out().Printf("count.%s\t%s", key, strconv.Itoa(result.Counts[key]))
}
return nil
}

View File

@ -0,0 +1,98 @@
package cmd
import (
"strings"
"testing"
"time"
"github.com/steipete/gogcli/internal/backup"
)
func TestBackupAccountHashStableAndOpaque(t *testing.T) {
got := backupAccountHash(" User@Example.COM ")
want := backupAccountHash("user@example.com")
if got != want {
t.Fatalf("hash not normalized: got %s want %s", got, want)
}
if len(got) != 24 {
t.Fatalf("hash length = %d, want 24 hex chars", len(got))
}
if strings.Contains(got, "user") || strings.Contains(got, "example") {
t.Fatalf("hash leaks account text: %s", got)
}
}
func TestBuildGmailMessageShardsBucketsSortsAndChunks(t *testing.T) {
accountHash := "accthash"
messages := []gmailBackupMessage{
{ID: "march-new", InternalDate: mustUnixMilli(t, "2026-03-02T10:00:00Z"), Raw: "raw-3"},
{ID: "april-later", InternalDate: mustUnixMilli(t, "2026-04-02T10:00:00Z"), Raw: "raw-2"},
{ID: "april-earlier-b", InternalDate: mustUnixMilli(t, "2026-04-01T10:00:00Z"), Raw: "raw-1b"},
{ID: "april-earlier-a", InternalDate: mustUnixMilli(t, "2026-04-01T10:00:00Z"), Raw: "raw-1a"},
}
shards, err := buildGmailMessageShards(accountHash, messages, 2)
if err != nil {
t.Fatalf("buildGmailMessageShards: %v", err)
}
if len(shards) != 3 {
t.Fatalf("len(shards) = %d, want 3", len(shards))
}
wantPaths := []string{
"data/gmail/accthash/messages/2026/03/part-0001.jsonl.gz.age",
"data/gmail/accthash/messages/2026/04/part-0001.jsonl.gz.age",
"data/gmail/accthash/messages/2026/04/part-0002.jsonl.gz.age",
}
for i, want := range wantPaths {
if shards[i].Path != want {
t.Fatalf("shards[%d].Path = %q, want %q", i, shards[i].Path, want)
}
}
if shards[0].Rows != 1 || shards[1].Rows != 2 || shards[2].Rows != 1 {
t.Fatalf("unexpected row counts: %d %d %d", shards[0].Rows, shards[1].Rows, shards[2].Rows)
}
var aprilFirst []gmailBackupMessage
if err := backup.DecodeJSONL(shards[1].Plaintext, &aprilFirst); err != nil {
t.Fatalf("DecodeJSONL: %v", err)
}
gotIDs := []string{aprilFirst[0].ID, aprilFirst[1].ID}
wantIDs := []string{"april-earlier-a", "april-earlier-b"}
for i := range wantIDs {
if gotIDs[i] != wantIDs[i] {
t.Fatalf("april shard IDs = %v, want %v", gotIDs, wantIDs)
}
}
}
func TestMergeBackupSnapshotsKeepsCountsAndShardOrder(t *testing.T) {
left := backup.Snapshot{
Services: []string{"gmail"},
Accounts: []string{"acct1"},
Counts: map[string]int{"gmail.messages": 2},
Shards: []backup.PlainShard{{Path: "data/gmail/acct1/messages/2026/04/part-0001.jsonl.gz.age"}},
}
right := backup.Snapshot{
Services: []string{"calendar"},
Accounts: []string{"acct1"},
Counts: map[string]int{"calendar.events": 3},
Shards: []backup.PlainShard{{Path: "data/calendar/acct1/events.jsonl.gz.age"}},
}
merged := mergeBackupSnapshots(left, right)
if merged.Counts["gmail.messages"] != 2 || merged.Counts["calendar.events"] != 3 {
t.Fatalf("unexpected counts: %+v", merged.Counts)
}
if len(merged.Shards) != 2 || merged.Shards[0].Path != left.Shards[0].Path || merged.Shards[1].Path != right.Shards[0].Path {
t.Fatalf("unexpected shard order: %+v", merged.Shards)
}
}
func mustUnixMilli(t *testing.T, value string) int64 {
t.Helper()
parsed, err := time.Parse(time.RFC3339, value)
if err != nil {
t.Fatalf("parse time %q: %v", value, err)
}
return parsed.UnixMilli()
}

View File

@ -64,6 +64,7 @@ type CLI struct {
Whoami PeopleMeCmd `cmd:"" name:"whoami" aliases:"who-am-i" help:"Show your profile (alias for 'people me')"`
Auth AuthCmd `cmd:"" help:"Auth and credentials"`
Backup BackupCmd `cmd:"" help:"Encrypted Google account backups"`
Groups GroupsCmd `cmd:"" aliases:"group" help:"Google Groups"`
Admin AdminCmd `cmd:"" help:"Google Workspace Admin (Directory API) - requires domain-wide delegation"`
Drive DriveCmd `cmd:"" aliases:"drv" help:"Google Drive"`