openclaw/crawlkit
openclaw/crawlkit
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 2 Wiki Activity
[BREAKGLASS] Append-only mirror of github.com/openclaw/crawlkit
25 Commits 5 Branches 25 Tags 500 KiB
Go 100%
b041445d0f
Go to file
Vincent Koc b041445d0f
feat(tui): tune pane palette and wheel scrolling
2026-05-02 09:41:17 -07:00
.github fix(ci): use available GitHub action versions 2026-05-01 09:49:38 -07:00
cache refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
config refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
control feat(control): add app metadata and status contracts 2026-05-01 15:23:03 -07:00
docs ci: add validation and publishing metadata 2026-05-01 08:43:29 -07:00
mirror refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
output refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
snapshot refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
state refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
store refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
tui feat(tui): tune pane palette and wheel scrolling 2026-05-02 09:41:17 -07:00
.editorconfig chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitattributes chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitignore chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
CHANGELOG.md docs(tui): document pane browser polish 2026-05-02 09:22:39 -07:00
CONTRIBUTING.md chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
go.mod chore(tui): tidy direct styling deps 2026-05-02 07:48:50 -07:00
go.sum feat(termkit): add shared terminal browser 2026-05-01 10:20:46 -07:00
LICENSE chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
Makefile ci: add validation and publishing metadata 2026-05-01 08:43:29 -07:00
README.md docs(tui): document pane browser polish 2026-05-02 09:22:39 -07:00

README.md

crawlkit

Shared Go infrastructure for local-first crawler archives.

crawlkit is not a universal Slack, Discord, Notion, or GitHub crawler. It is the reusable foundation beneath those tools: SQLite hygiene, TOML config defaults, portable JSONL/Gzip packing, git-backed snapshot sharing, sync state, CLI output helpers, and safe desktop-cache snapshot utilities.

Install

go get github.com/vincentkoc/crawlkit@v0.3.7

Go packages are published by tagging this repository. There is no separate package registry step. See docs/publishing.md for the release commands.

Packages

  • config: standard TOML config paths, runtime dirs, and token diagnostics.
  • store: SQLite open/read-only/transaction/query helpers.
  • snapshot: manifest.json plus JSONL/Gzip table snapshot export and import.
  • mirror: clone/init/pull/commit/push helpers for private snapshot repos.
  • state: generic crawler cursor and freshness records.
  • output: text/json/log output helpers.
  • tui: shared terminal archive browser with compact rows, context/detail panes, and local/remote source status.
  • cache: safe read-only local cache snapshot helpers.

Safety

Library tests use temporary directories. They do not touch app runtime stores such as ~/.config/gitcrawl, ~/.slacrawl, ~/.discrawl, or ~/.notcrawl.