openclaw/crawlkit
openclaw/crawlkit
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 2 Wiki Activity
[BREAKGLASS] Append-only mirror of github.com/openclaw/crawlkit
145 Commits 5 Branches 25 Tags 500 KiB
Go 100%
8986b4e057
Go to file
dependabot[bot] 8986b4e057
build(deps): bump github.com/charmbracelet/x/ansi from 0.11.6 to 0.11.7 
Bumps [github.com/charmbracelet/x/ansi](https://github.com/charmbracelet/x) from 0.11.6 to 0.11.7.
- [Commits](https://github.com/charmbracelet/x/compare/ansi/v0.11.6...ansi/v0.11.7)

---
updated-dependencies:
- dependency-name: github.com/charmbracelet/x/ansi
  dependency-version: 0.11.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-05-05 09:23:27 +00:00
.github fix(ci): use available GitHub action versions 2026-05-01 09:49:38 -07:00
cache refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
config refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
control feat(control): add app metadata and status contracts 2026-05-01 15:23:03 -07:00
docs docs: clarify go module publishing 2026-05-03 23:08:19 -07:00
mirror refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
output refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
progress feat(progress): add ci-safe tracker 2026-05-02 19:19:53 -07:00
snapshot feat(snapshot): report import table progress 2026-05-03 23:49:58 -07:00
state refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
store refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
tui fix(tui): open archive groups by recency 2026-05-04 01:28:54 -07:00
.editorconfig chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitattributes chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitignore chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
CHANGELOG.md docs: note sectioned tui context panes 2026-05-04 01:13:33 -07:00
CONTRIBUTING.md chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
go.mod build(deps): bump github.com/charmbracelet/x/ansi from 0.11.6 to 0.11.7 2026-05-05 09:23:27 +00:00
go.sum build(deps): bump github.com/charmbracelet/x/ansi from 0.11.6 to 0.11.7 2026-05-05 09:23:27 +00:00
LICENSE chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
Makefile ci: add validation and publishing metadata 2026-05-01 08:43:29 -07:00
README.md docs: clarify go module publishing 2026-05-03 23:08:19 -07:00

README.md

crawlkit

Shared Go infrastructure for local-first crawler archives.

crawlkit is not a universal Slack, Discord, Notion, or GitHub crawler. It is the reusable foundation beneath those tools: SQLite hygiene, TOML config defaults, portable JSONL/Gzip packing, git-backed snapshot sharing, sync state, CLI output helpers, a shared terminal explorer, and safe desktop-cache snapshot utilities.

Install

go get github.com/vincentkoc/crawlkit@latest

Go packages are published by tagging this repository. There is no separate package registry step. See docs/publishing.md for the release commands.

Packages

  • config: standard TOML config paths, runtime dirs, and token diagnostics.
  • store: SQLite open/read-only/transaction/query helpers.
  • snapshot: manifest.json plus JSONL/Gzip table snapshot export and import.
  • mirror: clone/init/pull/commit/push helpers for private snapshot repos.
  • state: generic crawler cursor and freshness records.
  • output: text/json/log output helpers.
  • tui: shared terminal archive explorer with gitcrawl-style responsive panes, entity/member/detail lanes, compact sortable headers, mouse selection, floating right-click actions, sorting/filtering, and local/remote source status.
  • cache: safe read-only local cache snapshot helpers.

Safety

Library tests use temporary directories. They do not touch app runtime stores such as ~/.config/gitcrawl, ~/.slacrawl, ~/.discrawl, or ~/.notcrawl.