openclaw/crawlkit
openclaw/crawlkit
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases 2 Wiki Activity
[BREAKGLASS] Append-only mirror of github.com/openclaw/crawlkit
159 Commits 5 Branches 25 Tags 500 KiB
Go 100%
bbc8e09c07
Go to file
dependabot[bot] bbc8e09c07
build(deps): bump github.com/mattn/go-isatty from 0.0.20 to 0.0.22 (#4) 
Bumps [github.com/mattn/go-isatty](https://github.com/mattn/go-isatty) from 0.0.20 to 0.0.22.
- [Commits](https://github.com/mattn/go-isatty/compare/v0.0.20...v0.0.22)

---
updated-dependencies:
- dependency-name: github.com/mattn/go-isatty
  dependency-version: 0.0.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-05-07 02:45:10 -07:00
.github Merge pull request #5 from openclaw/ci-security-baseline 2026-05-06 01:55:23 -07:00
cache refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
config refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
control feat(control): add app metadata and status contracts 2026-05-01 15:23:03 -07:00
docs docs: document crawlkit adoption 2026-05-05 19:16:51 -07:00
mirror feat(mirror): add safer share repo helpers 2026-05-05 17:18:53 -07:00
output refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
progress feat(progress): add ci-safe tracker 2026-05-02 19:19:53 -07:00
snapshot feat(snapshot): report import table progress 2026-05-03 23:49:58 -07:00
state feat(state): add legacy sync adapters 2026-05-05 17:21:30 -07:00
store refactor: rename public package nouns 2026-05-01 12:30:13 -07:00
tui fix(tui): open archive groups by recency 2026-05-04 01:28:54 -07:00
.editorconfig chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitattributes chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
.gitignore chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
AGENTS.md docs: define crawlkit app boundary 2026-05-05 17:16:57 -07:00
CHANGELOG.md docs: prepare v0.4.1 release notes 2026-05-06 14:05:30 -07:00
CONTRIBUTING.md chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
go.mod build(deps): bump github.com/mattn/go-isatty from 0.0.20 to 0.0.22 (#4) 2026-05-07 02:45:10 -07:00
go.sum build(deps): bump github.com/mattn/go-isatty from 0.0.20 to 0.0.22 (#4) 2026-05-07 02:45:10 -07:00
LICENSE chore: bootstrap crawlkit module 2026-05-01 08:34:01 -07:00
Makefile ci: add validation and publishing metadata 2026-05-01 08:43:29 -07:00
README.md docs: document crawlkit adoption 2026-05-05 19:16:51 -07:00

README.md

crawlkit

Shared Go infrastructure for local-first crawler archives.

crawlkit is not a universal Slack, Discord, Notion, or GitHub crawler. It is the reusable foundation beneath those tools: SQLite hygiene, TOML config defaults, portable JSONL/Gzip packing, git-backed snapshot sharing, sync state, CLI output helpers, control/status metadata, a shared terminal explorer, and safe desktop-cache snapshot utilities.

Install

go get github.com/vincentkoc/crawlkit@latest

Go packages are published by tagging this repository. There is no separate package registry step. See docs/publishing.md for the release commands. See docs/boundary.md for the crawlkit-versus-app ownership boundary.

Packages

  • config: standard TOML config paths, runtime dirs, and token diagnostics.
  • store: SQLite open/read-only/transaction/query helpers.
  • snapshot: manifest.json plus JSONL/Gzip table snapshot export and import.
  • mirror: clone/init/pull/commit/push helpers for private snapshot repos.
  • state: generic crawler cursor and freshness records.
  • output: text/json/log output helpers.
  • control: crawl app metadata, command manifests, status payloads, and database inventory for launchers and automation.
  • tui: shared terminal archive explorer with gitcrawl-style responsive panes, entity/member/detail lanes, compact sortable headers, mouse selection, floating right-click actions, sorting/filtering, and local/remote source status.
  • cache: safe read-only local cache snapshot helpers.

Downstream apps

  • gitcrawl and discrawl consume crawlkit on main.
  • slacrawl and notcrawl consume crawlkit on their feat/use-crawlkit integration branches until those app rewires are merged.
  • The apps keep provider schemas, auth, desktop/API parsing, privacy filters, and user-facing CLI contracts. crawlkit owns only the reusable mechanics.

Safety

Library tests use temporary directories. They do not touch app runtime stores such as ~/.config/gitcrawl, ~/.slacrawl, ~/.discrawl, or ~/.notcrawl.