pinterest/secor
Secor is a service implementing Kafka log persistence observed · 2026-08-28
Health v2 · maintenance only
64/100
- Activity 71
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4523
- days_rel: n/a
- days_push: 176
- n_releases_24m: 0
Adoption not part of the score
1856 stars · 537 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
Secor is a Java service that persists Apache Kafka log messages to object stores such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and OpenStack Swift. It offers strong per-message consistency, fault tolerance, horizontal scalability, configurable size/time-based upload policies, and partitioned output paths for direct import into systems like Hive.
Use cases
- persist kafka logs to s3
- archive kafka messages to cloud object storage
- export kafka topics to hive-partitioned files
- backup kafka data to a data lake
- move kafka events into s3 for offline analytics
- exactly-once kafka-to-object-storage pipeline
- run a scalable kafka consumer on kubernetes
When to choose
- You need guaranteed, exactly-once persistence of Kafka messages into S3, GCS, Azure Blob Storage, or Swift
- You want time/partition-organized output (day/hour/minute) for direct Hive or data-lake imports
- You need a fault-tolerant, horizontally scalable Kafka consumer that can be scaled by simply adding or removing processes
When to avoid
- You need real-time stream processing or per-message low-latency delivery rather than batched persistence to object storage
- Your target sink is not one of the supported object stores (S3, GCS, Azure Blob, Swift)
- You only need a simple single-node log dump without distributed consistency guarantees
Facets
service · maturity stable
streaming etl big-data cloud-computing analytics jvm cloud self-hosted kafka apache-kafka kafka-consumer amazon-s3 google-cloud-storage azure-blob-storage openstack-swift object-storage log-persistence log-archive data-lake hive partitioned-output exactly-once java data-engineering messaging kubernetes
1 source
- readme: https://github.com/pinterest/secor · fetched 2026-08-28 · 618f36924baf
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| pinterest/secor | main | 64 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem