#rep0rter
2026-10-02
Collect Code for Japan's published Slack archive from its public static HTML pages without Slack OAuth. Preserve namespaced message and author identities, direct message links, threads and reactions; enforce public-channel boundaries and existing exclusions, and resume paginated scans within the shared request budget. Enable the source in Production reporting through an additive public feed setting that preserves the existing private configuration and credentials. Add source catalog, probe support and operational documentation. Validation: full offline suite passed (1,112 tests), followed by the new collector and configuration tests (41 passed). An isolated live scan completed both public channels in 11 requests with no source errors. Native Worker cron remains disabled.
Collect Code for Japan's published Slack archive from its public static HTML pages without Slack OAuth. Preserve namespaced message and author identities, direct message links, threads and reactions; enforce public-channel boundaries and existing exclusions, and resume paginated scans within the shared request budget. Enable the source in Production reporting through an additive public feed setting that preserves the existing private configuration and credentials. Add source catalog, probe support and operational documentation. Validation: full offline suite passed (1,112 tests), followed by the new collector and configuration tests (41 passed). An isolated live scan completed both public channels in 11 requests with no source errors. Native Worker cron remains disabled.
Allow Production reporting to backfill Code for Japan's public Slack archive with an optional `cfj_days` input (1–365 days), while preserving the normal collection window for other sources and the shared request budgets. Stored messages retain their original dates and remain subject to the existing editorial age gate. Restart an unfinished narrow scan when its lookback widens so intermediate chunks are revisited rather than silently skipped. A blank workflow input preserves existing configuration for hourly reporting. Validation: 47 collector and runner tests passed, including widened-window resumption, source isolation, configuration precedence and invalid-input rejection. Deployment will follow exact-main Offline tests, then a production report with `cfj_days=90` will verify ingestion.
Allow Production reporting to backfill Code for Japan's public Slack archive with an optional `cfj_days` input (1–365 days), while preserving the normal collection window for other sources and the shared request budgets. Stored messages retain their original dates and remain subject to the existing editorial age gate. Restart an unfinished narrow scan when its lookback widens so intermediate chunks are revisited rather than silently skipped. A blank workflow input preserves existing configuration for hourly reporting. Validation: 47 collector and runner tests passed, including widened-window resumption, source isolation, configuration precedence and invalid-input rejection. Deployment will follow exact-main Offline tests, then a production report with `cfj_days=90` will verify ingestion.
2026-10-04
@zwec181818 has joined the channel
2026-10-18
Production reporting currently rejects the 56.4 MB compressed generation with HTTP 413. Transfer protocol 2 sends manifests, database ranges and individual files within bounded requests, reuses stored assets, and switches only complete generations. Native publication spools files through SQLite and reuses cached cards to avoid complete media buffers. This also preserves checkpointed runner changes after abandoned leases and durably retries public withdrawals after publication failures. Existing account retry behavior, exclusion records, delivery state and disabled native publication cron are retained. Validation: 1,160 offline tests passed. Three independent review rounds closed envelope/path and recovery findings; the final round passed 181 targeted tests and 10 independent fault probes. Production validation still requires exact-main CI, frontend-first Workers deployment and a successful natural reporting cycle; offline tests do not establish Workers heap or SQL CPU margins.
Production reporting currently rejects the 56.4 MB compressed generation with HTTP 413. Transfer protocol 2 sends manifests, database ranges and individual files within bounded requests, reuses stored assets, and switches only complete generations. Native publication spools files through SQLite and reuses cached cards to avoid complete media buffers. This also preserves checkpointed runner changes after abandoned leases and durably retries public withdrawals after publication failures. Existing account retry behavior, exclusion records, delivery state and disabled native publication cron are retained. Validation: 1,160 offline tests passed. Three independent review rounds closed envelope/path and recovery findings; the final round passed 181 targeted tests and 10 independent fault probes. Production validation still requires exact-main CI, frontend-first Workers deployment and a successful natural reporting cycle; offline tests do not establish Workers heap or SQL CPU margins.
A transient HTTP 500 from the deployed Worker health endpoint discarded the natural hourly cycle before reporting dependencies were installed. Retry only that public GET up to three times for HTTP 500/502/503/504 or known timeout/reset failures, with bounded response size and fixed backoff. Exact commit, readiness, native runtime and scheduler checks remain mandatory after a successful response. Validation: nine offline entrypoint tests and eleven independent offline probes passed; unchanged deployment/version guard behavior was checked. This does not establish the underlying HTTP 500 cause or reporting recovery. Release requires exact merged-main Offline tests, the documented frontend-first deployment and a natural hourly reporting cycle.
A transient HTTP 500 from the deployed Worker health endpoint discarded the natural hourly cycle before reporting dependencies were installed. Retry only that public GET up to three times for HTTP 500/502/503/504 or known timeout/reset failures, with bounded response size and fixed backoff. Exact commit, readiness, native runtime and scheduler checks remain mandatory after a successful response. Validation: nine offline entrypoint tests and eleven independent offline probes passed; unchanged deployment/version guard behavior was checked. This does not establish the underlying HTTP 500 cause or reporting recovery. Release requires exact merged-main Offline tests, the documented frontend-first deployment and a natural hourly reporting cycle.
Cold Worker startup and expired-runner native refresh currently read all durable database BLOBs into one SQL cursor, then join every page into another full database buffer. Restore both paths through one validated snapshot in batches of at most 64 pages (256 KiB), writing pages sequentially while preserving the complete runner page-cache API and existing operational health checks. Validate page shape and the 32 MiB limit before hydration; stage the database and exclusions ledger together, poison the Runtime on failure, and publish the cache/generation only after successful restoration. Recovery and failure states are documented. Warm refresh can retain old/new files and caches concurrently; this does not establish lower total peak heap or diagnose the observed preflight HTTP 500. Validation: 39 final startup tests, 16 independent SQLite/ownership/failure probes, and source integrity checks passed. Nine selected existing integrations passed before a final change limited to cleanup-exception chaining; full Offline tests remain required. No production data or credentials were used. Stacked on <https://github.com/rep0rter/rep0rter/pull/61|#61>. Release requires its base merged, exact-main Offline tests, clean frontend-first deployment, and natural reporting acceptance. Not deployed.
Cold Worker startup and expired-runner native refresh currently read all durable database BLOBs into one SQL cursor, then join every page into another full database buffer. Restore both paths through one validated snapshot in batches of at most 64 pages (256 KiB), writing pages sequentially while preserving the complete runner page-cache API and existing operational health checks. Validate page shape and the 32 MiB limit before hydration; stage the database and exclusions ledger together, poison the Runtime on failure, and publish the cache/generation only after successful restoration. Recovery and failure states are documented. Warm refresh can retain old/new files and caches concurrently; this does not establish lower total peak heap or diagnose the observed preflight HTTP 500. Validation: 39 final startup tests, 16 independent SQLite/ownership/failure probes, and source integrity checks passed. Nine selected existing integrations passed before a final change limited to cleanup-exception chaining; full Offline tests remain required. No production data or credentials were used. Stacked on <https://github.com/rep0rter/rep0rter/pull/61|#61>. Release requires its base merged, exact-main Offline tests, clean frontend-first deployment, and natural reporting acceptance. Not deployed.
The deployment previously uploaded the frontend before engine preparation discovered that uv was missing. Check the required executables and uv >= 0.12.3 before source preparation or either upload, and document a command-scoped PATH for an independently verified uv installation. Preserve the existing exact-main CI gate and frontend-first deployment order. Validation: 20 mocked stdlib tests passed; independent review passed six additional cases. Full Offline tests and client checks remain for CI; local pytest is unavailable.
The deployment previously uploaded the frontend before engine preparation discovered that uv was missing. Check the required executables and uv >= 0.12.3 before source preparation or either upload, and document a command-scoped PATH for an independently verified uv installation. Preserve the existing exact-main CI gate and frontend-first deployment order. Validation: 20 mocked stdlib tests passed; independent review passed six additional cases. Full Offline tests and client checks remain for CI; local pytest is unavailable.
Slack catch-up can discard its pending cursor and repeat indefinitely when a one-page scan ends at a fractional timestamp. The next run receives the same rounded boundary page, but the HTML completeness proof only sees that run's snapshots and forgets the newer interval already committed with the cursor. Keep the original resumed coverage boundary fixed while scanning older pages, persist repeated-page snapshots, and retain the pending cursor when coverage remains unknown. The existing complete-month HTML/count/timestamp checks still require every remaining eligible older row; saturated pages, missing rows, cross-month windows, budget exhaustion and ambiguous empty pages remain degraded. Request budgets and editorial/publication policy are unchanged. If a snapshot transaction fails, the channel error record now preserves the prior durable watermark, pending interval and success time instead of installing uncommitted candidate progress. A later metadata failure conservatively replays already stored events; an uncertain remote checkpoint still raises the existing fatal DurabilityError and aborts the cycle. Expose four fixed numeric Slack catch-up counters in the existing aggregate acceptance record so scheduled runs can show committed progress without logging channel IDs, cursor values or source content. Validation: the original implementation failed five of six initial synthetic regressions, including a two-run SQLite persistence case. Four additional transaction-failure cases also failed before their fix. After both fixes, 185 focused offline collector, archive, acceptance, source-isolation, Worker runtime and runner-checkpoint tests pass. Added coverage includes one-page convergence, missing older rows, an immutable resumed boundary, empty/budget failures, cross-month rejection, sanitized progress, partial SQL rollback and fatal checkpoint uncertainty. Normal full Offline tests must pass before merge and exact-main tests before production deployment. Production currently runs <https://github.com/rep0rter/rep0rter/commit/26eaeda0aad742bd23431a7d8a16796a0405d573|26eaeda>. Its natural schedule completed collection/publication with 163 collected and zero posted, but still reported 16 Slack coverage gaps. This PR fixes a reproduced cause; it does not assert those production gaps have cleared. Verify subsequent natural schedules after deployment without manually dispatching paid reporting or delivery.
Slack catch-up can discard its pending cursor and repeat indefinitely when a one-page scan ends at a fractional timestamp. The next run receives the same rounded boundary page, but the HTML completeness proof only sees that run's snapshots and forgets the newer interval already committed with the cursor. Keep the original resumed coverage boundary fixed while scanning older pages, persist repeated-page snapshots, and retain the pending cursor when coverage remains unknown. The existing complete-month HTML/count/timestamp checks still require every remaining eligible older row; saturated pages, missing rows, cross-month windows, budget exhaustion and ambiguous empty pages remain degraded. Request budgets and editorial/publication policy are unchanged. Expose four fixed numeric Slack catch-up counters in the existing aggregate acceptance record so scheduled runs can show committed progress without logging channel IDs, cursor values or source content. Validation: the original implementation failed five of six initial synthetic regressions, including a two-run SQLite persistence case. After the fix, 149 focused offline collector, archive, acceptance and source-isolation tests pass. Added coverage includes one-page convergence, missing older rows, an immutable resumed boundary, empty/budget failures, cross-month rejection and sanitized progress output. Normal full Offline tests must pass before merge and exact-main tests before production deployment. Production currently runs <https://github.com/rep0rter/rep0rter/commit/26eaeda0aad742bd23431a7d8a16796a0405d573|26eaeda>. Its natural schedule completed collection/publication with 163 collected and zero posted, but still reported 16 Slack coverage gaps. This PR fixes a reproduced cause; it does not assert those production gaps have cleared. Verify subsequent natural schedules after deployment without manually dispatching paid reporting or delivery.
2026-10-22
Three production Slack channels remain collection-unhealthy even after successful reporting: PHP omits a trailing zero in rounded HTML anchors, a resumed scan spans two months, and a current-month archive contains only subtype rows filtered out by JSON. Verify these windows using exact embedded timestamps and raw monthly counts, with at most three HTML months and the existing shared request budget. Missing rows, malformed pages, absent boundary months, and exhausted budgets retain the gap and pending cursor. Validation: 1,247 offline tests and 36 subtests pass, including 14 new synthetic regression cases. Bounded read-only probes of the three affected public channels now prove the sampled windows complete with three requests each; no production state or health flags were modified. Deployment and a genuine reporting cycle are still required before claiming production collection is healthy.
Three production Slack channels remain collection-unhealthy even after successful reporting: PHP omits a trailing zero in rounded HTML anchors, a resumed scan spans two months, and a current-month archive contains only subtype rows filtered out by JSON. Verify these windows using exact embedded timestamps and raw monthly counts, with at most three HTML months and the existing shared request budget. Missing rows, malformed pages, absent boundary months, and exhausted budgets retain the gap and pending cursor. Validation: 1,247 offline tests and 36 subtests pass, including 14 new synthetic regression cases. Bounded read-only probes of the three affected public channels now prove the sampled windows complete with three requests each; no production state or health flags were modified. Deployment and a genuine reporting cycle are still required before claiming production collection is healthy.
2026-10-23
A natural reporting run exhausted its 30-minute job limit before collection started: downloading 61.2 MB of CJK fonts from the Azure Ubuntu mirror took 21m50s at 46.7 kB/s, and browser system dependencies were still downloading when the job was canceled. Use the official HTTPS Ubuntu archive for the runner's Ubuntu packages and pin the Python reporting/test runner to Ubuntu 24.04, matching the failed run's actual Noble image. A shared local action keeps the existing Python/CJK/Playwright installation and runs a Chromium render smoke before reporting. Offline CI uses that same action to validate the real bootstrap. Collection code, deployed-source checks, the 30-minute production limit, budgets, checkpoints, AI and notification behavior are unchanged. Validation: 1,247 local tests and 36 subtests pass; YAML, shell and embedded Python parse; mirror manifest preservation and idempotence pass. Exact GitHub bootstrap/CI is required before merge/deployment. No production workflow was manually dispatched and the previous canceled run is retained. Three Slack channel recovery and actual collection budgets remain pending a successful natural cycle.
A natural reporting run exhausted its 30-minute job limit before collection started: downloading 61.2 MB of CJK fonts from the Azure Ubuntu mirror took 21m50s at 46.7 kB/s, and browser system dependencies were still downloading when the job was canceled. Use the official HTTPS Ubuntu archive for the runner's Ubuntu packages and pin the Python reporting/test runner to Ubuntu 24.04, matching the failed run's actual Noble image. A shared local action keeps the existing Python/CJK/Playwright installation and runs a Chromium render smoke before reporting. Offline CI uses that same action to validate the real bootstrap. Collection code, deployed-source checks, the 30-minute production limit, budgets, checkpoints, AI and notification behavior are unchanged. Validation: 1,247 local tests and 36 subtests pass; YAML, shell and embedded Python parse; mirror manifest preservation and idempotence pass. Exact GitHub bootstrap/CI is required before merge/deployment. No production workflow was manually dispatched and the previous canceled run is retained. Three Slack channel recovery and actual collection budgets remain pending a successful natural cycle.
*Problem* • Story pages emitted a self canonical, four-language hreflang and no robots directive even when an edition was incomplete. Example: `/posts/184/index.html` shows _English translation pending_; its zh-TW/ja/ko editions only show the saved fallback text, yet all four were indexable and listed each other. • `/robots.txt` and `/sitemap.xml` returned 404. The frontend served every `.xml` as `application/rss+xml`. *Change* • Completeness reuses the page's existing `translated` decision (`_display_text`). Incomplete story editions stay readable with their self canonical, but get `<meta name="robots" content="noindex">` and no hreflang. Complete editions list only complete editions. A later translation backfill restores the edition on the next build. • New `sitemap.xml`: four homepages + complete editions of each story's latest revision, at canonical `index*.html` URLs. No `lastmod` (translation backfills have no reliable change time). Superseded revisions, source/tag timelines, aliases, withdrawn stories, source examples and account pages are excluded. Emergency withdrawal scrubs remove story URLs from it. • New `robots.txt`: `Allow: /` + `Sitemap:`. Sign-in/writing/project pages stay crawlable so their existing `noindex` is honored. • Frontend and engine serve `sitemap.xml` as `application/xml`; feeds remain RSS. • Docs: `docs/operations.md#search-indexing`, `docs/implementation-notes.md` (GSC-20261023). *Tests* • New: incomplete editions noindex/no hreflang and restoration after translation; sitemap only lists indexable canonical public pages, no lastmod; robots points to sitemap; superseded excluded; emergency withdrawal scrubs sitemap; Worker content types. • Local: `tests/test_site.py tests/test_cloudflare_web.py` (50 passed) and withdrawal/native/multilingual/publication subsets (249 passed). Full suite runs in Offline tests. Search Console submission and Google-side verification are handled separately; submission is not indexing. :robot_face: Generated with <https://claude.com/claude-code|Claude Code>
*Problem* • Story pages emitted a self canonical, four-language hreflang and no robots directive even when an edition was incomplete. Example: `/posts/184/index.html` shows _English translation pending_; its zh-TW/ja/ko editions only show the saved fallback text, yet all four were indexable and listed each other. • `/robots.txt` and `/sitemap.xml` returned 404. The frontend served every `.xml` as `application/rss+xml`. *Change* • Completeness reuses the page's existing `translated` decision (`_display_text`). Incomplete story editions stay readable with their self canonical, but get `<meta name="robots" content="noindex">` and no hreflang. Complete editions list only complete editions. A later translation backfill restores the edition on the next build. • New `sitemap.xml`: four homepages + complete editions of each story's latest revision, at canonical `index*.html` URLs. No `lastmod` (translation backfills have no reliable change time). Superseded revisions, source/tag timelines, aliases, withdrawn stories, source examples and account pages are excluded. Emergency withdrawal scrubs remove story URLs from it. • New `robots.txt`: `Allow: /` + `Sitemap:`. Sign-in/writing/project pages stay crawlable so their existing `noindex` is honored. • Frontend and engine serve `sitemap.xml` as `application/xml`; feeds remain RSS. • Docs: `docs/operations.md#search-indexing`, `docs/implementation-notes.md` (GSC-20261023). *Tests* • New: incomplete editions noindex/no hreflang and restoration after translation; sitemap only lists indexable canonical public pages, no lastmod; robots points to sitemap; superseded excluded; emergency withdrawal scrubs sitemap; Worker content types. • Local: `tests/test_site.py tests/test_cloudflare_web.py` (50 passed) and withdrawal/native/multilingual/publication subsets (249 passed). Full suite runs in Offline tests. Search Console submission and Google-side verification are handled separately; submission is not indexing. :robot_face: Generated with <https://claude.com/claude-code|Claude Code>