ARCHITECTURE

A short, inspectable local pipeline

A standard Python package reads local files, normalizes records, aggregates them, and renders terminal or HTML output.

Overview

Local logs
JSON / JSONL

readers
normalized records

aggregate
range and periods

reporters
terminal / HTML

./out
local artifacts

Package layout

Path	Responsibility
`src/tokens_cli/cli.py`	Argument validation, runtime configuration, range routing, diagnostics.
`config.py`	Source paths, system timezone, output and platform cache locations, aliases.
`doctor.py`	Metadata-only environment diagnostics.
`readers.py`	Safe file scanning, source parsers, normalized records, file cache.
`aggregate.py`	Date filtering and day/week/month summaries.
`dashboard_payload.py`	Single-pass dashboard aggregates, pseudonyms, replay and provenance data.
`dashboard_wire.py`	Versioned compact dashboard payload encoding.
`live_dashboard.py`	Loopback-only HTTP service, safe file-signature checks, ETag snapshots, and refresh synchronization.
`report_*.py`	Terminal, static HTML, and dashboard rendering.
`dashboard_assets/`	Packaged HTML, CSS, and vanilla JavaScript assets.

Data flow

Configure: choose sources, timezone, output, range, and cache behavior.
Scan safely: reject symlinks, non-regular files, oversized files, and oversized JSONL lines.
Normalize: preserve each source total while mapping common fields.
Aggregate: calculate period, model, source, project, session, and composition views.
Pseudonymize when requested: replace identifiers before dashboard entities enter the serialized payload.
Render: print ANSI output, write a local HTML artifact, or serve the dashboard on loopback.
Refresh live views: compare safe file signatures, reuse unchanged parsed records, and publish a new ETag snapshot only when report data changes.

Operational boundaries

Runtime dependencies: the core pipeline uses the Python standard library; Windows installs include the data-only tzdata package for IANA timezone support.
Package data: dashboard assets ship inside the wheel.
Reports: default to the current working directory, never site-packages.
Cache: platform user cache directory, written atomically with private permissions where supported.
Browser opening: standard library URL opening; failure does not invalidate output.
Frontend: no framework, CDN, or remote scripts. Offline exports make no requests; live mode fetches only from its same-origin 127.0.0.1 service.
Live binding: loopback only, with strict Host and Origin checks and no LAN bind option.

Adding a source

Add a path and parser that returns the normalized record fields, register it in readers.SOURCES, expose it through CLI choices, document source accounting, and add parser, cache, aggregate, privacy, and packaging tests.

Keep the semantic boundaryA new source must not pretend missing project/session data is zero, and display code must not infer entity relationships that the source does not provide.