Pipeline Apps
The IONIS data pipeline ships 19 CLI apps across three packages, plus one
external tool used for validation. All pipeline apps use the ch-go native
protocol for ClickHouse connectivity with LZ4 compression. The PSK Reporter
collector uses MQTT for real-time spot streaming.
WSPR Ingestion
| App |
Type |
Package |
Description |
| wspr-turbo |
Go |
ionis-apps |
Streaming .gz → ClickHouse, 22 Mrps (16 workers) |
| wspr-shredder |
Go |
ionis-apps |
Uncompressed CSV → ClickHouse, 21 Mrps |
| wspr-parquet-native |
Go |
ionis-apps |
Parquet → ClickHouse, 8.4 Mrps |
| wspr-download |
Go |
ionis-apps |
Parallel archive downloader from wsprnet.org |
Solar Pipeline
| App |
Type |
Package |
Description |
| solar-download |
Go |
ionis-apps |
Multi-source downloader (10 NOAA/SIDC endpoints) |
| solar-ingest |
Go |
ionis-apps |
Solar/geomagnetic data ingester |
| solar-backfill |
Go |
ionis-apps |
GFZ Potsdam historical SSN/SFI/Kp (1932–present) |
| solar-refresh |
Shell |
ionis-apps |
Download + truncate + ingest pipeline |
| solar-live-update |
Shell |
ionis-apps |
Now-Casting updater, 15-min cron |
| solar-history-load |
Shell |
ionis-apps |
Training data loader, 6-hour cron |
Contest & RBN
| App |
Type |
Package |
Description |
| contest-download |
Go |
ionis-apps |
CQ/ARRL Cabrillo log downloader (15 contests) |
| contest-ingest |
Go |
ionis-apps |
Cabrillo V2/V3 parser → ClickHouse with enrichment |
| rbn-download |
Go |
ionis-apps |
RBN daily ZIP downloader (2009–present) |
| rbn-ingest |
Go |
ionis-apps |
RBN ZIP → CSV → ClickHouse ingester |
PSK Reporter
| App |
Type |
Package |
Description |
| pskr-collector |
Go |
ionis-apps |
MQTT real-time spot collector → gzip JSONL (~22M spots/day) |
Database
| App |
Type |
Package |
Description |
| ionis-db-init |
Shell |
ionis-core |
Initialize ClickHouse schemas (idempotent) |
| ionis-env |
Shell |
ionis-core |
Standardized environment variables (sourceable) |
| db-validate |
Go |
ionis-apps |
Validate ClickHouse table row counts |
CUDA Engine
| App |
Type |
Package |
Description |
| bulk-processor |
CUDA |
ionis-cuda |
Float4 embedding generator → wspr.silver |
Validation (External)
| App |
Type |
Package |
Description |
| voacapl |
Fortran |
External (local build) |
NTIA/ITS HF propagation prediction engine |