Evaluations
The tools in this repository were designed with measurements, not guesses. This section collects the evidence.
Skill vs. MCP serverThe same data access built twice - as an Agent Skill and as an MCP server - and an honest comparison of the two.
Eval runs
Section titled “Eval runs”Eval runs testing how well LLMs use the tools: timestamp labelling, field shape, and what changes comprehension and what doesn’t. Each run is a dated, immutable snapshot: a new finding adds a new run rather than editing an old one.
Hourly multi-series shape refinement — results (2026-07-11)Should the forecast JSON include a wind-gust field, and should temperature be nested or flattened? A 2×2 factorial test across model tiers says: ship both changes — neither hurts comprehension.
Forecast-JSON comprehension eval — results (2026-07-09)Does labeling forecast timestamps in local time vs. UTC change whether LLMs answer time-based questions correctly? A 13-provider sweep says yes — local time wins decisively.
The eval suite lives in
packages/meteoswiss-forecast-evals.