Working notes on scheduling and node pools. Mostly for myself.
Grouping nodes by capability rather than by hostname turned out to be the single change that made capacity planning tractable. Before that, every scheduling decision needed a lookup table maintained by hand, and the table was wrong roughly as often as it was right.
A drain that respects in-flight work is slower than one that does not, and the difference shows up as a long tail. The naive approach finishes in seconds and loses requests; the careful one takes minutes and loses none. Setting gracePeriod above the p99 request duration removed the tail entirely in our case.
Per-pool metrics beat per-node metrics for anything above a handful of machines. Node-level dashboards look impressive and tell you nothing when the node count changes daily.