An alert has one job: to tell you that something needs your attention now. Every alert that fires without deserving attention makes the next one a little easier to ignore. This article walks through the design tools that keep a rule set quiet and trustworthy — thresholds, hysteresis, cooldowns and severity tiers — and then rewrites one noisy rule from start to finish.
Why noisy alerts are a design failure, not a volume problem
Noise is usually treated as an inconvenience. It is more dangerous than that. When alerts arrive too often, people start skimming, second-guessing or ignoring them. The Google SRE book states this directly in its chapter on monitoring distributed systems (external site), and sets the standard that every page should be actionable and every response should require judgment.
Healthcare has studied the same effect for years under the name alert fatigue. The AHRQ PSNet primer (external site) cites a study in which monitors on 66 adult ICU beds produced more than 2 million alerts in one month — 187 per patient per day — and notes that clinicians override the vast majority of electronic warnings, including ones labelled critical. Among the remedies it lists: remove inconsequential alerts, tier them by severity, and make only high-severity alerts interruptive.
Trading systems are not exempt. In the SEC's Knight Capital order summary (external site), an internal system sent 97 automated emails before the market opened on August 1, 2012, flagging an error in an order router. They were not designed as system alerts and were not acted on. The firm lost more than $460 million in about 45 minutes. The messages existed; the design that would have made them stand out did not.
The lesson for anyone running an ai trading bot or a research monitor: the quality of an alert is set when you write the rule, not when it fires.
Four questions to ask before saving any rule
Before a rule goes live, answer these in writing:
- What will I do when it fires? If the answer is "look and probably nothing", it is not an alert — it is a chart annotation or a journal entry.
- How often should it fire in a normal session? Write a number. A rule with no expected rate cannot be judged noisy or quiet.
- What should stop it firing? Define the reset condition, not just the trigger.
- How urgent is it compared with my other rules? Severity is relative; if everything is high, nothing is.
These questions mirror the SRE book's checklist: can I act on this now, could it wait, and will I ever ignore it knowing it is benign?
Thresholds: choose the level from the data, not from round numbers
The most common noisy rule is a threshold placed where the value spends a lot of time. A level of 100.00 on an instrument that oscillates between 99.6 and 100.4 will cross that level dozens of times in a quiet session.
Better approaches:
- Scale to volatility. Express the threshold as a multiple of recent range (for example, 1.5× the 20-bar average range) rather than a fixed price.
- Require persistence. Fire only when the condition holds for N consecutive bars. The SRE Workbook's chapter on alerting on SLOs (external site) notes that a duration requirement filters short-lived noise, but warns it adds delay and can let intermittent conditions slip through by resetting the timer. Use it, but keep N small.
- Measure precision and recall. The same chapter defines precision (the share of alerts that were significant) and recall (the share of significant events that alerted). Tightening a threshold trades one for the other. Decide which you care about per rule.
Hysteresis: separate the trigger level from the reset level
Hysteresis means a rule fires at one level and only re-arms after the value moves back past a different level. A thermostat does this so it does not click on and off every second.
For a level-cross rule:
- Trigger: price crosses above 100.00.
- Re-arm: only after price falls back below 99.70.
Small oscillations around 100.00 now produce one alert, not twenty. The gap between trigger and re-arm (the band) should be wider than the typical bar-to-bar noise. If you do not know that noise, replay a session and measure it first.
Cooldowns: limit how often one rule can speak
A cooldown is a minimum time (or number of bars) between two firings of the same rule. Hysteresis handles oscillation around a level; cooldowns handle conditions that genuinely recur, such as repeated volume spikes during a busy period.
Set the cooldown from the question "how soon after the first alert would a second one change what I do?" If the honest answer is "not within 15 minutes", the cooldown is 15 minutes. While a rule is cooling down, the engine should still record the suppressed event in the journal, so nothing is lost — it just does not interrupt you.
Severity tiers: decide what interrupts and what waits
Use few tiers and define them by the response they demand:
| Severity | Meaning | Delivery | Expected rate (illustrative) |
|---|---|---|---|
| High | Act or decide now | Interruptive in-app notification | 0–2 per session |
| Medium | Review within the hour | Notification list, no interruption | 3–8 per session |
| Low / info | Context for later review | Journal only | Unlimited |
The table is a starting point, not a standard. The important part is the last column: if a high-severity rule fires ten times a session, it is either mis-tiered or mis-specified.
Worked example: rewriting one noisy rule
The numbers below are illustrative and come from a synthetic replay, not market data.
Original rule: "Alert (high) when instrument SYN-04 moves more than 0.5% over 3 bars."
In a replay of one synthetic session (390 one-minute bars), this rule fired 46 times. Reviewing the journal, only 4 of those firings were followed by a move you would have wanted to look at. Precision: roughly 9%.
Step 1 — scale the threshold. SYN-04's average 3-bar move in the replay was about 0.35%, so 0.5% sat inside normal variation. Raising it to 0.9% (about 2.5× the average) cut firings to 19.
Step 2 — add persistence. Requiring the move to hold for 2 consecutive bars removed quick spikes that reversed immediately: 11 firings.
Step 3 — add a cooldown. A 15-bar cooldown collapsed clusters of related firings into one: 6 firings.
Step 4 — split severity. Moves above 1.5% stay high; moves between 0.9% and 1.5% become medium. Result: 2 high, 4 medium. All 4 of the moves flagged as relevant in the original review still triggered at least a medium alert.
Illustrative numbers from a synthetic replay, not market data.
Show the numbers
| Alerts that interrupt | |
|---|---|
| Original | 46 alerts |
| Scaled threshold | 19 alerts |
| + Persistence | 11 alerts |
| + Cooldown | 6 alerts |
| Severity split (high only) | 2 alerts |
Keeping the rule set healthy over time
Noise creeps back as rules accumulate. A short routine keeps it in check:
- Count firings per rule per session and compare with the expected rate you wrote down.
- Mark each high-severity alert in the journal as acted-on or not. A rule with a low acted-on rate needs revision.
- Retire silent rules. The SRE book suggests that alerting configuration exercised less than about once a quarter is a candidate for removal.
- Change one parameter at a time, and note the change, so you can tell which edit did what.
A rule set you trust is a rule set you have measured. Start with the noisiest rule you have, run it through the four steps above, and compare the counts. For how these alerts land in a review record, see Keeping a Session Journal.