Make how many times a test is run before raising error flag configurable
notification_treshold: 2 # repeated-times
We can even set error and warning thresholds separately:
error_treshold: 4 # repeated-times
warning_treshold: 2 # repeated-times
The duration between trials can also be set:
error_treshold: 2mins
warning_treshold: 4mins