Treat an A/B test as an multi-armed bandit problem. First we need to record the performance for each option. Then every time we are faced with a decision, calculate the reward for each option, pick the best one. Leave a small chance (e.g. 10%) for exploration where a random option is selected.

If you have two source of measurements, how can you combine them to get a more accurate measurement?

In short, you take the weighted average with the weight=1/variance.

Please also read https://news.ycombinator.com/item?id=36971975 for the missing (but important) bits.

the Dunning-Kruger effect has nothing to do with human psychology.1 It is a statistical artifact — a stunning example of autocorrelation.