we are planning a large-scale A/B test on the checkout flow starting next week. since we expect a significant influx of visitors from our upcoming sale, i am worried about
sample size pollution . if the traffic composition changes drastically during the middle of the test, does it invalidate the results? i am trying to decide between running the test for a fixed duration or waiting until we reach a specific significance threshold.
>most people suggest sticking to full weekly cyclesis there a way to use
window.location.search
to segment users by campaign source and keep the data clean? i want to avoid
discarding all the data from the sale period because that would take too long. should i just focus on
statistical significance alone or is there a better way to normalize the incoming traffic?