I keep seeing the argument that Black Friday and Cyber Monday promos just train consumers to wait for the discount and pull their purchases forward into those few days. That might be true. But how could you test it?
1. A classic geo test
Some cities see the messaging and product promos and others don't. Compare sales per user in the exposed cities against the control cities. The big risks are operational complexity and not reaching statistical significance, since you'd only want to run it for a few weeks.
2. An email and push holdout
Isolate a group of users and don't send them any discount messaging. Compare that group's sales against users who did receive the messaging.
3. Product holdouts
Hold certain products out of the discounts. Compare the spike in purchases against discounted products, then look at purchase volume after Cyber Monday. Did people come back to buy the undiscounted item?
With all of these, you want a sample big enough to show statistical significance but not so big that it meaningfully hurts sales. That's why testing during strong seasonal periods can be scary: the stakes are just too high.
Has anyone found a way to test this that feels right without destroying holiday sales?