Marketing

19 Oct 2026

How to A/B test promotions without sacrificing profitability

Reza Javanian

Reza Javanian

Talon.One loyalty expert

Loyalty program strategies

7 minutes to read

A promotion should not scale because redemption rises. It should scale only when a randomized holdout shows positive incremental contribution after discount cost, cannibalization, and post-promotion effects. BCG found that diligent promotion planning and execution improved one retailer's gross margin by 2 to 6 points.

This blog post will explore how to test promotions without quietly bleeding margin. We will cover:

  • Why redemption numbers overstate success: Gross lift and topline sales growth both flatter a promotion that never changed anyone's behavior.

  • How to build a test that survives scrutiny: Control design, a preplanned hypothesis, and a commercial readout all have to line up before launch.

  • The guardrails that keep a winning test profitable: Margin, cannibalization, order value, and lifetime value decide whether a "winner" is actually worth scaling.

What is promotion testing?

Promotion testing refers to the practice of running controlled experiments on promotional offers. Teams compare a variant against a control or holdout group to isolate genuinely new revenue from subsidized demand.

If you do it well, it answers two questions: which offer performs better, and is either offer worth running at all? The second question is the one most measurement setups skip.

Redemption counts and gross sales lift show that customers responded, but they don't show whether the offer changed anyone's behavior. This gap is why apparent promotion success often shrinks under closer analysis.

Why redemption numbers overstate promotion success

Redemption and gross lift measure response, not incrementality, and the gap between the two is often larger than teams expect. BCG's incrementality research reports that rigorous testing often reveals 20% to 40% of active programs produce marginal to negative lift. Customers may respond to those programs, but their sales can reflect redirected demand rather than genuinely new demand. Cross-product effects like cannibalization and stock-ups are a common reason a program that looks profitable on paper is not.

Naive before-and-after comparisons make this worse. Say a brand runs a 15% discount for one week and sales rise 20%. It looks like a clear win, but a seasonal baseline might have increased sales by 10% without the promotion. True incremental lift, in that case, is half of what the topline suggests.

A full promotion accounting must include cannibalization, stock-ups, and halo effects. Cannibalization occurs when the promoted item takes sales from similar products. Halo effects capture changes in complementary-product sales. Stock-ups happen when customers buy discounted volume and skip future full-price sales, a pattern that shows up across categories in unified incentives programs.

A randomized control improves causal inference over a before-and-after comparison, but it does not automatically capture cannibalization, halo effects, or stock-up behavior. The measurement window and outcome definition determine whether a test actually captures category cannibalization, basket halo, and post-promotion pull-forward.

How do you measure promotion incrementality?

Measuring incrementality starts with control design. It then requires a preplanned experiment and a commercial readout.

Choose a control design

Randomized holdouts are the gold standard. A random set of customers receives the offer, while a holdout group receives nothing. The difference in buying behavior is the offer's incremental effect, since seasonality and competitor activity affect both groups at the same time.

The main constraint is organizational. A holdout requires the business to avoid marketing to part of its audience, and revenue-focused teams feel that opportunity cost immediately.

Geo-based tests work when teams cannot randomize individual customers. Marketers can expose selected markets to a promotion while matched control markets sit it out. This structure suits in-store promotions, broadcast campaigns, and channels without individual targeting.

Once teams complete several randomized tests, uplift modeling can identify who was worth targeting. Uplift models sort customers into four groups:

  • Persuadables: These customers buy only if they receive the offer.

  • Sure things: These customers buy whether they receive the offer or not.

  • Lost causes: These customers will not buy in either scenario.

  • Sleeping dogs: These customers become less likely to buy after receiving the offer.

Persuadables are the segment most likely to create a positive incremental response from an offer. Targeting decisions, though, should optimize expected incremental contribution rather than purchase probability alone. Traditional response models often target sure things because they cannot distinguish them from persuadables. Promotional budgets then end up paying for sales that were already likely.

Plan the experiment

A strong test starts with a written hypothesis. State the expected behavior change, target audience, offer cost, primary metric, and guardrails before launch. This prevents teams from redefining success after seeing the results.

Confirm these four things before launch:

  • Randomization unit: Customer-level assignment works when customers identify themselves consistently across sessions. Session-level assignment can contaminate results if the same customer sees multiple variants. Store or market assignment fits channels where individual identification is unavailable.

  • Stable assignment: Keep assignment stable for the test's duration. A customer who enters the control group should remain there, since switching exposure during the experiment reduces comparability and can blur the offer's true effect.

  • Sample size: Sample-size planning begins with the current baseline and the smallest useful improvement. Smaller detectable effects require larger samples because ordinary variation can hide modest changes.

  • Primary metric: Estimate sample needs using the primary metric, with redemption as a secondary measure. Conversion may stabilize before gross margin or revenue per visitor does, so the test must run until the commercial outcome has enough observations for a meaningful comparison.

Under the test assumptions, statistical significance indicates whether the observed difference is unlikely to reflect random variation. Profitability requires a separate commercial assessment, since a narrow confidence interval around a small loss still describes a losing promotion.

Avoid checking results every few hours and stopping when a variant briefly leads. Repeated early decisions increase the chance of selecting noise. Set the evaluation point before launch, then review the full result alongside all guardrails.

Judge commercial value

A hypothetical test splits 20,000 visitors evenly between a control and a discounted variant. Assume variable product cost is $65 per order in both groups:

  • Control: The control converts 500 orders at a 5% rate and $100 average order value, generating $35 contribution per order, or $17,500 in total contribution.

  • Variant: The discounted variant converts 550 orders at a 5.5% rate, but average order value falls to $90, generating $25 contribution per order, or $13,750 in total contribution.

  • Net effect: The offer creates 50 additional orders but reduces total contribution by $3,750.

A conversion-only dashboard would call the variant a winner. A revenue-per-visitor view would call it close, since revenue per visitor only drops from $5.00 to $4.95. Contribution is what actually rejects it: a winning variant must clear its guardrail metrics, not just its conversion rate, before it can scale.

Which guardrail metrics keep a winning test profitable?

Guardrails track outcomes you refuse to let slide. If a variant improves the primary metric but breaks a margin guardrail, reject it. Four useful guardrails for promotions are:

  • Promotional gross margin: Margin-adjusted return on investment (ROI) matters most in grocery and quick service restaurants (QSR), where discounts can exceed the gross margin on promoted items.

  • Cannibalization rate: High cannibalization means customers may switch from full-margin items to discounted ones. Total basket profit can drop even while volume grows, so profit and loss (P&L) does not improve.

  • Average order value: Discounts that replace full-price orders can create a conversion win while lowering average order value. In QSR terms, higher guest count helps little if the average check falls.

  • Customer lifetime value: Compare lifetime value by acquisition cohort. If promoted customers generate $150 against $200 for full-price customers, the offer is buying lower-quality demand.

For many ecommerce promotion tests, revenue per visitor is a useful diagnostic because it combines conversion rate and average order value in one number. It should not be the final decision metric. The decision should rest on incremental contribution per eligible customer, after discount cost, product cost, and other variable costs.

The gross-margin arithmetic is unforgiving. A 30% discount on an item with a 25% gross margin produces a negative gross profit on that item, assuming cost stays the same. A product with a 35% margin has $35 gross profit for every $100 of original price. After a 30% discount, it sells for $70 and contributes $5, about a 7.1% contribution margin on that discounted price.

The overall promotion can still be economically rational. It requires enough incremental basket margin, lifecycle value, or inventory benefit to offset the per-unit loss.

Wisp, a telehealth provider in reproductive and sexual health, shows this discipline in a category where trust matters as much as price. According to Talon.One's Wisp case study, the company uses a bundling strategy to help patients discover complementary products instead of leaning on deep discounts. That approach pairs with targeted incentives that create real patient value. Because the category involves sensitive personal data, the team treated compliance and data security as non-negotiable from the start rather than an afterthought.

wisp-promotions

How Wisp uses incentives to boost customer acquisition

Image source

BioTechUSA is a European dietary supplement manufacturer selling across more than 100 markets. Its loyalty program ran on blanket discounts that eroded margin without building lasting engagement.

According to the company's case study, BioTechUSA shifted to personalized incentives tied to individual customer journeys and buying behavior. It reported growth in average order value, customer lifetime value, buying frequency, and bottom-line profitability. That shift reflects a broader pattern in profitable promotion design. As with any vendor case study, these are the customer's self-reported results rather than an independently audited outcome.

Together, these guardrails reveal whether short-term activity creates durable commercial value. Teams should set explicit thresholds before the experiment begins.

Run the treatment period long enough to cover the relevant demand cycle, whether that's a weekly pattern, a 30-day reorder interval, or a seasonal swing. Collect the planned sample size before drawing conclusions. Maintain a post-offer observation window long enough to detect delayed conversion, returns, or pull-forward.

How do you run promotion tests without engineering tickets?

Promotion engine software with custom variants stacked on native discount logic can turn each new test into an engineering ticket. The fix is to move promotion logic outside the commerce platform.

Talon.One is infrastructure for personalized promotions and loyalty programs. For promotion testing, that unified approach helps teams connect offer exposure with buying behavior and downstream loyalty outcomes in one measurement framework.

A code-free Rule Builder lets marketers create conditions and effects without changing commerce-platform code. This kind of schema-independent architecture can evaluate cart contents and customer attributes against business rules in real time. Marketers can then configure and launch variants while engineering maintains the underlying integration.

A practical stack assigns variants separately from the systems that deliver messages and apply offers. One layer manages which customer sees which variant. A separate layer delivers the messaging, and the promotion engine handles the offer logic and deploys independently of the commerce stack.

Within one campaign-management environment, teams can run two rule variants, review confidence scores, then roll out winners without requesting engineering work for each version. A confidence score shows statistical reliability, not commercial value, so the guardrails from the previous section still decide whether a variant is worth scaling.

This separation increases campaign velocity without pretending implementation has no tradeoffs. Teams still need stable integrations, clear ownership, and agreement on measurement definitions before marketers can adjust variants while engineers focus on core product work.

What changes by vertical?

The constraints differ by industry. QSR and franchise economics, grocery basket effects, financial-services compliance, and targeting depth all change how teams structure a test.

QSR and franchise economics

QSR brands need daypart-level and franchisee-level analysis. A promotion that fills off-peak hours under one unit's economics may fail at another location.

Franchise economics also limit which offers teams can test. McDonald's franchisees have pushed to set local prices outside national promotions. Pricing, they argue, needs to reflect local market conditions rather than a single national calendar.

In a 2020 franchisee-association survey, roughly three-quarters of Subway franchisees said they would not participate in a proposed 2-for-$10 Footlong promotion. A single national result would have hidden that variation entirely.

Grocery and financial services

Grocery tests must account for basket composition and category halo effects. A discount can increase one category while reducing margin elsewhere in the basket. Teams should evaluate total basket contribution, with promoted-item volume as a secondary measure.

Financial services teams should route material changes through the institution's approved compliance-review process. Required workflows vary by institution, product, and jurisdiction. Reviewers typically need to approve variants that change disclosures or bonus structures before launch. They should also document what changed between variants to preserve the approval trail.

Talon.One can help teams configure offer variants, manage rules outside the commerce platform, and compare experiment results without rebuilding promotion logic for every test. Book your demo today.

FAQ about promotion testing

Monthly loyalty newsletter

Join thousands of marketers and developers getting the latest loyalty & promotion insights from Talon.One. Every month, you’ll receive:

Loyalty and promotion tips

Industry insights from leading brands

Case studies and best practices

Newsletter author

Reza Javanian

Loyalty & promotion expert at Talon.One

© 2026 Talon.One GmbH. All rights reserved.