Meta's reported conversions answer an attribution question: which conversions its system credits to ads under its settings. A geo-holdout test asks a different question: how did the business outcome change in comparable locations where the campaign was withheld, relative to locations where it ran? The second question can help estimate incremental effect, but only when the design and data support a credible comparison.
A geographic test is not a four-week recipe that guarantees a result or protects all revenue. The test may be too noisy, the available regions may be too different, or local promotions and stock changes may overwhelm the signal. Google researchers note that geo experiments can be difficult when the number of regions is small, outcomes are highly variable, or budget limits create interference between units. Design and feasibility come before switching campaigns off. Google Research: robust causal inference for paired geo experiments
Define the business question before choosing regions
Write the decision the test should inform. For example: “Should we keep this Meta prospecting campaign at the current level in these markets?” is more useful than “Does Meta work?” Specify the outcome that matters—such as delivered orders, contribution margin, qualified leads, or store visits—and the period in which it can be observed.
Keep four concepts separate:
| Concept | What it tells you | What it cannot establish by itself |
|---|---|---|
| Platform-attributed conversions | Outcomes Meta credits to the campaign under its attribution settings | What would have happened without the ads |
| Website or CRM conversions | Outcomes recorded by the business system | Whether the campaign caused the outcomes |
| A/B or conversion-lift experiment | Effect estimated under its assignment and eligibility design | That the result generalizes to every market, audience, or future period |
| Geo-holdout estimate | Difference associated with running or withholding ads in selected locations, under the design assumptions | A universal “true ROAS” that applies outside the tested conditions |
If the actual question is campaign or creative comparison within the platform, check whether an eligible randomized experiment is available. A geo test is more relevant when the decision concerns market-level spend and location-level outcomes can be measured reliably.
Check whether a geographic design is feasible
List the locations the business can target and measure separately. Confirm that orders, revenue, margins, leads, or other chosen outcomes can be assigned to the same geographic units in the pre-period and test period. The location rule must be consistent: billing address, delivery region, store catchment, or another recorded field. Do not change the rule after seeing the results.
Before selecting a test and holdout group, check:
- Whether enough non-overlapping locations exist to form comparable groups.
- Whether baseline outcomes move together over a sufficiently relevant historical period.
- Whether the product, audience, price, delivery, and sales process are available in all selected areas.
- Whether other campaigns or organic promotions will reach the holdout locations.
- Whether the business can keep promotions, stock availability, pricing, and fulfillment reasonably comparable.
- Whether delayed outcomes, cancellations, refunds, or lead qualification can be tracked by location.
- Whether the likely effect is large enough to distinguish from normal variation with the available data and budget.
If regions are materially different or location-level data is sparse, a geo holdout may not be the right design. Google Research describes matched-market approaches for cases where random assignment alone does not produce balanced groups, but those methods still rely on assumptions that need to be examined. Google Research: matched markets for geo experiments
Assign treatment and holdout without creating leakage
Choose the assignment method before the experiment begins. Where feasible, randomly assign comparable geographic units to treatment and holdout, possibly within blocks based on pre-period size or other preselected characteristics. If randomization is not feasible, document how matched regions were chosen and which assumptions the analysis requires. Do not select control regions after viewing the post-test outcome.
Create a location map with one row per region: assignment, pre-period outcome, campaign status, expected spend, other marketing activity, major operational differences, and measurement source. Keep the holdout free of the tested campaign across every relevant ad set, account, and agency workflow. Record any accidental exposure or concurrent campaign that reaches the control group. Leakage usually makes the groups less distinct and can weaken interpretation.
Avoid changing the treatment halfway through the test. If you need to change spend or creative for a safety or operational reason, log the date, affected locations, reason, and campaign change. The result then describes the actual intervention and should not be presented as a clean test of the original plan.
Set the measurement and decision rules in advance
Choose one primary business outcome and a short list of guardrails. Define whether the result counts orders placed or delivered, whether revenue is gross or net of refunds, how leads are qualified, and which variable costs enter contribution. Use the same definitions for both groups and both periods.
Estimate feasibility and statistical power before launch. A specialist should examine historical location-level variation, the number of available units, expected outcome delay, the minimum effect that would change the business decision, and the intervention cost. Do not import a universal minimum number of regions or a fixed test duration; the appropriate design depends on the data and variance. Research on geo experiments specifically warns that small numbers of heterogeneous regions can make incremental return estimates unreliable. Google Research: paired geo experiment design and inference
Write the rule before results are available:
- Primary outcome: [business measure, geographic assignment, source, and maturity window].
- Guardrail: [maximum acceptable cost, minimum contribution, or service-quality threshold].
- Analysis: [randomized paired comparison or preselected matched-market model and assumptions].
- Decision threshold: [the smallest effect that would change budget or market policy].
- Inconclusive condition: [insufficient data, tracking loss, control exposure, major imbalance, or interval spanning useful and harmful outcomes].
If results are too uncertain to distinguish a useful increase from no effect or harm, report “inconclusive.” A noisy point estimate is not proof of success or failure.
Analyze the difference, not just the before-and-after change
Revenue increasing in treatment regions after ads launch does not show what would have happened there without ads. Compare treatment and holdout in the same period, using the pre-period to evaluate balance and support the analysis. A difference-in-differences view may be appropriate for a chosen design, but it depends on assumptions about how the groups would have moved without treatment. A sophisticated model does not repair a poor assignment or missing outcome data.
Report the estimate with its uncertainty, the observed pre-period fit, the assignment method, the actual spend difference, the outcome definition, missing data, and any deviations. Show both group-level results and the calculation. Keep platform-attributed ROAS in a separate column. The result may suggest that spend contributed additional outcomes under the tested conditions; it does not establish that every attributed conversion was incremental or that another market will behave the same way.
For more on separating reported signals from business outcomes, see paid media conversion signals and their limits and how attribution reports can mislead decisions.
Short-video script: What a Meta geo holdout can and cannot tell you
Hook: “Meta says the ad received credit for a sale. Did the sale happen because of the ad?”
Scene 1: Show two maps. “A geo holdout compares selected regions where the campaign runs with comparable regions where it is withheld.”
Scene 2: Show the same order definition on both sides. “Choose the business outcome and location rule before the test. Check that prices, stock, promotions, and other campaigns do not make one side incomparable.”
Scene 3: Show an estimate with an uncertainty band. “A rise after launch is not enough. Compare the groups, disclose assumptions, and call it inconclusive if the data cannot support a decision.”
Close: “Use a holdout only when you can measure locations and keep the comparison credible.”
Educational carousel: Seven checks before a regional ad holdout
- Cover: Would a geo holdout answer your paid media question?
- Decision: Name the budget or market decision the result could change.
- Outcome: Define a business measure, not just a platform-attributed conversion.
- Geography: Confirm the location rule and location-level data for every region.
- Comparison: Check pre-period similarity and write the assignment method in advance.
- Contamination: Audit other campaigns, promotions, stock, pricing, and service differences.
- Power and risk: Estimate whether available regions and data can detect a decision-worthy effect.
- Decision: Report uncertainty and use “inconclusive” when the evidence is insufficient.
Field experiment thread: Design a Meta geo-holdout test
This is a proposed design checklist, not a completed client experiment.
1/ State the business decision and primary outcome. Define the location field, data source, and outcome-maturity window before selecting regions.
2/ Use historical outcomes to identify whether enough comparable, non-overlapping regions exist. Check for seasonality, local events, inventory, price, promotion, and fulfillment differences.
3/ Preselect the assignment approach: randomized geographic units where feasible, or a documented matched-market method with explicit assumptions. Freeze the assignment before launch.
4/ Keep the holdout clear of the tested campaign across relevant accounts and ad sets. Log actual delivery, budget changes, overlapping campaigns, and any operational exception.
5/ Estimate the expected uncertainty and smallest decision-relevant effect. If the available regions or outcome volume are inadequate, do not run a test that cannot answer the question.
6/ Compare treatment and holdout with the preselected analysis. Report the estimate, uncertainty, outcome definition, deviations, and limits; separate it from Meta-attributed ROAS.
Tool-neutral prompt pack: Red-team a geo-test design
Review this proposed paid media geo-holdout using only the supplied details: [business decision], [primary outcome], [location rule], [regions and assignments], [pre-period data], [campaigns reaching each region], [pricing/stock/promotion differences], [expected outcome delay], [analysis method], and [decision threshold]. Identify the causal question the design can answer, unsupported assumptions, possible contamination, data gaps, and operational risks. Check whether the test and holdout are comparable and whether the outcome is measured consistently. Recommend one change only if the provided evidence supports it. Do not invent a sample-size threshold, duration, lift, ROAS, or guaranteed result. Return an “inconclusive” path if the design cannot distinguish a decision-worthy effect.
Stop-doing swipe file: Replace weak incrementality claims
- Stop: “Revenue rose in the ad markets, so Meta caused the increase.” Use: “We compare assigned treatment and holdout locations over the same period and report the design assumptions.”
- Stop: “Meta reported 100 purchases; all 100 were incremental.” Use: “The platform attributed 100 purchases under its settings; the holdout estimates a separate incremental effect.”
- Stop: “Run the test for exactly four weeks.” Use: “Set the window from historical variation, outcome delay, and the effect needed to change the decision.”
- Stop: “Pick the control cities that look closest after the test.” Use: “Preselect regions and the matching method before observing outcomes.”
- Stop: “The point estimate is positive, so scale.” Use: “Review uncertainty, data quality, guardrails, and the prewritten decision threshold.”
Hook–Value–Template: Decide if a geo holdout is usable
Hook: “A regional ad test is useful only if the control regions give you a credible picture of what might have happened without the campaign.”
Value: Region matching, assignment, leakage, operational differences, outcome definition, and uncertainty determine whether a lift estimate can support a budget decision.
Template: “Decision: [what will change]. Outcome: [business measure, location rule, source, maturity window]. Eligible regions: [list]. Assignment method: [randomized/matched and why]. Pre-period fit: [evidence]. Holdout protection: [campaign exclusions]. Confounders: [events, stock, price, promotions, other media]. Analysis: [preselected method]. Smallest useful effect: [decision threshold]. Stop/inconclusive rule: [conditions]. Final report: [estimate, uncertainty, deviations, limits].”
Research sources and limits of geographic lift tests
- Google Research: Measuring ad effectiveness using geo experiments — describes assignment of non-overlapping geographies to treatment and control and the need for systematic experiment design.
- Google Research: A time-based regression matched-markets approach — explains matched-market design and assumptions when randomization is constrained.
- Google Research: Robust causal inference for paired geo experiments — discusses heterogeneous geographies, small numbers of units, and budget constraints that complicate inference.
- Marketing Science: Estimating the value of offsite tracking data to advertisers — studies randomized holdouts using Meta's experimental infrastructure; its user-level design is not a geo-test recipe.
No experiment guarantees a positive result or a revenue-safe holdout. Treat conclusions as specific to the tested markets, period, campaign, data, and design.