VeveeBlog · 5 min read
Blog · 5 min read

Test mode: break your pricing in the sandbox, not on your customers

You changed the free tier from 10 to 25 generations and somehow locked out every Pro user for an hour. Nobody tested it, because testing it meant tracking fake events into production analytics. There is a mode for this.

Last updated: 2026-06-06

Why pricing logic goes untested

Nobody ships a payment flow without testing it, yet metering changes go out blind constantly. The reason is friction: testing a limit means generating events, and generating events into production pollutes the analytics you make decisions with, skews the very counters you are trying to verify, and - if you meter your own usage - costs real quota. So devs test pricing changes by squinting at the diff. The fix is the same one Stripe taught everyone a decade ago: a test mode that is a real, full-fidelity environment, not a mock.

The key is the switch

Every Vevee app has four keys: pk_live / sk_live and pk_test / sk_test. The prefix is the entire mode mechanism - same SDK, same endpoints, same plans and limit groups, but test keys read and write an isolated set of runtime tables. End-user state (events, counters, subscriptions, reservations) is fully partitioned; configuration (apps, plans, match rules) is shared. That split is exactly what you want for pricing tests: you exercise the real plan you are about to ship, against throwaway users.

// The only line that differs between prod and test
const vevee = createClient({
  apiKey: process.env.NODE_ENV === 'production'
    ? process.env.VEVEE_SECRET_KEY!       // sk_live_...
    : process.env.VEVEE_TEST_SECRET_KEY!, // sk_test_...
});

The pre-ship ritual, scripted

Because test mode is just the SDK with a different key, a pricing change gets an integration test like any other code. Subscribe a fake user to the changed plan, walk them to the limit, assert the wall appears exactly at the new number, assert the premium sub-cap still binds. Run it in CI if you like - test events are isolated, so the suite is rerunnable. This is the test you could never write against production.

// pricing-change.test.ts - runs against sk_test
const u = 'test_user_' + Date.now();
await vevee.upsertSubscription({ userId: u, planId: 'free' });

for (let i = 0; i < 25; i++) {
  expect(await vevee.can(u, 'image_generation')).toBe(true);
  await vevee.track(u, 'image_generation');
}
// the 26th must hit the new wall
expect(await vevee.can(u, 'image_generation')).toBe(false);

The dashboard follows you

Test mode is not headless. Flip the workspace toggle and the entire dashboard - usage charts, event feeds, subscription lists - reads the sandbox universe, with a yellow TEST MODE banner so you always know which reality you are looking at. This is how you demo the product to a stakeholder with realistic-looking data, debug a customer-reported limit issue by reproducing it safely, or check that your seeded staging environment looks right, all without a single fake row in production.

The fine print that keeps it honest

Two boundaries worth knowing. Test data older than 30 days is hard-deleted by a daily sweep - the sandbox is a workbench, not an archive, so never point anything you care about at a test key. And test-mode operations count toward your workspace’s plan cap just like live ones; the sandbox is isolated for data, not unmetered, so routing production traffic through a test key buys you nothing except missing analytics. Use live keys for users, test keys for everything else - staging, CI, demos, and every pricing change before it meets a customer.

More from the blog

engineering · 9 min

The race condition in "if (usage < limit)" that is costing your AI app money

A user at nine of ten images opens six tabs and clicks generate in all of them. Every tab passes your limit check, every tab gets an image, and you pay for all six. The bug is one read-then-write - and your unit tests will never catch it.

engineering · 9 min

How to build a credits system for your AI app (ledger design, rollover, refunds)

A user emails: "I had 40 credits this morning and now I have 12, and I only made one image." If your balance is a column you mutated, you cannot answer that email. Here is the ledger design that can.

thinking · 7 min

Stripe metered billing for AI apps: what it solves and what it does not

You are sketching the pricing page for your AI app and every plan ends in "per generation" - because your costs arrive per generation. Stripe meters the charging half beautifully. The question is what happens at request time.

thinking · 8 min

Helicone is in maintenance mode. An honest map of where to go

Somewhere in your codebase a base URL points at Helicone, and it has been quietly doing four jobs at once. The team got acquired - good for them - but now you have to replace each job separately, and the one everyone forgets is the one your users will exploit.

engineering · 8 min

How to stop free-tier abuse without killing signups

The usage graph spikes at 3:41am: four thousand free generations in an hour, all from disposable emails. Your first instinct is a credit-card wall. That instinct will cost you more than the abuse does.

thinking · 7 min

Tokens, credits, or requests: choosing the unit you meter (and price)

The pricing doc has three columns: $9, $19, $49. The prices took twenty minutes. The row above them - what a user actually gets for the money - has been blank for a week. That blank row is the real decision.

engineering · 3 min

I rewrote my cancel flow with one LLM call. It argues better than I do.

The cancel flow is where SaaS revenue goes to die politely. "Are you sure? You'll lose access to Premium features" has never changed a single mind - but reminding a user of their own 312 generations this month is an argument.

engineering · 3 min

The trial-ending email everyone sends is the same email. Here's the one that converts

"Your trial ends in 3 days! Upgrade now to keep access." You've received a hundred of these. You've deleted a hundred of these. The email fails because it's about your product, not about the user's trial.

engineering · 3 min

Stop making users do math on your pricing page: recommend their plan from their own usage

Every pricing page asks the user to solve an estimation problem: "How many credits will I need per month?" Nobody knows. But for any user who has actually used your product, their usage history IS the answer - here is how to put it on the pricing page.

engineering · 3 min

Win-back emails fail because they're written for "users." Write them for the one user instead.

Every dormant-user campaign in history: "We miss you! Here's what's new." Open rate: pity clicks. The email fails because it's about your changelog - and the user left because of something in their experience.

engineering · 3 min

Spotify Wrapped is a growth loop, not a year-end gimmick. Ship one for your AI app in an afternoon.

Spotify Wrapped works because people love seeing their own behavior reflected back as a story. Every AI app with usage data can run that loop monthly - and almost none do, because turning usage rows into narrative used to be a content problem. It is now a schema problem.

engineering · 3 min

"AI-personalized copy converts better" - prove it or delete it. Here is the 40-line A/B harness

Half of the "we added AI personalization and conversions went up 40%" posts have no control group. The other half measured clicks, not revenue. If LLM-generated copy is going in front of your paywall, you owe yourself a real experiment - and the harness is tiny.

thinking · 3 min

AI personalization without the creepy part: opt-out as a first-class return value

Users have learned that "personalized for you" means "we mined everything you ever did" - and the backlash is rational. When I added LLM-generated personalized copy to my app, the part I sweated was not the generation. It was making declining it a real, respected choice.

engineering · 3 min

I stopped walking into demo calls blind: every lead now comes with a usage brief written 10 seconds before the call

Founder-led sales has one structural weakness: you have no time to prep. If the lead has touched your product, their usage history is the best discovery call you'll never have to run - here's how I turn it into a one-page brief, automatically, before every call.

thinking · 4 min

The "founder email" converts like crazy and scales like garbage. Here is the middle path.

A personal email from the founder converts trials at a rate no automated sequence touches - and at 50 signups a week it stops scaling. The middle path: drafts generated from each user's real usage, that you read, edit, and send yourself.

engineering · 3 min

Churn doesn't announce itself. My Monday Slack digest does it instead.

Every founder finds out about churn the same way: the cancellation email. By then the user has been gone for weeks - the decision happened earlier, quietly, in their usage. So I made the metering tables write me a memo.

engineering · 3 min

Half of every support ticket is asking what the user already did. Attach the answer instead.

The ticket says "it's not working" - and the next twenty minutes go to figuring out who this user is, what plan they're on, and whether "it" is a bug, a quota, or a misunderstanding. The actual fix usually takes two. All of that context lives in your usage data; here is how to attach it to every ticket automatically.

thinking · 4 min

Your users are telling you your roadmap, in writing, every day. It's in your prompt logs

Founders pay for interviews and beg for survey responses to learn what people are actually trying to do. Meanwhile, your users type their intent into your product, in their own words, hundreds of times a day - and nobody performs for a prompt box.

engineering · 5 min

How to reset usage limits when a subscription renews

A weekly plan, used twice. First week: ten generations. Second week: zero, because the counter never reset. This is the cron-job mistake - and the fix is one field on one call.

engineering · 6 min

How to manage subscription renewals: aligning Vevee with Stripe

A user signs up on Jan 15. Stripe charges them on the 15th of every month. Your metering layer resets on the 1st. Two clocks. One angry support ticket per cycle.

engineering · 5 min

Stop hardcoding your pricing page - render it from your metering layer

Every B2B SaaS I have shipped repeats the same mistake: plans live in two places - the dashboard that enforces them, and a const PLANS = [...] on the marketing site. They drift within a quarter.

thinking · 4 min

Meter AI by user, not by account - your margin depends on it

A few users will cost you 100x what your median user costs. If you only meter at the account level, you will not see them coming until your gross margin is gone.

engineering · 5 min

reserve / commit / release: the only correct way to enforce AI quotas

Every team I have seen build per-user AI metering has shipped a version of canUse → call OpenAI → track. It looks correct in single-threaded tests. It is broken in production.

thinking · 4 min

Why Stripe Billing is not enough for AI products

Stripe is excellent at one thing: turning usage into invoices. AI products need three other things, and Stripe does not do any of them.

engineering · 6 min

Dynamic onboarding: a different first step for every user

A teacher and a student sign up the same minute. The teacher wants to build a quiz; the student wants to summarize a lecture. Your onboarding shows them both the same five-step checklist. One of them bounces.

thinking · 6 min

Paywall copy that rewrites itself for every user

Your paywall says "Upgrade to Pro for unlimited generations." A teacher reads it and shrugs. A student on a budget reads it and closes the tab. The same words, two lost conversions - because the words were written for nobody in particular.

engineering · 5 min

Add usage limits to your AI app in 10 minutes (no backend required)

You shipped an AI feature on Friday. By Monday one user had burned $212 of OpenAI credit on your free tier. The fix is not a TODO comment that says "add rate limiting" - it is two method calls.

engineering · 5 min

Meter LLM tokens, not requests - your flat per-request limit is lying

Two users, both at 100 requests. One sent tweets, the other sent novels. Your cost for them differed by 400x. Your limit treated them identically - and your margins noticed.

engineering · 7 min

The upgrade nudge that writes itself: convert free users before they hit the wall

By the time a user hits your paywall, they are blocked, annoyed, and halfway to a competitor’s signup page. The best moment to make the pitch was three days earlier - when they were winning. Here is how to catch it, automatically, for every user at once.

engineering · 5 min

One event, two limits: gate your premium model without forking your code

The premium model launch was going great until you looked at the bill: free users had figured out the good model and were living on it. You need a sub-limit. You do not need a second code path.

engineering · 5 min

The support ticket that solves itself: log the prompt behind every AI event

A user says your AI feature "broke" on Tuesday. You have a charge for the call, a timestamp, and no idea what they asked or what the model said. The evidence existed for exactly one request - the one you didn’t log.

engineering · 5 min

Your funnel has one broken step. Find it without writing a single SQL query.

A hundred people saw your paywall this week. Three upgraded. Is that a copy problem, a price problem, or did ninety of them never generate anything worth paying for? You cannot fix what you cannot locate.

engineering · 5 min

One user, three ghosts: fix your funnel with identify()

Your funnel says signup conversion is 4%. It is actually 11%. The missing users didn’t bounce - they came back on another tab and got counted as someone new. Every number downstream of that split is wrong.

thinking · 5 min

The usage bar that sells the upgrade (build it with a public key in an afternoon)

An invisible limit feels like a trap. A visible one feels like a fuel gauge. Same quota, same plan, same user - and a measurably different reaction when the wall finally arrives.

engineering · 6 min

How to cancel a subscription without burning the bridge (or your data)

The user cancels. Do they drop to a free tier, or lose access when the paid month runs out? Those are different products, different SDK calls, and different mistakes when you get them wrong.