← Research library

Fall conference season: questions to ask agentic marketing vendors

AI-assisted research. This note was prepared with the assistance of AI tools. It may contain errors and is not professional advice. See our Legal Disclosures.

Agents have become the headline for many marketing technology products. We expect fall conference season to make that obvious, with booth after booth promising agents that plan, write, and launch campaigns.

Some of these products will be useful. Others will be demos that work well on stage and poorly against real data, real permissions, and real volume. The way to tell them apart is to ask the same specific questions of each vendor and compare the answers.

A conference demo is built to succeed. It runs on a prepared account, with clean data, a narrow task, and no competing priorities. That is reasonable for a booth, but it tells you little about how the agent behaves when the CRM has duplicates, the brief is ambiguous, or a tool call fails halfway through.

The questions below are designed to move the conversation from the demo to your environment. We group them by theme so you can split them across a team, and we explain why each group matters. Expect some questions to take a follow-up call to answer properly. That is fine, and often a good sign.

What the agent actually does

Start with scope. Many agent pitches describe outcomes ("it runs your lifecycle program") rather than tasks, and the gap between the two is where disappointment lives. These questions pin the agent to specific work.

  • What specific tasks does the agent perform, from start to finish?
  • Which actions does it take on its own, and which need human approval?
  • What does it do when a request is ambiguous or it is uncertain?
  • Can we see it work on our own data rather than a prepared demo account?

Listen for whether the answer to the ambiguity question is a behavior or a promise. A good answer describes what the agent does: it asks, it pauses, or it escalates to a named queue. A weak answer says the model is smart enough not to need that.

Control and safety

Agents act through tools, and tools touch budgets, audiences, and customer inboxes. The questions here test whether limits are enforced by the system or only requested of the model. A limit written into a prompt is a suggestion. A limit enforced in code outside the model is a control.

  • Can we restrict which actions the agent can take, at a granular level?
  • Are limits on spend, audience size, and send volume enforced outside the model?
  • How do we pause the agent immediately, and who can do it?
  • Does it act under its own identity in our systems, or under a user's credentials?
  • How does it handle untrusted content, such as inbound emails or web pages that contain instructions?

Ask also what happens when a tool call fails partway through an action. Retries are a common source of duplicate sends and double charges. We cover the patterns that prevent them in Tool Errors Are Marketing Errors.

Visibility and audit

When something goes wrong at 11pm during a promotion, someone on your team has to work out what the agent did and why. If the logs are hard to read or locked inside the vendor's console, that work gets slow at exactly the wrong moment.

  • Can we see each action the agent took, with its inputs and reasoning?
  • Can we trace any published output back to the run that produced it?
  • Can marketers, not only engineers, read the logs?
  • Can we export logs to our own systems for audit and analysis?

We think the export question matters most over time. Your legal, brand, and finance teams will want their own records. Our post on observability for marketing agents describes what a useful run history contains.

Quality and evaluation

Quality claims are easy to make and hard to check. The goal of these questions is to learn whether the vendor measures quality in a way you can see and repeat, and whether you can test changes before they reach live work.

  • How do you measure the agent's quality, and can we see the results?
  • Can we test changes against our own examples before they affect live work?
  • What happens to our agents when you update the underlying models?
  • How will we know if quality drops after an update?

The model update question is the one vendors find hardest. Underlying models change, and behavior can shift even when nothing in your setup changed. A credible answer involves a test set you control and a way to compare results before and after. We describe how to build one in Golden Datasets.

Data and consent

Agents need access to customer data to do useful work, which makes data handling a central part of any evaluation. Bring your privacy and security partners into this part of the conversation, and confirm specific legal requirements with counsel rather than relying on a vendor's summary.

  • What data does the agent access, and where is it processed and stored?
  • Is our data used to train models, and can we opt out?
  • How are consent preferences and opt-outs enforced when the agent acts?
  • How long are prompts, outputs, and logs retained?

Commercial terms and evidence

We expect pricing models for agents to keep changing, and the model a vendor chooses shapes your costs at high volume. A per-task price that looks small in a pilot can grow quickly in a launch month or at year-end, especially if retries are billed. Consider a hypothetical team planning 10 times its normal volume in December: ask the vendor to price that month, not an average one.

  • How is the agent priced: per seat, per task, per outcome, or by usage?
  • What does it cost at our expected volume, including retries and our busiest periods?
  • What happens to our configurations, test sets, and logs if we leave?
  • Can you share production examples, with results and the method used to measure them?

The last question is about method as much as results. A result without a described method is hard to weigh. A vendor that explains how it measured, and what it did not measure, is giving you something you can check.

Reading the answers

Vendors with strong products usually welcome these questions, because the answers show their work. Evasive or generic answers are informative too. We suggest scoring each theme as clear, partial, or unclear, and comparing vendors on that basis rather than on the demo.

The question we are still studying is how buyers should weigh control against capability. An agent with tight limits may do less on day one and still be the better choice a year later. We do not yet have a reliable way to price that trade-off, and we expect it to shape how agent products are compared next year.

How to apply this: bring this list to your next vendor conversation and split the themes across your team. Ask to see the agent work on your own data, and ask how you would pause it and trace its actions before you discuss price.

Related research

Put this research to work

Marketix turns these patterns into software: agents, automated workflows, and a hub that keeps people in control. Sign up for early access.