Test your AI agent on the real thing.
To test an AI agent in Arclyx, run the draft from Try it before you create it. It's a dry run: it reads with your real tools, shows every step as it happens, and lists anything it would post or send instead of sending it. Once the agent is live, Run it now is a real run.
- Gmail · searchis:unread newer_than:1d
- Gmail · read14 messages
- ThinkingSorting into Needs you, Can wait, FYI
- Email · send to meToday's inbox: 5 need you
Why test an AI agent on real tools?
Because the inbox, the CRM and the docs it will work in are what decide whether it's right. A test on made-up data passes and then fails on the first real Monday. Try it reads your real tools, so you see what the agent finds and what it would send before anything goes out.
Without a test run
- The first real run is the test
- You find the wrong channel after it posted
- You guess what a run will cost
With a test run
- You read what it would send before anything goes out
- You see every tool it called, in order
- You see the tokens a run takes
What does a test run show you?
The record every run keeps, while it happens, plus everything it held back, so you can judge the draft before it goes live.
Every step, live
Each round shows what the agent decided and which tool it used, while it runs.
Every tool call
What it searched, read or tried to write, with the result it got back.
Tokens and cost
Input, cached, output and reasoning tokens for the run, and its price on paid plans.
What it would send
Each post, email or connector write it held, with where it would go and what it would say.
What does a test run look like in Arclyx?
Try it sits under the draft. Add what to ask it, or leave it empty, and press Run: each round appears as it happens, anything it would post or send is listed under its answer instead, and the result links to the full run.
AgentsNew agentTry it
Inbox triage
DraftTry it
Optional. A dry run: it reads with your real tools and posts or sends nothing. It counts toward your usage.
- Round 1Search today's unread emailGmail · search threads
- Round 2Read the 14 threadsGmail · get thread
- Round 3Sort into Needs you, Can wait, FYIThinking
- Round 4Email you the listEmail · send to me
Dry run. Nothing below was sent.
- Email to you (dry run)Subject: Today's inbox: 5 need you Acme renewal: the contract is back with two changes. Launch review moved to Thursday.
What should you check in a test run?
Read what the agent did, not just what it said. A good answer can still come from the wrong tool or the wrong data, and an answer can open with “Sent” for something that was held. The held list is what counts. Check these five things before you create the agent.
The right tools, in order
Did it search before it read, and read before it wrote? Each round shows the tool it used.
The right inputs
What did it search for, which messages or rows did it read? A wrong filter shows up here first.
The right place
Would the result land where you said: your inbox, the one Slack channel, the right page? Each held item shows where it was going.
A sensible cost
How many tokens did the run take? A run that loops shows it in the count.
It stopped
Did it finish when the job was done, or keep going? Rounds have a limit you can set.
How do you test an AI agent on edge cases?
Give the test run a specific input. Try it has a box for what to ask it, so you can test the case you're worried about instead of waiting for it to happen. Run it a few times with different inputs before you create the agent.
- An empty inbox, no new rows. Does it say so, or invent something?
- An email from your biggest customer. Does it land at the top?
- Ask it to email a colleague. Its email only goes to you, so check the answer says so rather than promising otherwise.
- A long thread, a spreadsheet with blank rows, a page that won't load.
How do you fix an agent after a test?
Say what should change, in plain words: “only flag customers”, “put meetings first”. Arclyx rewrites the draft, and you test again. Repeat until the result is right, then create the agent.
Inbox triage
- Read today's unread Gmail, leaving out newsletters and receipts. Sort each one into Needs you today, Can wait or FYI, with one line on why. Send the list as one email.
- Gmail · readEmail · send to me
- Weekdays · 8:00
- Arclyx picks a model for each run
Is a test run a real run?
Not on a draft. Try it is a dry run, and Run it now on a live agent is real. Here's what that means.
- Try it is a dry run
- It reads with your real tools and counts toward your usage. Anything it would post, send or write through a connector is held and listed, not sent.
- Unfamiliar tools are held
- A connector tool runs only when it clearly just reads: a name like search or get, or a server that marks it read-only. A name that could also write is held either way.
- Helpers too, not the sandbox
- Work it hands to helpers is held the same way. Code it runs in the sandbox isn't held, but the sandbox has no internet.
- Run it now is real
- On a live agent, Run it now uses your real tools, so it posts and sends for real. You can stop it any time.
- Only the tools you picked
- The draft can use nothing you haven't granted.
- Drafts stay drafts
- A schedule or webhook can't start a draft. It only runs when you run it yourself.
How do you test an agent after you change it?
Change it with Improve, in plain words, then press Run it now on the agent's page. That's a real run, so it posts and sends. Every run goes into the agent's run history, so you can open the new run and an earlier one and see what changed: the steps, the tools and the tokens.
Is a test run the same as an AI agent evaluation?
No. A test run is one run that you read yourself. An evaluation scores an agent against a set of cases, again and again. For most agents a few test runs on real inputs, then the run history once it's live, is the check you need. We'd rather say that plainly than call a test run something it isn't.
Questions about testing an AI agent.
How do you test an AI agent?
Run the draft once from Try it, before you create it. It's a dry run: it reads with your real tools, shows every step as it happens and lists anything it would send instead. Change the agent in plain words and test again.
Does a test run use my real data?
Yes. Try it reads with your connected tools, only the ones you granted, so you see what the live agent would find. It counts toward your usage.
Will a test run send anything?
Not from Try it. A draft's test run holds every post, email and connector write and lists each one with what it would say. Run it now on a live agent is a real run, so it posts and sends. In Gmail and Outlook agents write drafts for you to review and send, and Try it holds even the draft.
How do you test an AI agent's accuracy?
Run it on inputs where you already know the right answer, then check its sources: every item should come from something the run actually read. The run record shows exactly what that was.
Do I have to test before creating an agent?
No. Testing is optional. Once an agent is live you can run it once with Run it now, which is a real run.
Where to go next.
Test your first agent.
One sentence in, a working agent out. Join the beta and we'll set you up.