Testing the AI agent

The playground lets you safely test your AI agent before deploying it with real customers. You can experiment with responses, test guidance, and ensure everything works correctly without affecting actual customer interactions.

What is the playground?

The playground is a simulation environment where you can chat with your AI agent using real customer data from your Shopify store. Test different scenarios, validate responses, and refine your AI agent's behavior before enabling it on live conversations.

You can find the playground by navigating to AI agent → Playground or simply clicking the Test button on other AI agent pages, which opens the playground on the side of the page.

Good to know: Order lookups only work when the Playground contact is a real customer with orders in Shopify. On the Playground page you select a contact before you write the first message. In the side panel that opens from the Test button, a test contact without orders is used unless you select one. To test order questions, select a customer who has orders, or click Test under a customer message in a real conversation.

Testing your AI agent

Chat directly

Type questions and messages to see how your AI agent responds. Test various customer scenarios and language patterns to ensure appropriate responses.

You can also click the Add image button to attach images to your question, or send an image on its own without text, to test how your AI agent handles questions with photos.

Replay existing conversations

Import previous customer interactions using the Test button found below any customer message in your conversation history. This lets you test how your AI agent would handle real-world scenarios with real order data from your actual support tickets.

Select channel

Choose which channel you want to simulate: Chat, DMs, Email, Social-media comments or Voice. The AI agent adapts its tone to the selected channel and only uses guidance that lists that channel under Apply to channels.

When you select Voice, you can also choose the Spoken language for the call. Testing voice does not require a phone number.

Simulate customers

Pick specific customers to simulate by searching their name or email address. This is required for testing order lookups and Shopify actions, since the AI agent verifies the customer's identity against Shopify before sharing order details. Select a customer with recent orders to use their real order data in your test.

Which guidance runs in the Playground

  • Guidance with the status Testing or Enabled runs. Disabled guidance never runs, in the Playground or live.
  • A guidance only runs when its Apply to channels list includes the channel you selected. The Playground starts on Email.
  • If an enabled guidance has unpublished changes, the Playground uses your draft. Customers get the published version until you publish.
  • Every test message uses your guidance and knowledge as they are at that moment. Replies in an older test conversation stay as they were, also when you disabled the guidance later.
  • Above each customer message, a chip shows which guidance matched, including the guidance ID after its name (for example, #4). When a message matches multiple guidances, a chip appears for each one.

What is simulated and what is real

Nothing in the Playground reaches a customer. Playground conversations don't count towards your usage, and you can test while the AI agent is switched off.

Simulated. The step appears as a line that starts with "[Simulated]" and nothing changes:

  • Handovers. You see a line that starts with "[Simulated] Escalation to agent" and the AI agent stays in the test conversation, so you can keep testing.
  • Assigning to a teammate or team, labels and status changes.
  • Order changes in Shopify: cancellations, refunds, reshipments, address changes and item changes.
  • Subscription changes in Loop, Recharge, Skio, Stay AI, Appstle and Kaching.
  • Custom actions set to change (the default for POST, PUT, PATCH and DELETE). The endpoint is never called; the AI uses the Playground response value you configured on the action instead. A blank Playground response field gives the AI {"simulated": true} as a placeholder.

Real:

  • Order and customer lookups read the selected contact's Shopify data.
  • Knowledge and product searches use your current knowledge.
  • Custom actions set to lookup (the default for GET). These call your real endpoint in Playground. Test lookup actions with care.
  • The Test button inside the custom action editor. This is separate from the Playground — it sends a real HTTP request to your endpoint regardless of the action's type. Use it to verify that your endpoint is reachable and responding correctly, not for testing AI conversation flow.
  • Private notes are written, but only into the Playground conversation.

Two things differ from live conversations. The check whether an order has shipped always answers "not shipped", so a cancellation can go through in a test and be refused live. And the customer's earlier conversations are not loaded.

Common problems

The AI agent says it handed over, but no handover shows

A real handover never happens in the Playground. Look for the line that starts with "[Simulated] Escalation to agent". If it is missing, the AI agent did not decide to hand over. Check that a guidance matched and that it lists the channel you are testing. Write the handover as its own step with a clear condition, for example "If no order is found, hand over."

A private note, label or assignment from my guidance is missing

Labels and assignments are simulated, so look for a line that starts with "[Simulated]". Private notes are written into the test conversation. If you see neither, the AI agent did not take that step. Write each one as a separate step in your guidance, in the order you want them, and name the person, team or label.

Managing AI knowledge

If you notice issues like repetitive canned responses or inappropriate knowledge sources, visit the Knowledge page. You can exclude or edit problematic knowledge sources to help your AI agent respond more naturally and effectively.

Regular testing in the playground helps identify knowledge gaps and ensures your AI agent provides consistent, accurate responses across all customer interactions.