Connect Claude, ChatGPT, or another AI assistant to Commslayer to find what to automate, write and test AI agent guidance on real past conversations, and track how it performs.
Jacob (@jforjacob) automates over 90% of his store's tickets this way:
The examples use a subscription store, since changing subscriptions is one of the easiest things to automate. The same steps work for any store. Swap in your own top topics, like order status, returns, or product questions.
Hand this to your assistant
Paste this into a new chat:
Read this guide: https://docs.commslayer.com/articles/1790768405-build-and-improve-ai-agent-guidance-with-an-ai-assistant
Then use my Commslayer connection to walk me through its 6 steps, one at a time. Tell me which step we're on and what you found. Ask before you save, test, turn on, or change anything, and wait for my yes. Warn me first if a test could run a custom action. Work on one topic and about 10 conversations at a time. If our policy isn't clear, ask me.
Follow the guide's "Rules and preferences".
Before you start
- Connect your assistant. Follow Connect Claude and ChatGPT via MCP. Any MCP assistant works, and we recommend Claude.
- Use a manager or admin account. Your assistant sees the inboxes you do. Only managers and admins get the reports in steps 1 and 6 and can change actions, though agents can create and turn on guidance.
- Approve every change. In your assistant's connector settings, set tools that change things to ask each time.
- Every change is versioned. Commslayer saves each edit to a guidance as a new version, whether you or your assistant made it. Version history shows who changed what and when, and Restore loads any earlier version back into the editor so you can save it again.
- Stay in one chat for steps 1 to 5, and ask for samples. Assistants can read up to 20,000 messages a day per account.
Step 1: Find what to automate
Jacob started with one question:
"can you look at commslayer the last 30 days and tell me ... what the most common reasons for her passing tickets onto real agents are..."
For a sharper list:
Use our contact reasons report for the last 30 days. Keep the top 10 reasons, skipping spam, promo emails, auto-replies, and notifications. Check sub-reasons of big topics. For the top 5, read 5 recent conversations my team resolved by hand, one topic at a time. Tell me which the AI agent could handle, whether each needs guidance or just a knowledge article, and which need an action that's off. Don't change anything.
Before you move on: pick one topic.
Step 2: Check your apps and actions
Actions are what the AI agent can do, like refunds or subscription cancellations. Guidance can't use an action that's off.
Actions come with default conditions that we set up and maintain, so you usually don't need to check or change them. Focus on turning on the actions your topics need and on the settings that depend on your policy, like discount offers or asking a human before running.
List the actions this topic needs, custom ones included, whether each is on, and if its app is connected. Check "Ask a human before running" and discount offers against our policy. Skip reviewing action conditions unless a test shows a problem. Don't change anything.
- Apps: You connect apps in Commslayer, not your assistant. Actions for Loop, Recharge, Skio, Stay AI, Kaching, and Appstle show up once connected.
- Changes: With your yes, your assistant can turn actions and discount offers on or off and change Ask a human before running. Set offer amounts and refund limits yourself in AI agent → Actions.
- Timing: Changes apply from the AI agent's next reply, live customers included.
Before you move on: the actions this topic needs are on and match your policy.
Step 3: Write the guidance
Write guidance for return requests based on how my team handled the conversations you read. If guidance for this exists, improve it as a draft. Run it on email and chat, save it in Testing, and show me what you wrote.
- Start from a template. For common situations, like complex order status questions (WISMO), our templates are a great starting point. Your assistant can't open templates, so add one in AI agent → Guidance → Templates and save it in Testing. Then ask your assistant to adapt it to your policy.
- Testing: New guidance starts in Testing, which only runs in the playground.
- Channels: Guidance with no channels doesn't run anywhere. Replays always run as email, so include email while you test.
Before you move on: the guidance is in Testing, on the channels you want.
Step 4: Test it on real past conversations
Your assistant replays past conversations in the playground, up to 10 at a time.
Find 10 recent conversations asking to return something and replay each in the playground using only the customer's first message. Rerun any that don't start. Compare each answer with my team's reply: one line per conversation, the full exchange only for ones that look wrong.
- Nothing sent: Customers see nothing, original conversations don't change, and tests don't count in reports or billing.
- Simulated: Refunds, cancellations, address changes, and subscription changes. Shipping checks always return not shipped, so cancellations can look better than they will live.
- Real: Order and customer lookups read your store data. Custom actions call your real endpoint.
Editing live guidance
Edits to live guidance save as a Draft, and customers get the live version until you publish. A draft test replays up to 10 conversations the guidance handled in the last 90 days (follow-ups included) on email, chat, or social comments. One test runs at a time, up to 50 conversations per account per day.
Our shipping guidance hands over international delivery questions. Save a fix as a draft, test it on up to 10 of them, and show new answers next to old ones. Don't publish yet.
Before you move on: the test answers look right.
Step 5: Turn it on
The tests look good. Turn on the return requests guidance.
- New guidance switches from Testing to Enabled and answers customers on its channels, in inboxes where the AI agent is on.
- A draft is published and replaces the live version.
Turn on one guidance at a time to see which moved your numbers.
Step 6: Check how it's doing
Because every edit is a version, your assistant can compare them: each guidance's conversations, resolution rate (resolved without a handover), and handover rate per version, up to 92 days back. Only trigger or instruction edits count as new versions here. Review any guidance above about 80% handovers.
It also compares resolution in the 28 days before and after your latest edit. With fewer than 20 conversations on either side, the change shows as "-" (too few to compare), and a move under 10 points is Within normal variation.
Run this daily the first week, then weekly:
Check the returns guidance for the last 90 days: conversations, resolution, and handover rate per version, and whether my latest edit has enough data to trust. Then open up to 25 of its conversations from the last 7 days, stopping at 10 handovers. Check knowledge gaps and suggest fixes. Don't change anything yet.
Test any change as a draft (step 4) before publishing.
Rules and preferences
Blocked: Commslayer won't save the text. Warning: it saves with a warning.
Where things go
- Facts go in knowledge, where every guidance can use them. Don't paste canned responses into guidance.
- Tone goes in AI agent → Settings → Tone of voice. (Warning)
- General rules apply to every reply, so keep handovers and catch-alls out. "Hand over any message under 8 words" sends every "thanks" to your team.
Actions
- Conditions are requirements only, like "Order must not be fulfilled". What to say or do goes in guidance. (Blocked)
- Don't repeat an action's conditions in guidance. The action checks them.
Triggers and instructions
- No keyword lists. They miss paraphrases and catch unrelated messages.
- No catch-all triggers. They drown out other guidance. (Warning)
- Write "if this, then that" steps: what to check, what to do in each case, when to hand over. (Warning)
- Say what to do, not what to say. Scripted wording gets copied into replies. (Warning)
- No identity or order number checks. Commslayer verifies customers before sensitive actions. (Blocked)
- Nothing about ticket status, like closing, resolving, or pending. (Blocked)
- Hand over only when a person must decide or act.
Preferences
- Prefer one guidance per situation. When two overlap, merging them is usually cleaner than pointing one at the other or adding "don't match" lists.
- Prefer one intent per trigger, like "Customer wants to cancel their subscription or stop future deliveries." Details that change the answer, like order status or country, usually work best in the instructions.