Automation End-to-End Testing Guide

Prerequisites

Info

Fresh authentication token (expires in ~1 hour). Try hardcoding variable value in message - If that works, interpolation is broken - If that doesn't work, step execution is broken

Warning

Fresh authentication token (expires in ~1 hour). Try hardcoding variable value in message - If that works, interpolation is broken - If that doesn't work, step execution is broken

Success

Fresh authentication token (expires in ~1 hour). Try hardcoding variable value in message - If that works, interpolation is broken - If that doesn't work, step execution is broken

Error

Fresh authentication token (expires in ~1 hour). Try hardcoding variable value in message - If that works, interpolation is broken - If that doesn't work, step execution is broken

Required:

Fresh authentication token (expires in ~1 hour)
Workspace ID: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">UIDRinD8Ef8UK1pxvdlv</code>
Automation ID: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">xjDJ2Haamnapqf9V7Y4h</code>
Test widget key (inbox/channel ID the automation is attached to)
Localhost backend running on port 3030
Browser dev console open to monitor real-time messages

Phase 1: Pre-Test Verification

1.1 Verify Automation Structure

          bash
          
        

curl -X GET http://localhost:3030/api/v1/automations/xjDJ2Haamnapqf9V7Y4h \
  -H "Authorization: Bearer <FRESH_TOKEN>" | jq '.data'

Verify these fields are present:

✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">trigger</code> — the entry point (e.g., conversationcreated, messagereceived, etc.)
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">triggerNextStepId</code> — points to first step after trigger
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">steps</code> — array of all step objects
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">channels</code> — array of channels trigger applies to
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">frequency</code> — everytime / oncepercontact / onceper_conversation
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">schedule</code> — { type: 'anytime' | 'officehours' | 'custom', ... }
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">audienceFilters</code> — filtering rules { logic: 'AND'|'OR', groups: [...] }
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">goal</code> — success metric { enabled: bool, type, window }
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">status</code> — must be 'live' to trigger

Phase 2: Trigger Testing

Scenario 2.1: Initial Trigger Activation

Setup:

Open test page with widget (or create test contact widget at <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">localhost:3000</code>)
Widget loads with test key
Watch browser console for Pusher messages

Action:

Send first message from widget as anonymous visitor
Message: "Hello" (simple starter)

Expected Flow:

✅ Backend creates new conversation
✅ Automation trigger matches (conversation_created fires)
✅ Backend creates enrollment record
✅ First step after trigger begins executing
✅ Pusher event fires: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflowQuestion</code> or <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflowChoices</code> or <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">message:new</code>
✅ Widget receives event and updates UI

What to Watch:

Browser console: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">Pusher <private-conversation-ID> message:new</code>
Browser console: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">Pusher <private-conversation-ID> workflowQuestion</code> (if ask_question)
Browser console: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">Pusher <private-conversation-ID> workflowChoices</code> (if choices)
Backend logs: no errors during step execution

Common Bugs:

❌ No Pusher event fired → step executor failed
❌ Event fired but widget doesn't update → subscription issue
❌ Trigger doesn't fire at all → status not 'live' or frequency/schedule blocked

Scenario 2.2: Frequency Control

Test: Ensure frequency setting prevents duplicate automations

Setup:

Automation frequency = "oncepercontact"

Action:

Send message #1: "Hi"

- ✅ Automation should trigger

Send message #2: "Hi again"

- ❌ Automation should NOT trigger again (frequency blocks it)

Expected:

First message: automation runs
Subsequent messages from same contact: automation skipped

Verify in Firestore:

Check <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">enrollments</code> collection
Contact should have only 1 enrollment for this automation
Status: 'completed' or 'in_progress'

Scenario 2.3: Schedule Control

Setup:

Automation schedule = "office_hours" (9am-5pm)
Test outside office hours

Action:

Send message at 6:00 PM
❌ Automation should NOT trigger (schedule blocks it)

Verify:

Check backend logs for schedule evaluation
Message arrives but automation enrollment not created

Phase 3: Step Execution Testing

Scenario 3.1: ask_question Step

Setup:

First step after trigger = <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">ask_question</code>
Question text: "What is your name?"
Variable name: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">user_name</code>

Action:

Send message: "Hello"
Automation triggers
Bot asks: "What is your name?"
WAIT — automation pauses

Expected:

✅ Message appears in widget conversation
✅ Conversation conversation has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflow.awaitingQuestion = true</code>
✅ Conversation has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflow.questionVariableName = 'user_name'</code>
✅ Widget shows input mode or question UI

What Breaks:

❌ Question doesn't appear → step didn't execute
❌ Question appears but widget crashes → UI doesn't handle event
❌ Response gets lost → resumeQuestion handler failed

Scenario 3.2: Resuming From ask_question

Setup: (continuing from 3.1)

Action:

User types response: "John"
Submit via widget

Expected Flow:

✅ Widget sends POST to <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">/api/v1/automations/question/answer</code>

<code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em"></code>`json { "conversationId": "...", "answer": "John", "variableName": "user_name", "enrollmentId": "..." } <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em"></code>`

✅ Backend resumes automation from pause point
✅ Variable <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">user_name</code> = "John" available to next steps
✅ "John" response is saved to conversation messages
✅ Next step executes

What to Monitor:

✅ User's response appears in conversation as message
✅ Check Firestore <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">conversations.workflow.variables.user_name</code> = "John"
✅ Check <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">enrollments.variables.user_name</code> = "John"

Common Issues:

❌ Response doesn't appear in conversation → message creation failed
❌ Variable not stored → resumeQuestion didn't save it properly
❌ Automation doesn't continue → nextStepId not set

Scenario 3.3: choices Step

Setup:

After ask_question, choices step appears
Prompt: "Which service do you need?"
Choices: ["Billing", "Technical Support", "Sales"]
Branches: 3 branches (one per choice)

Action:

Bot sends: "Which service do you need?"
Widget shows choice buttons
User clicks "Billing"

Expected:

✅ Prompt appears as bot message
✅ Conversation has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflow.awaitingChoice = true</code>
✅ Conversation has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflow.currentChoices = ["Billing", "Technical Support", "Sales"]</code>
✅ Conversation has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflow.choiceConnections</code> = [nextStepId1, nextStepId2, nextStepId3]

What to Monitor:

Browser console: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">Pusher workflowChoices</code> event
Widget displays choice buttons
Each button is clickable

Scenario 3.4: Resuming From choices

Action:

User clicks "Billing"

Expected Flow:

✅ Widget POST to <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">/api/v1/automations/choice/select</code>

          javascript
          
        

   `json   {     "conversationId": "...",     "selectedChoiceIndex": 0,     "variableName": "last_choice",     "enrollmentId": "..."   }   `

✅ Automation branches to choice[0].nextStepId
✅ "Billing" selection appears in conversation
✅ Next step in Billing branch executes

Verify in Firestore:

Choice selection saved to messages (either as user message or bot message)
Variable <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">last_choice</code> = "Billing"
Enrollment continues on correct branch

Phase 4: Action Steps Testing

Scenario 4.1: send_message Action

Setup:

Step after choices: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">send_message</code>
Content: "You selected {{last_choice}}. We'll connect you soon!"

Action:

Trigger automation through full flow
Select a choice
Wait for message

Expected:

✅ Message appears in conversation
✅ Content interpolates variable: "You selected Billing. We'll connect you soon!"
✅ Message has <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">senderType = 'bot'</code>

Common Issues:

❌ Variable not interpolated → context.variables not passed
❌ Message doesn't appear → step executor failed

Scenario 4.2: assign Step

Setup:

After choices: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">assign</code> step
Assign to: "Billing" inbox (from choices)
Message: "Routing to {{last_choice}} team..."

Action:

Trigger automation to assignment step
Select "Billing"

Expected:

✅ Conversation routed to Billing inbox
✅ <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">conversation.inboxId</code> = billingInboxId
✅ Notification message appears: "Routing to Billing team..."
✅ Agent sees conversation in Billing inbox

Verify:

Check Firestore conversation.inboxId
Check agent inbox (might need to refresh)

Scenario 4.3: routetoinbox Step

Setup:

Step: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">routetoinbox</code>
Config: map choices to specific inboxes

- "Billing" → Billing Inbox - "Technical Support" → Support Inbox - "Sales" → Sales Inbox

Action:

Full automation flow, select choice
Step routes to mapped inbox

Expected:

✅ Conversation moves to selected inbox
✅ Assignment message appears
✅ Agent can see conversation

Scenario 4.4: Condition Branching

Setup:

After choices, condition step
Condition: "If last_choice == 'Technical Support', then..."
True branch: technical support flow
False branch: sales flow

Action:

Choose "Technical Support"
Follow automation

Expected:

✅ Condition evaluates to true
✅ Automation follows true branch
✅ Technical support actions execute

Test All Branches:

✅ last_choice = "Billing" → false branch
✅ last_choice = "Sales" → false branch
✅ last_choice = "Technical Support" → true branch

Phase 5: End-to-End Scenario Testing

Scenario 5.1: Complete "Route to Support" Flow

Full Path:

          plaintext
          
        

Trigger (conversation_created)
  ↓
ask_question: "What is your name?"  [PAUSE]
  ↓ [User: "John"]
send_message: "Thanks, John!"
  ↓
choices: "Which service?"  [PAUSE]
  ↓ [User: "Technical Support"]
condition: if last_choice == "Technical Support"
  ↓ [TRUE]
assign: route to Support inbox
  ↓
send_message: "Connecting to support team..."
  ↓
[COMPLETED]

Manual Test Steps:

[ ] Load widget
[ ] Send: "Hi"

- Verify: Bot asks "What is your name?"

[ ] Send: "John"

- Verify: "Thanks, John!" appears - Verify: Choice buttons appear

[ ] Click: "Technical Support"

- Verify: "Connecting to support team..." appears - Verify: Conversation moves to Support inbox

[ ] Check inbox: Conversation should appear in Support

Monitor for Bugs:

[ ] Variables not available in interpolation
[ ] Messages in wrong order
[ ] Condition branching wrong
[ ] Inbox assignment fails
[ ] Pusher events delayed

Phase 6: Error Scenarios

Scenario 6.1: Missing Required Data

Setup:

Automation with variables in actions

Action:

Skip askquestion step (jump to sendmessage using variable)
Variable not defined

Expected:

❌ Message appears with empty variable: "You selected ." (graceful degradation)
OR ✅ Error logged, message not sent

What to Monitor:

Check <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">enrollments.variables</code> object
Verify interpolation handles missing variables

Scenario 6.2: Inbox Not Found

Setup:

assign step references non-existent inbox ID

Action:

Trigger automation to assignment

Expected:

❌ Error logged in backend
❌ Conversation NOT assigned
❌ Automation should not crash

Verify:

Check backend logs for error
Check conversation status (should remain open, not assigned)

Scenario 6.3: Agent Offline / Unavailable

Setup:

assign step to specific agent
Agent is offline

Action:

Trigger automation to assignment

Expected:

✅ Conversation assigned anyway (no availability check)
OR ⚠️ Queue conversation for next available agent (if implemented)

Phase 7: Variable & Context Testing

Scenario 7.1: Contact Attributes in Messages

Setup:

Contact created with phone, company, language
Automation message: "Hello {{contact.firstName}}, from {{contact.company}}"

Action:

Send message as logged-in contact

Expected:

✅ Message: "Hello John, from Acme Inc"

Scenario 7.2: Conversation Attributes in Conditions

Setup:

Condition: if conversation.language == "fr"
Contact set language to French

Action:

Trigger automation

Expected:

✅ Condition evaluates correctly based on contact.language

Phase 8: Performance & Real-Time

Scenario 8.1: Message Latency

Setup:

Automation with multiple steps
Browser dev tools Performance tab open

Action:

Send message to trigger
Measure time for response to appear

Expected:

Bot response appears within 2-3 seconds
No janky rendering or lag

Scenario 8.2: Concurrent Conversations

Setup:

Multiple browser tabs / incognito windows
Each with different visitor identities

Action:

Send message in each tab
Different automations should run independently

Expected:

✅ Each conversation enrolls in automation separately
✅ Variables don't leak between conversations
✅ All receive responses correctly

Debugging Checklist

If automation doesn't trigger:

          bash
          
        

# 1. Check automation status
curl http://localhost:3030/api/v1/automations/xjDJ2Haamnapqf9V7Y4h \
  -H "Authorization: Bearer <TOKEN>" | jq '.data.status'
# Should be: "live"

# 2. Check trigger matches
curl http://localhost:3030/api/v1/automations/xjDJ2Haamnapqf9V7Y4h \
  -H "Authorization: Bearer <TOKEN>" | jq '.data.trigger'
# Should have: type, config

# 3. Check enrollment created
# Look in Firestore → enrollments
# Filter by: conversationId & automationId
# Should have status "in_progress" or "paused"

If step doesn't execute:

Check Firestore <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">enrollments.{enrollmentId}</code>

- Look at <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">currentStepId</code> — is it correct? - Look at <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">variables</code> — did previous steps run?

Check backend logs

- Search for <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">[AutomationEngine]</code> - Look for step executor errors

Check Pusher events

- Browser console: <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">Pusher <channel> <event></code> - Should see <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">message:new</code>, <code style="font-family:ui-monospace,monospace;background:#f1f5f9;color:#dc2626;padding:0.1em 0.35em;border-radius:3px;font-size:0.875em">workflowQuestion</code>, etc.

If variable not available:

Check enrollment.variables object

- Should contain variableName = value

Check context passed to step

- Backend logs should show context.variables

Test interpolation

- Try hardcoding variable value in message - If that works, interpolation is broken - If that doesn't work, step execution is broken

Success Criteria

✅ Automation is working when:

[ ] Trigger fires correctly
[ ] First step executes (question/choice/action)
[ ] ask_question pauses and resumes correctly
[ ] Choices branch correctly
[ ] Variables are saved and available
[ ] Assignments route to correct inboxes
[ ] Messages appear in conversation
[ ] Multiple conversations don't interfere
[ ] No console errors
[ ] Pusher events arrive in real-time

❌ Bugs to report:

Variables not interpolating
Steps not executing in order
Branching takes wrong path
Assignments go to wrong inbox
Messages appear out of order
Automation stops mid-flow
Variables disappear
Slow responses (>5s)

Next Steps

Get fresh token — current one expired
Run Phase 2 — verify automation structure is correct
Run Phase 3 — test basic steps (ask_question, choices)
Run Phase 5 — full end-to-end scenario
Document bugs found and their exact steps to reproduce
Create Firestore queries to verify enrollment state

Automation End-to-End Testing Guide

Automation End-to-End Testing Guide

Prerequisites

Phase 1: Pre-Test Verification

1.1 Verify Automation Structure

Phase 2: Trigger Testing

Scenario 2.1: Initial Trigger Activation

Scenario 2.2: Frequency Control

Scenario 2.3: Schedule Control

Phase 3: Step Execution Testing

Scenario 3.1: ask_question Step

Scenario 3.2: Resuming From ask_question

Scenario 3.3: choices Step

Scenario 3.4: Resuming From choices

Phase 4: Action Steps Testing

Scenario 4.1: send_message Action

Scenario 4.2: assign Step

Scenario 4.3: route<em>to</em>inbox Step

Scenario 4.4: Condition Branching

Phase 5: End-to-End Scenario Testing

Scenario 5.1: Complete "Route to Support" Flow

Phase 6: Error Scenarios

Scenario 6.1: Missing Required Data

Scenario 6.2: Inbox Not Found

Scenario 6.3: Agent Offline / Unavailable

Phase 7: Variable & Context Testing

Scenario 7.1: Contact Attributes in Messages

Scenario 7.2: Conversation Attributes in Conditions

Phase 8: Performance & Real-Time

Scenario 8.1: Message Latency

Scenario 8.2: Concurrent Conversations

Debugging Checklist

If automation doesn't trigger:

If step doesn't execute:

If variable not available:

Success Criteria

Next Steps