Providing AI Feedback

If you encounter a response that you believe your AI Agent should have handled in a different way, you can provide AI Feedback on your agent's response.
 

User Permissions Required

To provide response feedback: By default, users with Manager and/or Administrator roles can submit AI Feedback on any eligible lead response. Reps can only submit AI Feedback on their assigned leads.

Things to Consider when providing AI Feedback

AI Feedback is intended to help your agent learn and get better at handling future conversations, it is not intended to change the lead or customer status nor the outcome of the current conversation.

Submitting AI Feedback does not allow you to communicate directly with your agent, nor will it allow you to direct them on what to do or say next.

For situations where you'd like to influence a conversation in real-time, consider the following.

• If you think your agent should not be messaging your customer any longer, use Stop Messaging to prevent your RDA from sending any further messages.
• If you have recently talked with your contact and want your agent to momentarily put a hold on their messaging for 48 hours and then follow-up to make sure they're all set, use Skip to Follow Up.
• If you want your agent to provide specific information in real-time, use Reply as Assistant, or if you want to hand the conversation off to yourself or another rep, use Forward From Assistant.

Accessing AI Feedback 

You can access AI Feedback and AI View through the Conversation History which can be reached from the Conversica dashboard.

Additionally, the Conversation History can be reached through your alerts by accessing the Lead Profile.
 

Utilizing AI View

Before providing AI Feedback, it can be useful to understand what actions your assistant took on this response and what will happen next. Select AI View underneath the response you'd like to see more details for.

In AI View, you'll find Assistant Actions:

• Assistant Insights - this field reflects the intent of the contact's message as assessed by your agent
• Assistant Actions - these fields let you know the status that your assistant applied, if applicable, if an alert was sent, and whether or not the messaging continued.

After reviewing these actions, if you believe that your assistant has made a mistake, you will want to move on to submit AI Feedback.

Providing AI Feedback

Follow these simple steps to provide AI Feedback from the Conversation History which will be reviewed upon submission by an analyst at Conversica and used to help your AI Assistant improve on future conversations.

1. Click on the lead's response.
2. Click AI Feedback. 
3. You can provide AI Feedback on both the Sent Message and the Assistant Actions for the Lead Response. Before beginning, review the conversation history and start with Sent Message.  If you have any feedback to provide on your agent's generated response, please select the 'Thumbs Up' or 'Thumbs Down' icon to reflect your thoughts. 

4. Now take a look at the Lead Response. Because your agent did not send this message, you aren't providing feedback on the message your contact sent, instead you are looking at the Assistant Actions. The Assistant Actions are going to give you insight on what your agent will do in response to the lead. This is where you determine if you disagree with the next steps the agent took like stopping or continuing the conversation. 

   
    

5. If you selected the 'Thumbs Down' icon an Optional Comment box will appear. Please provide a brief description of what isn't working. For example: "My agent included some unnecessary details and was too exuberant in their messaging" or "The answer my agent provided to this question is outdated and we have a new answer to this question in our fact base. Does my fact base need to be updated?" 
6. Click Submit. 
   
    

AI Feedback FAQ

Q: I clicked on "AI Feedback" and got a message that AI Feedback is unavailable for this response. Why can't I submit feedback on this response?
A: There are three reasons why you might see this message when attempting to submit feedback. The first is that feedback has already been submitted on that response. Feedback should not be submitted twice on a response, as each feedback form will be reviewed by a human member of our team at Conversica.

The other reason is that only responses to messages sent by your AI assistant are eligible for AI Feedback. This means that if a lead responds to a message has been sent by you or a teammate using the yellow Reply as Assistant box, that response is not eligible for AI Feedback.

Lastly, you may encounter this message if you attempt to submit feedback without selecting a variable in the Response indicates drop-down menu, which is situated directly above the text box. Please ensure that you are selecting a value in the Response indicates menu before attempting to submit feedback on a response.

Q: How does my agent make decisions on how they respond to leads?
A: Your agent will review each response from the lead individually, looking for what we call “intents” from the lead. Your agent uses those identified intents to decide what to do next. For example, if a lead indicates in their first response that they are interested in a phone call, your agent will be able to identify that the lead is "interested" and "would like to be contacted by phone". Your RDA will send out the next message asking for or confirming their phone number. In the other direction, if a lead indicates that they are not interested in any further information, your RDA will stop any further messages from going out, and the conversation will be over.

Q: A lead was marked Further Action, but I then reached out to the lead myself and they’re now Satisfied. How do I get the status to change to Satisfied?
A: The status of your lead is dependent on the most recent response of the lead to your AI Assistant. Reaching out independently to Further Action leads is the right thing to do, but we do not currently support manually changing the status of the lead. To keep track of leads that still need action versus leads that have been taken care of, we recommend using the Stars (leads marked with statuses such as Further Action and Lead to Review are automatically starred). Once the lead is taken care of, unstarring the lead can help you organize which leads still need your attention. 

Q: If a customer responds in a different language, what does the AI do?
A: Certain agents, such as Auto Sales, have the ability to start a conversation in one language, but switch to a different language if the lead responds in that language or requests to have further responses be in that language. For example, an agent using the Spanish Real Time Lead Nurturing conversation can continue the conversation in Spanish if the lead requests it or responds in Spanish. If a lead responds in a language that the RDA does not cover in that conversation, for example, a lead responding in Spanish to an assistant using the conversation “Real Time Chat”, that lead will be marked Lead to Review.

Q: If a lead or customer has already been marked as a Response to Review/Lead to Review and deactivated, but later emails that they want a call or email does the Revenue Digital Assistant respond?
A: If your lead sends a response to your agent that causes the assistant to mark the lead with “Stop Messaging” or “Lead to Review”, but then the lead sends another message that indicates further interest, your agent will start the conversation back up from where it left off. The exceptions to this rule are if the lead is marked “Do Not Contact”, your agent will be unable to send any more messages or have Reply as Assistant available, or if the lead has been manually stopped from within the dashboard, alerts, or by a status used in your integrated system. Leads that are manually stopped will not have the ability to have your RDA pick up messaging if the lead replies with interest. In the rare event that a lead/contact is unsubscribed, please submit feedback with a comment requesting that the lead is removed from the 'Unsubscribe' list.

Q: Sometimes when I submit AI Feedback, the status of a lead changes, or a lead will begin to message again, but sometimes nothing happens. Does submitting AI Feedback actually do anything to the lead?
A: When AI Feedback is submitted on a response, our AI Operations team from Conversica reviews that response, and if it is determined that the AI trained that response incorrectly, the response is updated. In the case of a lead being retrained from “Lead to Review” to “Hot Lead”, the status of the lead will change, and your agent would continue messaging, because Lead to Review will stop messaging, whereas a “Hot Lead” would have your agent continue messaging. Your agent will resume messaging a lead after being updated if the lead was misclassified as a status that would stop the lead from messaging (such as Stop Messaging or Lead to Review) and is reclassified as a status that would continue messaging (Such as Skip to Follow Up or Hot Lead). Some exceptions to this rule are leads that are manually stopped will not have the ability to have your AI-powered assistant pick up messaging if the lead replies with interest. An additional exception is if a response is retrained, but there are responses from the lead that have come in after the retrained response that have been trained in a different way than the initial response. For example, if a lead classified as “Continue Messaging”, where it should have been classified as “Check Back Later”, and the AI-powered assistant has already continued their messaging based on the initial classification, your AI-powered assistant cannot change the track of the conversation and move to “Check Back Later” instead.

Q: How can I tell what my agent is going to do next? Is there a way to see in the Conversation History which Exchange we're in so that I can guess how the next message will look? Am I able to tell the status of a (not Hot) lead when I'm looking at the Conversation History?
A: To see what your agent is going to do next for a particular lead, take a look at the Assistant Actions found under AI Feedback on the most recent response for your lead in the Conversation History. The Assistant Actions will tell you how the agent treated that response, and what the next step will be, whether that step is to stop messaging, or qualify the lead, or skip to follow up.

Q: Why does she show that she has a next message scheduled for some leads in a few minutes and for others in a few days and for others none at all?
A: A scheduled message will appear in the Conversation History when the conversation is still active and your Revenue Digital Assistant has more messages lined up to send to the lead. When a lead response is received, your agent will respond to that lead right away instead of scheduling a message for the future, unless the lead is in the “Hot/Interested” window, then there will be a gap of a few days (48 hours is default) before the RDA moves to the follow up messaging. This 48 hour window can also be triggered by using the Skip to Follow Up button in your dashboard or alerts. If the lead is not responding, there are a certain number of attempts that your AI-powered assistant will make to reach that lead (usually between 4 and 7), and those attempts will be staggered at least 24 hours apart (your AI-powered assistant will never message an unresponsive lead twice in one day), up to 9 days apart. The gap between messages in the same exchange is called the “cadence” of the conversation, and if you’d like to look into adjusting cadence, your Customer Success Manager can help.

Q: Under AI Feedback, why does the next step after Hot say "Qualify"? Isn't it already qualified at that point?
A: “Qualify” is the name of the exchange that comes after the first exchange, which is “Engage”. Some conversations, often dependent on industry, have leads that are marked “Hot” after a positive response to the “Engage” exchange (the first one). For some conversations, leads aren’t marked “Hot/Interested” until after a positive response to the “Qualify” exchange. In that “Qualify” exchange, your Revenue Digital Assistant will ask for or confirm their phone number or preferred contact method.

Q: Sometimes leads say they might be interested in a different offering and yet you still mark the lead as Hot. Why?
A: Your Revenue Digital Assistant is able to pick up on the interest of a lead based on positive phrases. If a lead is expressing interest in general, it is very likely that your RDA will mark that lead as “Hot/Interested” and continue the conversation, unless there are other intents in the message that counteract the “interested” intent. As a reminder, “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. Your RDA uses those identified intents to decide what to do next.

Q: Why do some questions get the lead marked as Hot and some as Lead to Review?
A: Your Revenue Digital Assistant is looking for various “intents” in every response that is received. “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. Your RDA uses those identified intents to decide what to do next. Your RDA may mark a response with a question as “Hot” if there are “interested” intents in the message. If there is a question in the response, but no interested intents, that lead will often get marked as Lead to Review, as that indicates that the lead wants an answer to their question, but isn’t necessarily interested in what the RDA is asking about. For additional information, check out the article Understanding Commonly Confused Responses. 

If your industry is eligible for Conversica Answers, leads who ask questions that qualify as a question set up in your Conversica Answers tab will have their question answered in the next response from the lead.

Q: Why can't my Revenue Digital Assistant guess at a good time to Check Back Later instead of marking it as a Lead to Review?
A: Check Back Later is specifically designed to check back with a lead during a more convenient time indicated by the lead. When the Check Back Later date rolls around, your AI-powered assistant will ask the lead something along the lines of “you mentioned that now may be a good time to connect.” This type of response wouldn’t make sense if the lead does not provide some sort of time frame in their original response, such as “check back with me in May”, or “I’ll be looking more seriously in the fall”. Leads without these specific timeframe indicators, who say something like “check back with me later” or “I’ll be ready sometime in the future” are marked Lead to Review so that they can possibly be brought into a ReEngage campaign at an appropriate time.

Q: Why can't my Revenue Digital Assistant weed out people who are just looking for free stuff or training and will never buy?
A: Your RDA makes decisions based on intents found within the lead’s response. “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. If there are “interested” intents in the message, your RDA will continue to message those leads. Unless the lead specifically says that they are not interested in buying or some other intent that indicates that they are not qualified lead, your RDA will do their best to connect them with a rep.

Q: Why does the Revenue Digital Assistant sometimes mark empty emails as Hot? Or emails that only contain "Hello"?
A: Occasionally, lead responses can come through the system weirdly formatted, such as the main body of the message coming through as one response, but then the signature, or an “empty” message coming in soon after. Those are all considered part of the same response, and as long as that second message doesn’t have any intents that are opposite of the intents in the original message, then they’ll get trained the same way. So if a lead sends in an interested response and it gets marked as “Hot”, but then due to weird formatting, an empty message comes in directly after that, the empty message will be trained the same way (Hot) as the original response.

Your RDA will consider responses that simply say “Hello [Assistant Name]” as “Hot/Interested” because the lead took the time to respond in the first place, and that is considered enough of an interest to continue messaging.