Providing AI Feedback

If you encounter a response that you believe your Revenue Digital Assistant™ should have handled in a different way, you can provide AI Feedback on your AI-powered assistant's response. 

By providing your AI-powered Assistant with feedback, you will be helping to boost their intelligence and accuracy when handling future conversations.

User Permissions Required

To provide response feedback: By default, users with Manager and/or Administrator roles can submit AI Feedback on any eligible lead response. Reps can only submit AI Feedback on their assigned leads.

Things to Consider when providing AI Feedback

AI Feedback is intended to help your AI-powered assistant learn and get better at handling future conversations, it is not intended to change the lead or customer status nor the outcome of the current conversation.

Submitting AI Feedback does not allow you to communicate directly with your Revenue Digital Assistant, nor will it allow you to direct your RDA on what to do or say next.

For situations where you'd like to influence a conversation in real-time, consider the following.

• If you think your Revenue Digital Assistant should not be messaging your customer any longer, use Stop Messaging to prevent your RDA from sending any further messages.
• If you have recently talked with your lead and want your AI-powered assistant to momentarily put a hold on their messaging for 48 hours and then follow-up to make sure they're all set, use Skip to Follow Up.
• If you want your AI-powered assistant to provide specific information in real-time, use Reply as Assistant, or if you want to hand the conversation off to yourself or another rep, use Forward From Assistant.

Accessing AI Feedback 

You can access AI Feedback and AI View through the Conversation History which can be reached from the Conversica dashboard.



Additionally, the Conversation History can be reached through your alerts by accessing the Lead Profile.

Utilizing AI View

Before providing AI Feedback, it can be useful to understand what actions your assistant took on this response and what will happen next. Select AI View underneath the response you'd like to see more details for.

In AI View, you'll find Assistant Insights and Assistant Actions:

• Assistant Insights - this field lets you know what your assistant saw as the overall meaning of the message.
• Assistant Actions - these fields let you know the status that your assistant applied, if applicable, if an alert was sent, and whether or not the messaging continued.

After reviewing these actions, if you believe that your assistant has made a mistake, you will want to move on to submit AI Feedback.

Providing AI Feedback

Follow these steps to provide AI Feedback from the Conversation History which will be reviewed upon submission by an analyst at Conversica and used to help your AI Assistant improve on future conversations.

1. Click on the lead's response.
2. Click AI Feedback. 
3. Review Assistant Actions to see how your AI Assistant interpreted and handled the lead's response. If you still think that the response was handled incorrectly, proceed to Step 4.
4. Scroll to the Response Indicates: drop-down menu and choose the most accurate description of the response.**
5. Add a comment to provide additional details regarding your AI Feedback (optional).
6. Click Submit. 

**To help identify and select the most accurate response, the below chart provides a description and common examples of the choices available in the Response Indicates drop-down menu.
 

AI Feedback FAQ

Q: How does my AI-powered assistant make decisions on how they respond to leads?
A: Your Revenue Digital Assistant will review each response from the lead individually, looking for what we call “intents” from the lead. “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. Your RDA uses those identified intents to decide what to do next. For example, if a lead indicates in their first response that they are interested in a phone call, your AI-powered assistant will be able to identify that the lead is "interested" and "would like to be contacted by phone". Your RDA will send out the next message asking for or confirming their phone number. In the other direction, if a lead indicates that they are not interested in any further information, your RDA will stop any further messages from going out, and the conversation will be over.

Q: One of my leads said that they had some questions for me, but my Revenue Digital Assistant stopped messaging. Why does my AI-powered assistant make mistakes?
A: Your Revenue Digital Assistant is smart, but can get confused. If a message has conflicting intents, or intents that your RDA thinks conflict, it can cause them to take an incorrect action with a response. For example, if the RDA asked the question "is this a good email address to reach you?", and the lead replies with "Yes, this is my correct email, thank you for reaching out. However, I don't think I'm ready for a phone call at this time, I'm just starting to do my own research." There is a positive question answer here where the lead confirms their email address, but at the end of the message, the lead indicates that they don't need any additional contact at this time. Your AI-powered assistant may see the positive question response, and continue on with the messaging. If your RDA misclassifies a lead like that, submitting AI Feedback using the methods above will allow Conversica's team to assess the decision your RDA made and retrain the AI if necessary, so they can be "smarter" for next time.

Q: I clicked on "AI Feedback" and got a message that AI Feedback is unavailable for this response. Why can't I submit feedback on this response?
A: There are three reasons why you might see this message when attempting to submit feedback. The first is that feedback has already been submitted on that response. Feedback should not be submitted twice on a response, as each feedback form will be reviewed by a human member of our team at Conversica.

The other reason is that only responses to messages sent by your AI assistant are eligible for AI Feedback. This means that if a lead responds to a message has been sent by you or a teammate using the yellow Reply as Assistant box, that response is not eligible for AI Feedback.

Lastly, you may encounter this message if you attempt to submit feedback without selecting a variable in the Response indicates drop-down menu, which is situated directly above the text box. Please ensure that you are selecting a value in the Response indicates menu before attempting to submit feedback on a response.

Q: A lead called me and asked to unsubscribe from the AI Assistant’s messages, but my Revenue Digital Assistant is still reaching out. How can I unsubscribe my lead?
A: Submitting AI Feedback requesting a lead to be unsubscribed will not allow the lead to be put on a Do Not Contact list. A lead can only be unsubscribed through the AI if the lead asks in a response to “unsubscribe”, or states “never send me another email ever again”, or something of that sort. The lead can also manually opt out through the “Unsubscribe” link at the bottom of your AI-powered assistant's attempts. If the lead hasn’t done either of the previous examples, but have asked you to unsubscribe from the AI-powered assistant’s messages for them, start by manually stopping the lead in the dashboard (or having a manager with appropriate permissions do it for you). Then submit a request to support@conversica.com with the lead name and email address that you’d like to be unsubscribed. Once a lead is unsubscribed, your Conversica assistant will not be able to send any further messages in that conversation or in any future conversations, and Reply as Assistant will no longer be available.

Q: A lead was marked Further Action, but I then reached out to the lead myself and they’re now Satisfied. How do I get the status to change to Satisfied?
A: The status of your lead is dependent on the most recent response of the lead to your AI Assistant. Reaching out independently to Further Action leads is the right thing to do, but we do not currently support manually changing the status of the lead. To keep track of leads that still need action versus leads that have been taken care of, we recommend using the Stars (leads marked with statuses such as Further Action and Lead to Review are automatically starred). Once the lead is taken care of, unstarring the lead can help you organize which leads still need your attention. 

Q: If a customer responds in a different language, what does the AI do?
A: Certain Revenue Digital Assistants, such as Auto Sales, have the ability to start a conversation in one language, but switch to a different language if the lead responds in that language or requests to have further responses be in that language. For example, a Revenue Digital Assistant using the Spanish Real Time Lead Nurturing conversation can continue the conversation in Spanish if the lead requests it or responds in Spanish. If a lead responds in a language that the RDA does not cover in that conversation, for example, a lead responding in Spanish to an assistant using the conversation “Real Time Chat”, that lead will be marked Lead to Review.

Q: If a lead or customer has already been marked as a Response to Review/Lead to Review and deactivated, but later emails that they want a call or email does the Revenue Digital Assistant respond?
A: If your lead sends a response to your AI-powered assistant that causes the assistant to mark the lead with “Stop Messaging” or “Lead to Review”, but then the lead sends another message that indicates further interest, your AI-powered assistant will start the conversation back up from where it left off. The exceptions to this rule are if the lead is marked “Do Not Contact”, your Revenue Digital Assistant will be unable to send any more messages or have Reply as Assistant available, or if the lead has been manually stopped from within the dashboard, alerts, or by a status used in your integrated system. Leads that are manually stopped will not have the ability to have your RDA pick up messaging if the lead replies with interest.

Q: Sometimes when I submit AI Feedback, the status of a lead changes, or a lead will begin to message again, but sometimes nothing happens. Does submitting AI Feedback actually do anything to the lead?
A: When AI Feedback is submitted on a response, an AI Conversation Analyst from Conversica reviews that response, and if it is determined that the AI trained that response incorrectly, the response is updated. In the case of a lead being retrained from “Lead to Review” to “Hot Lead”, the status of the lead will change, and the AI-powered assistant would continue messaging, because Lead to Review will stop messaging, whereas a “Hot Lead” would have your AI Assistant continue messaging. Your AI-powered assistant will resume messaging a lead after being updated if the lead was misclassified as a status that would stop the lead from messaging (such as Stop Messaging or Lead to Review) and is reclassified as a status that would continue messaging (Such as Skip to Follow Up or Hot Lead). Some exceptions to this rule are leads that are manually stopped will not have the ability to have your AI-powered assistant pick up messaging if the lead replies with interest. An additional exception is if a response is retrained, but there are responses from the lead that have come in after the retrained response that have been trained in a different way than the initial response. For example, if a lead classified as “Continue Messaging”, where it should have been classified as “Check Back Later”, and the AI-powered assistant has already continued their messaging based on the initial classification, your AI-powered assistant cannot change the track of the conversation and move to “Check Back Later” instead.

Q: How can I tell what my Revenue Digital Assistant is going to do next? Is there a way to see in the Conversation History which Exchange we're in so that I can guess how the next message will look? Am I able to tell the status of a (not Hot) lead when I'm looking at the Conversation History?
A: To see what your RDA is going to do next for a particular lead, take a look at the Assistant Actions found under AI Feedback on the most recent response for your lead in the Conversation History. The Assistant Actions will tell you how the RDA treated that response, and what the next step will be, whether that step is to stop messaging, or qualify the lead, or skip to follow up.

Q: Why does she show that she has a next message scheduled for some leads in a few minutes and for others in a few days and for others none at all?
A: A scheduled message will appear in the Conversation History when the conversation is still active and your Revenue Digital Assistant has more messages lined up to send to the lead. When a lead response is received, your RDA will respond to that lead right away instead of scheduling a message for the future, unless the lead is in the “Hot/Interested” window, then there will be a gap of a few days (48 hours is default) before the RDA moves to the follow up messaging. This 48 hour window can also be triggered by using the Skip to Follow Up button in your dashboard or alerts. If the lead is not responding, there are a certain number of attempts that your AI-powered assistant will make to reach that lead (usually between 4 and 7), and those attempts will be staggered at least 24 hours apart (your AI-powered assistant will never message an unresponsive lead twice in one day), up to 9 days apart. The gap between messages in the same exchange is called the “cadence” of the conversation, and if you’d like to look into adjusting cadence, your Customer Success Manager can help.

Q: Under AI Feedback, why does the next step after Hot say "Qualify"? Isn't it already qualified at that point?
A: “Qualify” is the name of the exchange that comes after the first exchange, which is “Engage”. Some conversations, often dependent on industry, have leads that are marked “Hot” after a positive response to the “Engage” exchange (the first one). For some conversations, leads aren’t marked “Hot/Interested” until after a positive response to the “Qualify” exchange. In that “Qualify” exchange, your Revenue Digital Assistant will ask for or confirm their phone number or preferred contact method.

Q: Sometimes leads say they might be interested in a different offering and yet you still mark the lead as Hot. Why?
A: Your Revenue Digital Assistant is able to pick up on the interest of a lead based on positive phrases. If a lead is expressing interest in general, it is very likely that your RDA will mark that lead as “Hot/Interested” and continue the conversation, unless there are other intents in the message that counteract the “interested” intent. As a reminder, “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. Your RDA uses those identified intents to decide what to do next.

Q: Why do some questions get the lead marked as Hot and some as Lead to Review?
A: Your Revenue Digital Assistant is looking for various “intents” in every response that is received. “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. Your RDA uses those identified intents to decide what to do next. Your RDA may mark a response with a question as “Hot” if there are “interested” intents in the message. If there is a question in the response, but no interested intents, that lead will often get marked as Lead to Review, as that indicates that the lead wants an answer to their question, but isn’t necessarily interested in what the RDA is asking about. For additional information, check out the article Understanding Commonly Confused Responses. 

If your industry is eligible for Conversica Answers, leads who ask questions that qualify as a question set up in your Conversica Answers tab will have their question answered in the next response from the lead.

Q: Why can't my Revenue Digital Assistant guess at a good time to Check Back Later instead of marking it as a Lead to Review?
A: Check Back Later is specifically designed to check back with a lead during a more convenient time indicated by the lead. When the Check Back Later date rolls around, your AI-powered assistant will ask the lead something along the lines of “you mentioned that now may be a good time to connect.” This type of response wouldn’t make sense if the lead does not provide some sort of time frame in their original response, such as “check back with me in May”, or “I’ll be looking more seriously in the fall”. Leads without these specific timeframe indicators, who say something like “check back with me later” or “I’ll be ready sometime in the future” are marked Lead to Review so that they can possibly be brought into a ReEngage campaign at an appropriate time.

Q: Why can't my Revenue Digital Assistant weed out people who are just looking for free stuff or training and will never buy?
A: Your RDA makes decisions based on intents found within the lead’s response. “Intents” are key words and phrases in a response that provide insight into what the lead is trying to say. Some examples are "interested", "preferred contact method", "contact me in a few months”. If there are “interested” intents in the message, your RDA will continue to message those leads. Unless the lead specifically says that they are not interested in buying or some other intent that indicates that they are not qualified lead, your RDA will do their best to connect them with a rep.

Q: Why does the Revenue Digital Assistant sometimes mark empty emails as Hot? Or emails that only contain "Hello"?
A: Occasionally, lead responses can come through the system weirdly formatted, such as the main body of the message coming through as one response, but then the signature, or an “empty” message coming in soon after. Those are all considered part of the same response, and as long as that second message doesn’t have any intents that are opposite of the intents in the original message, then they’ll get trained the same way. So if a lead sends in an interested response and it gets marked as “Hot”, but then due to weird formatting, an empty message comes in directly after that, the empty message will be trained the same way (Hot) as the original response.

Your RDA will consider responses that simply say “Hello [Assistant Name]” as “Hot/Interested” because the lead took the time to respond in the first place, and that is considered enough of an interest to continue messaging.