Quelles sont les bonnes pratiques pour mener une session de test utilisateur de chatbot ?

Before diving into testing, clarity on goals is paramount. Understand what you want to achieve through testing—whether it's refining user interactions, improving response accuracy, or enhancing overall user satisfaction. Metrics, be it completion rates or user satisfaction scores, give you tangible benchmarks.

So, I think of it like throwing a party for your chatbot, and everyone's invited to give it a spin. It is very important to know what you want to learn; you want to see how they play it, what they enjoy, and what leaves them scratching their heads. In that sense, are you testing how users find information, how they navigate, or maybe how they feel about the chatbot's personality?

Vejo que a definição do público pode ser segmentada através de quais resultados estão buscando para o fluxo implementado, exemplo de fraseologias, semântica e contexto segue em uma linha mais textual de validação combinado ao público, voltada para dados qualitativos, quando há avaliações de fluxos transacionais, torna essencial métricas de conclusão de fluxo, retenção e também entendimento qualitativo do usuário não seguir adiante no fluxo proposto. Ter o registro de cada etapa do cliente nos fluxos para ambos os casos é essencial, pois mudanças serão necessárias e saber onde mudar é o básico. O diferencial é a tempestividade de atuação nas mudanças necessárias.

1. Clearly Define Objectives: Set specific goals and expectations for the chatbot user testing session to focus on relevant aspects of the bot's performance. 2. Diverse Participant Selection: Include a diverse group of users to ensure a range of perspectives and experiences are considered during testing. 3. Realistic Scenarios: Design realistic scenarios that reflect typical user interactions to evaluate the chatbot's effectiveness in practical situations. 4. Provide Minimal Guidance: Avoid providing excessive guidance during testing to mimic real-world usage and identify any usability issues. By following these best practices, you can gather valuable insights and refine your chatbot to better meet user needs and expectations.

In chatbot testing, we want users to be happy. We check if the chatbot does tasks well and responds fast. We aim for no errors and a smooth talk. We want users to like it, come back, and tell us what they think. We look at numbers like task success and response time to make the chatbot better and keep users smiling.

Testing your chatbot is crucial to meeting user expectations. Methods such as surveys, interviews, usability testing, and tools such as Dialogflow, UserTesting, and Google Analytics guarantee success by ensuring user satisfaction.

Choose methods and tools: - Select from surveys, interviews, observations, think-alouds, usability tests, and A/B tests. - Utilize chatbot platforms (e.g. Dialogflow, IBM Watson, Rasa), user testing platforms (e.g. UserTesting, UserZoom, Userlytics), and analytics tools (e.g. Google Analytics, Mixpanel, Dashbot). - Tailor choices based on goals, budget, and resources. - Ensure chatbot meets user expectations and achieves success.

I can think of another way of testing these chatbots, "Introduce targeted feedback questions at the end of each conversation." This method goes beyond assessing the chatbot's performance, it focuses on user expectations across various scenarios. For instance, following a customer support chat, inquire about the chatbot's helpfulness ( may be CSAT, SUS, or NPS questions). By strategically placing feedback prompts, you gather insights into user interactions, preferences, and expectations. This not only refines the chatbot's behavior but also fosters a responsive and user-centric design, boosting its adaptability to diverse user needs and scenarios.

Choosing appropriate testing methods and tools is vital for a chatbot's success. Methods like surveys and usability tests, along with tools like Dialogflow or UserTesting, help gather qualitative and quantitative insights. A strategic combination of these resources ensures thorough evaluation and optimization of the chatbot.

Choosing the right methods and tools is the next crucial step. Depending on your goals, you may opt for qualitative methods like user interviews or focus groups, or quantitative methods like surveys. Tools like usability testing platforms can offer insights into how users interact with your chatbot and where they face issues. In my projects, combining different methods has provided a comprehensive understanding of the chatbot's performance.

When recruiting users for chatbot testing, aim for diversity in backgrounds, ages, and experiences. This variety helps uncover a wide range of perspectives. Once recruited, segment users based on factors like familiarity with technology, frequency of chatbot use, or specific needs. This segmentation helps analyze feedback more effectively, ensuring your chatbot meets the diverse needs of its users.

Find people who are like the ones you expect to use your chatbot. This could be current users or from testing platforms. Divide them into groups based on things like how much they know, their feelings, and their background. This helps make sure your chatbot works well for different types of users and makes everyone feel heard.

1. Define Segments: Clearly define user segments. 2. Recruitment Criteria: Establish clear criteria for diverse participant recruitment. 3. Channels: Use various channels for recruitment. 4. Consent: Ensure informed consent from participants. 5. Diversity: Strive for diversity in the participant pool. 6. Pilot Test: Conduct a small-scale pilot test. 7. Task Scenarios: Design realistic and relevant task scenarios. 8. Moderator's Guide: Equip moderators with a well-prepared guide. 9. Task Mix: Include a mix of simple and complex tasks. 10. Post-Task Feedback: Encourage participant feedback after each task.

Focus on your segment but don't forget the rest of the users. It could help found issues that could be over step by native users

The recruitment and segmentation of users are vital. Ensure that your test users represent your chatbot’s target audience. This might involve segmenting users based on demographics, tech-savviness, or previous interactions with chatbots. Involving a diverse group of users helps in identifying a wide range of potential improvements, making your chatbot more robust and user-friendly.

Before even having a session, see what you're users have questions about! If you have a call center or help desk, look at what tickets take up a majority of resources and go from there. Always try and see where data can make your decision process easier. At Microsoft, I created an AI chatbot for an approximately 5,000-user base project. To decide on what questions I was going to prioritize to have the most complete process for first, I looked at our logs. 80% of tickets were created to reset a user password. When walking through the AI chabot with users, I made sure this flow worked and used it as a model for the rest. That meant at least 80% of people would already have success upon launch! Not bad.

Surveys: Use closed-ended questions to gather data on user demographics, experience levels, and their perceived features or functionalities. This can help identify segments with differing mental models. Clickstream Analysis: Analyze user interaction data like navigation paths and clicks to understand user behavior patterns and identify potential areas of confusion or misalignment with their expectations.

Prepare and conduct session: - Develop test script outlining purpose, tasks, and questions. - Provide consent form and disclaimer for privacy. - Set up comfortable environment and equipment. - Brief users on chatbot and session expectations. - Monitor and record user interaction and feedback. - Debrief and thank users for participation.

Create a structured testing plan with scenarios and tasks for users to complete. Create guidelines and scripts as needed, but also leave room for unscripted interactions. Before the session, make sure the chatbot is running smoothly. During the conducting session, take note of how users interact with the chatbot, including their actions and reactions. Encourage participants to think aloud, which will reveal insights into their thought process. Provide minimal guidance to ensure natural interaction patterns. Record the sessions for further analysis while adhering to privacy and consent regulations.

Something that many people forget: make your candidates feel comfortable! Offer them some water, start with an easy (and ideally fun) first question. Remind them that these are not YOUR designs, but someone's else (so they will be more open to provide honest feedback). Also, if you have any gratification, give it to them BEFORE you start :)

Review user feedback and data after chatbot testing. Look for patterns and issues. Create a clear report with key insights and suggestions for improvement. Share it with your team to enhance the chatbot's performance.

Analyze and report results: - Organize, clean, and categorize data. - Use qualitative or quantitative methods to label and group data. - Interpret and synthesize data into key insights and findings. - Present findings using charts, tables, or diagrams. - Communicate results through reports, presentations, or dashboards. - Ensure user testing session is effective and enjoyable.

Focus on specific tasks or goals: Understand how users approach and execute specific tasks within your product or domain. Compare with existing models: Analyze competitor products or similar domains to identify established mental models users might be familiar with. Use visual aids: Encourage users to draw diagrams or sketches to represent their understanding of the system. Test and iterate: Don't assume your first interpretation is correct. Continuously test your understanding of the user's mental model through prototypes and user feedback.

Gosto de separar o resultado de um teste em duas partes. Na primeira eu categorizo em: - Usuário conseguiu concluir a tarefa com facilidade - Usuário conseguiu concluir com algumas dificuldades - Usuário não conseguiu concluir Dessa forma você poderá ter um panorama geral do resultado, principalmente quantitativo. Na segunda parte eu sigo categorizando e separando como no texto acima. Gosto de utilizar um repositório de pesquisas e testes para que seja encontrado com facilidade o material.

Pro tip: Use dwell time over specific parts of the conversation to identify where your users struggled the most :) With long conversations, GSR (Goal Success Ratio) or TCT (Task Completion Time) may not be useful to identify where in the conversation your spent more time.

Simple question... is this enough? All the content above is great but it's still regular user testing methodology. What has this got to do with a chatbot?! Remove "chatbot" and put something else, it still sounds like a metholodgy or process. Where is the core objective of the chat bot in this testing. So now my question is, is the focus on the methodology or is it on the value the chat bot brings to the table? How do we determine the value of the chat bot? Is it only about getting AI/ML right or could we do more with it?! Is it only about training it to speak right answers or are there more "missed opportunities". Let's define the What first, we get to the How later!

Remember, researching the user's mental model is an ongoing process. Be open to adapting your design as you learn more about how users think and interact with your product

Measure Your Chatbot Success: 1. On a scale of 1 to 10, how satisfied am I with the answer the chatbot is providing me. 2. Do I have a feedback loop which informs me if the chatbot gave the right answer? 3. Is the answer too short or too long? 4. How many questions did it take me to get the right answer? 5. Is the chatbot using a Machine Learning model to keep on learning how to answer better?

Ultimately chatbot cannot replace real human interaction, so there should be some gap of experience between bot and real agent. So closing the gap here should be worth exploring. It's also worth considering whether to invest on chatbots or adding more agents based on your short term or long term business strategy.

Chatbot testing works very well with A/B testing to see if it's inclusion really produce sefficiency or positive User experience ( over other UI options)