Improved Student Engagement in a WhatsApp chatbot-led learning model by 80%.
Challenge: Despite gamification for students, there was a high dropout rate and low engagement. (metrics: submission of activities, attendance in weekly sessions, and click rate of self-learning material).
Design Changes:
1. Crafted a narrative design tied to habit-building incentives as a quest around India’s cities.
2. Personified the chatbot and wrote a conversational copy to improve engagement by setting a fun and quirky tone.
3. Reduced friction in the onboarding user journey through behaviorally informed nudges.
Client: The Apprentice Project, Mumbai
Duration: 9 months
Skills: Participant Recruitment, Contextual Inquiry, A/B testing, User flows, User personas, Product Management, UX writing, Service Design, Chatbot design
Role in team: Worked as a User Researcher and product manager with 4 operations team members
“I am way too behind with the activity submission to care about the points, badges, and leaderboard”
The Apprentice Project (TAP) works to empower under-served students by equipping them with critical 21st-century skills using TAP Buddy, an artificial intelligence-powered WhatsApp chatbot.
TAP empowers students by providing them with the tools to succeed in our rapidly evolving world.
The idea was revolutionary until we realized that there are two sides to the story…
What the TAP team saw
What the students felt
Challenge
TAP suddenly transitioned from in-person teacher-led traditional learning to an online, smartphone-based, self-learning model owing to the pandemic and scalability factors.
Despite gamification and social incentives for students, there was a high dropout rate and low engagement
(metrics: submission of activities, attendance in weekly sessions, and click rate of self-learning material).
TAP buddy, a chatbot that sends video-based learning materials and prompts students to perform projects independently, did not promote a friendly student experience and wasn’t personalized to their needs.
Goal
My key role as the sole designer and behavioral science specialist was to uncover the needs and motivations of students enrolled in the TAP program.
The goal was to redesign the user flow and the gamification model personalized to different student personas.
Targets:
Increase the submission rates of activities for TAP students across different states of India
Reduce dropout rates from the program by 70%
Improve engagement with the learning material and ease activity submission via the TAP Buddy chatbot through a workflow redesign
This was my first time doing high-impact, real world problem solving. There were many unknown unknowns.. 
Process
Along with an operations team of 4, I led the entire product development process, from defining the problem statement to designing and testing the concept.
Literature review and competitor analysis
Research planning
Designing an interactive discussion guide for in-depth interviews
Conducting 100 in-depth interviews with students and stakeholders
Data Analysis and synthesis through thematic coding
Creating user personas and user journey maps for debriefing to leadership
Concept generation through co-design with students and stakeholders
Chatbot and gamification design
UX copy for chatbot
A/B testing
Target Group and Recruitment:
Students, aged 11-17, were selected through stratified sampling. They represented all groups: each city in TAP’s program, the choice of skill they had chosen to learn, high and low submission, and engagement scores.
They were informed about the research and gave consent to participate in it.
Market Research: What could we learn from others?
Duolingo and Khan Academy were two platforms studied in detail for behavioral gamification and incentive structures.
Duolingo’s chatbot had a unique conversational style that was fun, quirky, and engaging.
Also, the reminders for completing activities were timely and notifications delivered a sense of achievement.
Khan Academy’s onboarding process created a sense of ownership and competence through tiny step-by-step tutorials and choice of avatars which drastically reduced the probability of dropouts.
Literature Review
The self-determination theory highlighting the role of autonomy, relatedness, and competence formed the basis for the gamification redesign.
Octalysis’s model of extrinsic and intrinsic motivation aided in focusing more on the intrinsic factors of incentives rather than simply extrinsic motivators.
Need finding: Research Plan
Choice of research method: The team was in a “Learn” phase, having released the first version of the TAP online program, and wanted to learn if the model was serving the needs of the students.
Hence, we chose interviews and participatory design research to collect user stories and ideate changes for future versions.
In a span of 6 months, I conducted 100 interviews to uncover why the students had enrolled in the program, what kept them going, and the barriers they faced in the self-learning model.
The chatbot (TAP) buddy was personified by conducting a remote interactive co-design session with students. They described how they imagined TAP buddy to help them learn and a graphic designer collaborated with me to design the visuals of the chatbot as per the students’ needs.
It was imperative for the change to be bottom-up, designed by students instead of a top-down approach Research Findings and Data Synthesis
The insights hinted at a multifaceted root cause for poor engagement and retainment.-
The gamification was not resonating with the students.
They did not feel challenged and the gamification focused on extrinsic rewards like points, badges, and leaderboards that meant nothing to the students.
There was no focus on building mastery.
-
TAP Buddy, the Whatsapp chatbot had a disengaging and robot-like microcopy.
The options for responses were confusing for students.
-
Students had different needs based on their skills and will.
Some needed more scaffolding while others felt under-challenged.
3 attributes were uncovered as the guiding force for desired user behavior of high engagement and regular submissions:
High Will
High Skill
Knowing their “WHY” for joining the TAP program.
The User Flow Map revealed the top 2 factors for high dropout rates:
Major frictions in the onboarding process
Psychological distancing from the chatbot, due to a complex and confusing microcopy.
Based on the above students were clustered into 4 buckets:
USER TYPES
The Nudgees: High skill, low will- These students did not feel challenged enough in the activities and learning goals set for them; their interest dropped over time. If they face any constraints or logistical issues, they tend to drop out without making any effort to find a solution, for which they find excuses. The classes have not been firmly established in their routine.
The Climbers: High will, low skill- These students were very self-motivated but did not get the appropriate scaffolding when they could not cope with the learning materials.
My Life, My Rules: High skill, high will- These students felt challenged and enthusiastically accomplished all the tasks. However, they sometimes lacked the “Why” for joining the TAP program and didn’t stick to timelines and schedules. They have clear expectations from the classes and have a spirit of determination and fulfillment through learning Despite constraints of time, or network problems, they strive to complete and stay at par with the pace of learning. They might be excited to see the points but they are not particularly motivated to complete the activities for points.
The Lost One: Low skill, low will- These students struggled the most, lacking personalized help and scaffolding, leading to high dropout rates. Right from the beginning they seem disinterested and come with a closed mind. Some areas or topics might pique their interest, but that is short-lived and they tend to drop out very quickly. Some of them might have genuine reasons- structural constraints for attending the classes, but most of them are simply absent due to a lack of motivation.
From Insights to Interventions
The technical and resource constraints and small team meant that we needed quick and easy changes to implement, built on existing designs. We couldn’t build a new software or product. But, it forced us to think more creatively. We prioritized 3 changes:
Gamification narrative
Microcopy of the Chatbot
Design of the Chatbot
Chatbot Design Changes
GHUMO INDIA
(A virtual tour of India whilst learning)
Gamification narrative: Ghumo India was conceptualized as a gamified narrative that took students on a journey across India, visiting a new state every week with diverse cultures and flavors.
Chatbot redesign: Every week with a new State, TAP Buddy’s visual appearance, conversation style, tips, and nudges changed. Students got an activity specific to that State, for eg. creating a video recreating the dance form for Dehli.
Microcopy of Chatbot redesign: The UX copy for TAP Buddy was redesigned to motivate and engage the students by adding a unique conversational style and leveraging biases like loss aversion and social norms
Incentive structure redesign:
Badges, virtual stamps, and souvenirs were incentives for starting, completing activities and sharing feedback.
Group tours and games were designed as mini-challenges that improved collaboration and a sense of achievement.
Learnings and results
Personalized behavioral interventions and simplifying the chatbot language led to improved engagement and greater submission rates.
The dropout rate of students reduced by almost 80% in specific cities
The team could relate to the students and see their vantage point
