Gina Gotthilf on growing Duolingo to 200 million users
For Duolingo, the world's most downloaded education app, growth is fundamentally about retention.
If users don’t stick around, they won’t learn and inevitably won’t share Duolingo with their friends. As VP of Marketing and Growth at Duolingo through 2017, improving retention was the top priority of Gina Gotthilf. In her five years there, she helped take Duolingo from 3 million users to more than 200 million. And these aren’t just signups who’ve gone dark; Duolingo users complete 7 billion lessons each month.
Gina stepped away from Duolingo at the beginning of 2018 to help non-profits better understand the growth and marketing principles that thrive in today’s tech world. So, I had her join me on the podcast to share her Duolingo learnings while they’re still fresh. Our chat covers how gaming principles influenced her growth experiments, her team’s most successful retention tactics, and much more. What follows is a lightly edited transcript of the interview.
This is part of an ongoing series of interviews about unlocking the potential of growth. If you enjoy the conversation and don’t want to miss the rest of the series, check out more episodes of our podcast. You can subscribe to it on iTunes, stream on Spotify or grab the RSS feed in your player of choice.
Adam: Gina, welcome to Inside Intercom. Can you give us a quick feel for your career to date and what you were doing most recently at Duolingo?
Gina: My career has been a zigzag. I studied philosophy and neuroscience in school, and what I most wanted to do afterward was work for a non-profit. I also wanted to pay rent in New York, and those two things weren’t compatible. Plus, I needed a visa to stay in the US, so I ended up going into marketing. It’s what I’ve been doing ever since.
I’ve worked with some agencies in New York and then went back to Brazil. Tumblr asked me to help them grow in Brazil and other Latin American countries, so I led their growth there. That’s how I got into growth, even though it wasn’t called “growth” at the time. I later opened my own company to help tech companies grow in Brazil and Latin American countries, because I realized there was a big need and I happened to be in this really advantageous position to do that. That’s how I started working with Duolingo: they were actually one of my clients.
They basically asked Tumblr, “Hey, you guys grew a bunch in Brazil last year, what did you do?”, and then they referred me.
Bridging the gap between marketing and growth
Adam: Your first role at Duolingo was in PR, right?
Gina: Yeah, I was a consultant, but the role wasn’t really called PR. It was more, “help us grow.” I ended up resorting to PR a lot, because it’s something I could do on my own. I didn’t have to touch product, it didn’t require any money, and I really felt like there was a cool story to tell. It’s similar to what I did with Tumblr. It worked out well in terms of getting users, and I ended up doing that all over the world.
Adam: How was the transition from classic marketing to working more with designers and engineers on the growth team? Did you find that difficult, or were there pieces of your background that actually helped make that an easier transition?
Gina: Nobody ever asks me this, and I think it’s such a good question, because it was really difficult. When you’re in marketing at a tech company, you’re not taken very seriously. Product people – engineers, designers – sometimes view marketing as this side thing that’s not really necessary. They think if you build something cool, people will come, and meanwhile you’re just there having lunches with people and doing whatever it is you’re doing. So to be in charge, and to have people who were engineers and designers report to me, was very intimidating. I wanted them to know that I was very serious, I was intelligent and I could help get things done.
Coming from a communications background helped me in ways that nobody really foresaw.
The skills are also very different. When you’re managing people in marketing or PR, you understand or have a lot of expertise in what they’re doing. When you’re managing someone in a completely different field, like engineering, design, product management or analytics, you don’t have experience doing what they do. They’re better than you at that thing.
I do think that coming from a communications background helped me in ways that nobody really foresaw. For example, I was really good at getting engineers to talk to me and tell me what was going on, what was on their minds, what difficulties they were facing and what kinds of projects they were interested in. They weren’t used to talking about themselves all that much. Getting designers to talk to engineers and understand each other was another part that really benefited from my communications background.
Adam: Do you think this is part of a larger trend of marketing and product roles blending, or are you more the exception to the rule?
Gina: I think it’s a trend, but I don’t know if it’s a trend that’s here to stay. There’s definitely a trend where suddenly marketers are being asked to do product-like things, whereas before we were kept completely separate, like church and state. Hopefully it’s an ongoing trend that marketers are going to be more and more responsible for real metrics and for numbers. For that to work, to be responsible for those metrics, you need to actually have something to do with the product so you can see whether what you’re doing is having the effect that you want.
Duolingo’s early growth experiments
Adam: One thing people are always curious about is how a startup got its first users. At Duolingo, one of the founders very famously gave a TED talk that resulted in the first couple of hundred thousand users. When you arrived, how many users did you have?
Gina: When I started there were three million users. Now Duolingo has more than 200 million users, which is crazy, but we were very lucky to have those three million.
Our co-founder Luis von Ahn is the guy the who invented the captcha – those things that you type into on the internet to prove that you’re not a bot. He ended up selling two companies to Google and he won a MacArthur genius grant, which led to the TED Talk. Through that TED Talk, he was able to discuss his new project, Duolingo, and we got a lot of our first users that way. That helped so much because we were able to A/B test very early on, which is something a lot of smaller startups struggle with.
Adam: What are some of those early A/B tests that stick with you most?
Gina: In the beginning, I was doing communications-related things and partnerships. I would travel all over the world to launch Duolingo for Japan, China, India, Korea, Turkey, all these places that I’ve never even been to. I became Head of Growth, which is this product team that was A/B testing stuff, in the last two years of my time at Duolingo. There were definitely a lot of A/B tests that were done before my time that can be credited with a lot of our growth too, but I’ll mention some of the most successful ones that my team did.
First, letting people sign up after checking Duolingo out. When you’re looking at your funnel, you normally want to get as many people as you possibly can to convert, or to do whatever it is that you want them to do. If I’m sending people to Duolingo.com, or getting people to download Duolingo, that’s when I have the most eyeballs on this. I want to get people to sign up right now, because then I can email them later. I don’t want to lose any of these people.
But it turns out that by letting people actually use Duolingo a little bit without signing up, which is something that was hard for us to do, we got a much better conversion rate. That was a blend of things done before my time and during my time.
Basically, we tested a lot of different variations. First, not making people sign up until a much later period, then asking them to sign up one time or two times or three times, and letting them dismiss the message. When I started the growth team, you would do a lesson and then it said, “Do you want to save your progress or discard your progress?”
“Discard your progress” was a big red button, so we asked ourselves, “What if people are clicking on this red button because it’s red, and not because they want to discard their progress, and then they discarded it? Now there’s no reason for them to continue, and now we lost that user.” Just making people see this thing that says “Hey, want to sign up?” and then, if the answer was no, letting them do it later was super successful.
The metrics that matter most
Adam: The most important thing for a product like Duolingo, where you have to go in and experience the success, find that “Aha” moment and continue working towards the ultimate goal of learning a language, is retention. What metrics were you monitoring most closely when it came to retention?
Daily active users was our non-bullshit metric.
Gina: Daily active users was our non-bullshit metric. That gave us a sense of whether what we were doing was or wasn’t working out. But there are other metrics that definitely were important, and every time we ran an A/B test we looked at a whole slew of metrics to make sure that we weren’t impacting one number out of luck or causing people to use Duolingo more but learn less. Those metrics included lessons completed, lessons completed per session, number of minutes spent on Duolingo per session and number of average total minutes per day per user. We also looked at number of people who got from a certain part in the language tree, which was our curriculum, to another part. We had different funnels to see if different experiments impacted whether people got further ahead or not from an education standpoint.
Adam: What were the types of “Aha” moments you tried to create for users so that they really felt like they were making enough progress and felt encouraged to continue?
Gina: The most important thing for us was to get the user to feel some sort of connection to the product. Duolingo is very cute and very intuitive. We use language that’s very friendly throughout our entire communication, be it email, notification or in the product, and same with our design.
The second thing is for people to feel like they’re learning something. That’s what they’re there for; they’re not just there to play a game. We thought about a bunch of different ways to do that, but there was none that fit really well with how the product is designed. For example, maybe we could show a billboard at the end of the lesson, or a photo of a newspaper piece, and then people would read it and think, “Whoa, I understand this now.” We tried to bake that into the experience in as much as possible and felt like that was the most core thing that we could do.
What the gaming world can teach us about retention
Adam: You said, “People aren’t there just to play a game, they are there to learn.” At the same time, your team did lean on a lot of the principles used in the gaming world. Did you find that that was a difficult tightrope to walk between gamification and learning?
Gina: All the time. Duolingo was meant to be a game from the get-go. That comes from our founders. They thought, “You know what? Learning is a drag. Learning a language takes forever. We need to find a way to get people to keep coming back,” so making it a game was baked in from the very beginning.
My team would do things like assign games to different members of the team. I’d say, “Okay, this week you’re going to play this game,” a top grossing game or top downloaded game, “and then we’re going to give a little three-minute presentation, each person, in our next meeting, about what we thought was effective in this game.” That could be something like how they onboard people. Look at this metric system for points. Look at how you have this set of points, but you also have this other type of points. Look at how you earn them and how one influences the other. We were constantly talking about game mechanics and applying them to Duolingo with the goal of getting people to stay interested in learning a language and automatically, almost like a habit, go back to Duolingo whenever they were bored, instead of to a normal game on their phone.
As you said, it was a fine line. Our team was the growth team, and we were just trying to get more users and users to stick around. There’s a separate learning team and a separate monetization team. Having those three metrics separate and having teams advocate for each one of them was really important. Otherwise, we could have just made a product that was super easy and fun, that people would play more of and stick around more and buy more things, but not learn anything at the end. Because if it’s easy, it’s more motivating, but you’re not learning. If it’s harder, you might be learning more, but then you might also give up more easily because it’s frustrating. We didn’t want to make something that was just there to entertain, or to make money and get users and be the next thing that everyone forgot a year later.
We definitely had to have hard conversations, and sometimes one team won, sometimes another team won. There were things that we A/B tested that hurt monetization, for example, or an A/B test that the monetization team launched that hurt our retention. Same goes with learning, although we prioritized learning above all at Duolingo, so it was a harder battle if you wanted to fight against the learning people.
Duolingo’s most successful retention tactics
Adam: One of the most well known retention aspects of Duolingo is the streak, an individual’s consecutive days of completing lessons. Can you walk me through what the experiments were that you ran for that and how you landed on it as being such a successful strategy?
Gina: The streak preceded my time. I don’t know actually who came up with that, but it’s something that’s used a lot in games. When you do something several days in a row you can keep a streak. We ran a lot of experiments with that, even in my time, because we realized that introducing this concept of a streak really helped with retention massively. Basically, if you use Duolingo several days in a row, you can see this number going up, and the day that you forget to use it, that number turns to zero and you have to start again. That was super effective, because when you’re building your streak you feel like you’re investing into the app and you’re building something that you don’t want to lose.
In my team, we took that and experimented with a bunch of different things, like making it harder to lose your streak or getting more of a chance to earn back your streak by doing some exercises. Eventually it became a monetization thing where if you lose your streak on one day you can pay a fee and then you can repair it. But we did play around with that a lot, because we knew it was one of our most effective levers in terms of retention. It was worth paying a lot of attention to that.
It was similar with notifications, which have a huge effect. We were able to improve conversions over notifications by around 5 percent just by testing the copy or testing the timing of sending them. There’s just certain things like the streak or the notifications that have a high leverage in the sense that if you mess with them, you’ll see big numbers up or big numbers down. Those are really worthwhile with your time.
Another one worth mentioning is we created badges. This came from FourSquare back in the day, but there’s a lot of different apps and games where you can collect badges as you go. We really wanted to introduce that into Duolingo, but actually when we first tried it failed massively. We thought that badges didn’t work for a whole year, and then decided to try it again and it was so successful. Talking about all of these different drawbacks to other metrics that I was mentioning before, this was one case where it helped retention, it helped monetization, it helped learning, it helped everything, because we could just tell people, “Do this for a badge.”
If you offer badges to people to display different behaviors that are beneficial to them and are also beneficial to the app. For example, use Duolingo every day is an obvious one. It’s not a badge, but if you have an X-day streak, then you earn a badge, so you really want to get there. Use Duolingo in the morning, use Duolingo at night, or click on this tab and engage with someone. Invite a friend. Buy an item. There’s all this different stuff, some of which is related to learning and some of it which is just related to getting the user to see more of the app and experience more of it, which leads them to like it more and retain longer. Also it works for very short-term things too, like invite their friends and buy something on the app.
Adam: You test badges, you sit on it for a year, and then all of a sudden it’s this game changing thing that you added to the app. How did the team do such a 180 on that project? Was it how the experiment was run? Timing? What happened there?
Gina: It was the experiment, and I can credit my team with really insisting for badges way after I said, “You know guys, we shouldn’t waste our time. It’s too much of an investment. We don’t really know what the return is going to be.” When we first thought about introducing badges, we thought about why people liked them. “Maybe it’s because they really like this feeling getting something and feeling rewarded and like they did something good,” we thought, and we decided to replicate that.
We would often come up with a minimum viable test for things. So instead of creating an entire badge system, what’s the simplest thing we can do to test whether that’s going to have an effect or not. If it has an effect then we can go and build an entire badge system, and that’s going to take us months.
When you test something and the results are really good or really bad, don’t rest on your laurels.
We just did this thing where if you signed up you got this pop-up of a girl with balloons, and it was basically a congratulations for signing up thing. In retrospect it sounds really stupid, and I can’t even believe that we thought it was going to work, but we were really convinced that that would replicate that feeling of getting something, and if we saw an increase in metrics there, then we could start including more of those throughout the app and they would be badges.
Unsurprisingly, at least to me now, that did nothing, and unfortunately our conclusion was that it’s not worth investing in badges. Then we spent all this time just trying to shoot for lower-hanging fruit, things that you can test that take less time to develop, less time to design and just require less. Things that might bring us smaller gains, rather than trying to go for these big bets, for a really long time.
The whole time my team kept saying, “Let’s do badges, let’s do badges, let’s do badges, let’s bring it back.” Finally I was said, “Fine, if we’re going to do it this time then we should really think about why that didn’t work, and let’s really invest.” It took two or three months to fully design and implement badges. It seems simple, but where do they live on the app? Where do people see them? When do they get triggered? When do you receive them? How many of them are there? Are there going to be tiers? Can your friends see them? There’s so many layers to the badges, so it took us a really long time. But for us as a team it was probably our most successful experiment. Because we were so excited about it, we spent so much time on it, and it really, really worked out not only for us, but other teams’ metrics too.
Adam: For something like the badges or the streak or anything else you worked on, how important is it to optimize those things over time rather than resting on your laurels? Do you start to see diminishing returns, or when it’s something that big, can you put it out there and move on to something else for a while?
Gina: When you test something and the results are really good or really bad, you don’t rest on your laurels. You don’t just say, “That was really bad, let’s ignore it,” or “That was really good, let’s leave it.” Now that you know that has a really big impact, it’s worth your time going and trying to squeeze as much juice as you possibly can out of it, or improve that feature as much as you can.
With badges, we went on to make tiered badges. Now you can get level one, level two, level three, and one was gold and one was silver. We introduced other types of badges with other types of behaviors. Other teams were asking, “Hey, can you do a badge for getting people to do this,” because they wanted their metric to be helped by badges.
Same with streaks. We spent a long time thinking about what can we do with streaks and how can we make this experience even better, because we know it matters to people. If you try something and you get a whatever, a really tiny percent change, then great. If it’s statistically significant and it’s worth the engineering costs and the code decks that’s going into it, then you launch it, but you don’t go back and keep trying and trying because you already know that it’s not super impactful.
Gina’s quick growth tips
Adam: To close out, I’ve got a few lightning-round questions that we’re asking all the growth guests in this latest conversation series. First up, favorite growth tactic that you think is underused?
Gina: I would say PR is an underused growth tactic. People think of PR as this lame side thing where you have to talk to journalists and convince them to write about you and it’s annoying. If you work at a company where you really believe in what you’re doing, or if you have amazing talent, or if your CEO is incredible, or if you’re trying to change the world in some way that is actually significant and not Silicon Valley-esque, then you have something on your hands, and journalists are constantly looking for good stories and interesting people to interview.
PR is an underused growth tactic.
I saw a lot of very clear growth wins from getting PR in all of these different countries that I went to, and our little obsessive compulsion was to make sure that journalists included a link to our site or to our app every time they published, which is hard to do because journalists don’t like being told what to do. I started being called “Link-zilla” at Duolingo, because it was my number one obsession. That makes a huge difference on whether people click and go to your site or just never think about you again. It really matters!
Adam: Who in the growth community do you look up to or think that we have the most to learn from?
Gina: Personally I think Casey Winters is the bomb. He was at Pinterest for a long time and is now doing VC work. He’s just extremely smart, none of what he’s writing is just fluff, and he shares his knowledge freely.
Adam: One app or tool you can’t live without these days?
Gina: Flo. Flo is an app for tracking your period, and it’s important. It’s important because it makes life so much easier. It also helps you know whether or not you’re fertile, which helps you make life decisions.
Adam: What’s a common mistake that you’ve seen growth teams make when it comes to running experiments?
Gina: The number one thing is not really understanding statistical significance. You should be able to know if something is statistically significant or not, because otherwise your results mean nothing. A lot times people say they sent 60 emails, and 25 people did this, therefore the other one wins. And you’re like, “That doesn’t mean anything.” That’s a waste of time.
I also think that spending a lot of time on things that aren’t “big levers” is a big mistake. For example, we spent a really long time at Duolingo redesigning our emails, and it was largely driven by our design team. To their credit, Duolingo is where it is largely because of design, so they’re super important at Duolingo and they call a lot of shots. They really wanted to redesign our emails, and it had no effect on any of our numbers. From a branding perspective that makes sense. From a user experience perspective, sure. If your goal is to grow, to improve retention or click-through or whatever, that’s not really what you should be focusing on. It’s a huge time sink.
Really spend time to prioritize upfront. List all of the ideas that you and your team have in terms of things that you think can lead to growth, and come up with some hierarchy of your own. It can be something like, “This is how much effort this is going to take, and this is how much we think the return on this investment might be.” Actually spend time upfront weighing those two things.
Adam: Gina, thanks so much for sharing your insights with us. As we mentioned at the top of the show, you have just moved into a new challenge. You’re going to be helping non-profits with growth and marketing. What are you most excited about with this new challenge and how are you settling into it?
Gina: Leaving Duolingo was super hard. I was there for five years, and it’s a very big part of my identity today. People recognize me as “Gina from Duolingo”. It was a very difficult decision to make, but I’m really excited because I’ve always wanted to work with non-profits, and I think that one of the things they often lack is an understanding around marketing and growth and tech. It’s because they’re really focused on making an impact, not so much on telling the story or making sure that their conversion rates for donations are super high.
I hope to take all of these learnings from the for-profit world and apply them to non-profits and help them more effectively raise money, tell their story, get people to care, etc. I’m really excited about that, but I’m also nervous because it’s such a completely different sector. I have so much to learn, and it’s a completely different world in terms of how you communicate and what you can and cannot say, and what’s okay. So I’m learning on the job.
Adam: Best of luck in the new journey. Where can we keep up with what you’ve got going on in the meantime?
Gina: I’m pretty quiet on social media these days, but you can connect with me on LinkedIn. That’s the most effective way to connect with me.
