Deliberate product improvements

It’s risky to try to improve any part of a product without understanding the job that it does for customers, and what their success criteria are.

We’ve been rolling out lots of improvements lately in Intercom, and all of them have been receiving very positive feedback. There are different ways to improve a product. You can add new features which captures existing non-consumption, or you can improve the ones you currently have.

Much to the delight of our customers, we recently improved our daily email. This is an email sent every morning at 7am in a customers local time zone reporting on what happened in their product the previous day. Let’s take a look at how we did that.

1. Understanding the job it does

Our customers already loved our daily email. In order to improve it, we had to work out why. As always, we started by talking with customers. The key questions focused on why they look at the email, how they read it (scan vs read), what they were looking for when scanning, and what they wanted to do next.

The key things learnt were the following:

• Customers open this email as part of a “starting work” routine. “Work” in this case is a mental state, not a physical place. The email was read long before customers arrive at work—usually in bed, while having a morning coffee, or on a bus or train to work.
• Customers use this email to get to know their interesting sign-ups. Many customers told us that without this email they would have never noticed that someone from TechCrunch/Foursquare/Microsoft had signed up.
• The next step when they saw a noteworthy customer signed up was to message them with a personal note in the hope of engaging them further.

2. Assess the current solution

Once we know exactly what customers are trying to achieve with our daily email, we then ask a simple question: “How well does this do the job it’s hired for?” Lots of problems started popping out:

• The first experience is mobile – Customers were looking at this email on their phone before they go to work. It looked okay but, simply put, we hadn’t designed for this so we had to fix it.
• No next step – We hadn’t included the next step here (messaging users), so customers were bouncing through meaningless screens simply to talk to someone that we said they should talk to.
• Not scalable – Eyeballing a list of 50 people to identify any stand out ones is certainly easy, but many of our customers have far more than 50 sign-ups a day. Finding the interesting people in a list of 400 is far harder. Our email wasn’t designed for this.
• Incomplete – It’s naive to only highlight sign-ups in this report. Two related questions to complete the view are “Who have we recently lost”, and “Who has stuck around”.

3. Address the issues identified

Focusing on these 4 areas let us improve the feature precisely based on the success criteria our customers had. The new design, shown below, is the result of just a couple of iterations with customer feedback.

The key changes are that it looks great on mobile, it incorporates the next steps for messaging, it pulls out the top 3 sign-ups you should focus on, the remaining sign-ups are ranked by how “noteworthy” they are based on a combination of their domain’s Alexa rank and their social status, and finally we’ve included 2 new sections (not shown here) showing you user anniversaries (active users who signed up a year ago today) and recently lost users (inactive for 30 days).

Deliberate improvements

Well researched improvements on already popular features get great feedback. I use the phrase Deliberate Improvement to describe this type of product work as opposed to more speculative or exploratory work where you’re trying ideas to see what gets traction. With a deliberate improvement you pick a popular feature, find out what job customers use it for, and then focus you improvements on that exact job.

The foundation of deliberate improvements has to be a real understanding as to why and how the feature is used. Otherwise, you risk either taking a step backwards on a key feature, or adding complexity that no one appreciates.