Product Judgment: How some people can repeatedly create product success
Main illustration: Kasia Bojanowska
Let’s talk about that trickiest of subjects: Product Judgment.
Also known as Product Intuition or Product Instinct or Product Taste, it is the idea that you can use your own judgment to (1) accurately predict what your customers need, want and value, and (2) design and ship the right solution for them. Here, I will tackle some of the common questions around it:
- Does it exist? What exactly is it?
- Who has it? Who doesn’t?
- Who thinks they have it but don’t?
- Is it natural or learned? How do I get it?
Product Judgment is a tricky subject because any time someone working in a Product role has their judgment questioned, the natural reaction can often be to take it personally, to infer that the person critiquing is challenging their ability to do a good job. How can you be good at Product if you don’t have strong Product Judgment?
I have experienced this many times on both sides, as the receiver and the critiquer. The truth is that Product Judgment is a complex topic, and in my opinion, one that is very poorly understood by many. I hope this post allows people and teams to safely talk about Product Judgment.
- If you have ever wondered why some people are extremely good at creating successful products, and others aren’t, this post will help you.
- If you ever had to face a Manager, Director or Exec as they make bad product decisions and you’re struggling to persuade them otherwise, this post will help you.
- If you think you have excellent Product Judgment, this post will definitely help you.
Before we dive in, we must first acknowledge that this isn’t a hard science. No matter what, there is always some subjectivity to Product Judgment. Two people with very strong Product Judgment can disagree. What I hope to achieve here is to remove as much of the subjectivity as possible, so we can all make better product decisions more often.
There is a lot to this, so let’s start to unpack it into discrete themes.
How to obtain product judgment
Product Judgment does exist, and it is learned
No one starts out with strong Product Judgment. It is not innate. It takes years to build, and therefore ranges from very weak to very strong. It takes deliberate work and dedication to build it. Therefore some people have it, whereas others don’t.
All great Product Managers and all great Designers have strong Product Judgment. I’m nearly 20 years into my career, and I’ve yet to meet a great PM or Designer without strong Product Judgment. They all have it because they understand how to get it, and were deliberate in doing so.
Here are examples of Product Judgment:
- Roadmapping: When faced with two competing feature ideas, the person with great product judgment can accurately predict which one will be better for customers. They simply don’t ship changes that people don’t value.
- Scoping: When faced with a scope call, whether a feature is in or out for a release, they can find the right balance between shipping value sooner by building less, but still solving the customer problem. They don’t waste any effort.
- Designing: When faced with different design solutions, they can pick the solution that customers will more easily understand and be able to adopt and use. They understand how all the features in a product are connected together, and they can predict with accuracy which workflows and paths their customers will take, and how design changes will impact those paths.
So if it is not innate, how do you get it?
Product Judgment is obtained through direct experiences with customers
That is it. It is that simple. The more direct experiences you have with real customers doing real things, the faster your product judgment grows. You need to observe and talk to real customers, who are really using your product. You therefore control how fast you build Product Judgment by deciding how often you personally observe and talk to your customers.
These experiences, whether it is observing someone use your product, or interviewing them about your product, need to be wide ranging. Full Product Judgment needs to take in your customers’ perceptions and behavior. How they use your product, what they think about your product, what they think about competing products, how they think about how your product is positioned and priced. These experiences need to cover both sides of product-market fit: the product, and the market. They also need to include all types of customers, those who are loyal, those who are new, those who have quit, those who are considering you.
Watching and talking to your customers must be direct, and must be wide ranging.
Growing your Product Judgment is a never-ending journey
The value of using your Product Judgment is to get the team to make faster, high quality decisions. Because you understand customers so well having spent so much time with them, you can emulate them when looking at design proposals and proposed product decisions. In the same way you can predict how your friends and family will behave, because you have spent so much time with them, you can predict how your customers will behave.
This ability to predict is about shooting for repeatable product success. But time does not stand still. To have repeatable success you must continually evolve your understanding. Because your product changes, your industry evolves, technology evolves, and your competitors get better, you must continue to have direct experiences with your customers. You must keep talking to them about your product, about what you have shipped, about real usage, and how it too is changing.
You can’t obtain Product Judgment through indirect means
You cannot obtain Product Judgment from reading summarized lists of feature requests. Nor from reading survey data. Nor from reading research reports from your research team.
Here is a common shortcut for busy people: your research team does the direct watching and talking to customers, and they play video clips of those customers using your product. This is not enough. Observing a recording is certainly better than reading reports or lists, and so it helps a little bit, but that is a very slow path to growing Product Judgment.
Another shortcut is to talk to the people in your company who do most of the talking to customers, notably the Sales and Support team. These people are directly talking to customers, and that’s good because they have built some judgment doing that, but you won’t build much personal judgment this way.
The indirect sources of customer insight above are all really important because they can help round out your Product Judgment, but they can’t create it.
Product Judgment is attained from direct experiences with customers, but there is more to it than that. It is domain specific. This is good when you can transfer that judgment across domains, but it is often the case that you cannot, and you need to get in front of those customers.
To understand how your customers talk about your product, you must use it yourself
To properly understand a customer when they talk about your product, you must first deeply understand it yourself. You must know how it works, and how it is supposed to work. Without this deep understanding you will get lost when your customer is getting into the weeds, or you will struggle to read between the lines and ask perceptive questions. The ability to ask perceptive questions is critical to running high signal interviews from which you can learn the most. You must know your product deeply to fully understand and interpret what customers say and do when using it.
It may sound obvious that you should use your own product to understand it, but many product designers don’t actually use the product themselves as they are not the target customer. They might play around a little in it, and think they are using it, but I am talking about real usage. Really using it for the real thing it was intended to be used for. Really understanding one’s own product only comes from this real usage.
There are ways to do this in any company. For example, at Intercom we build customer communication software. Part of Intercom is a product for Support teams. But our PMs, Designers, Engineers don’t work in Support. So we have a process internally where they periodically work with our Support team, for a day or half day at a time. They then experience real usage, which helps them become better at interviewing customers, and also rounds out their own Product Judgment.
Product Judgment and domain knowledge
Product Judgment is domain specific, it is not universal
Because Product Judgment is based on direct experience, you can’t have strong judgment in areas you haven’t observed directly – no one has strong Product Judgment for all products. I’ve learned this the hard way. Early on in my Intercom career, I made some bad product and design decisions.
Before Intercom, which is a customer communications product for businesses (in other words it’s a B2B product), I spent most of my career working on consumer products like Gmail and YouTube. Having never worked on B2B products, I had zero product judgment in this domain. But as someone who had designed lots of communication software, and as someone who hadn’t yet figured out this Product Judgment thing, I simply assumed I had the judgment to make the right decisions.
I incorrectly assumed that if I had designed products for people communicating with their family and friends, I could design products for people communicating with businesses. I was wrong, and I made mistakes. I had to go and learn Product Judgment for B2B communication products by observing and talking to Intercom customers. There was no other way. There is no other way.
Some domain knowledge is transferable, but you must know your limits
In my early days at Intercom I was missing the B2B piece, and learned the hard way, but I wasn’t starting from scratch. Having worked on social software for much of my career, I had strong Product Judgment in that domain, and could apply it to Intercom. Things like the psychology behind human communication, people’s behavior patterns with different communication channels, and what easy-to-use communication workflows and UI look like.
Knowing the limits of the Product Judgment you have built in the past is critical for knowing how to fast track the new judgment you need to build. This is really important when you change teams to a new domain or change companies. Unless moving to a direct competitor (and honestly even then), you need to get talking to customers asap.
Different parts of building a product are also different domains, you must go wide
It is obvious that different industries are different domains, for example the difference between a communication product for family and friends to use, and one for businesses and customers to use. Different parts of building software are also different domains.
You can talk to customers about how they perceive value in your product, how they think about your competitors, and that conversation will improve your Product Judgment on making roadmap decisions. It won’t help you at all in designing workflows and UI.
You can observe and talk to customers as they engage in using your product and its workflows, deeply interrogating UI decisions you’ve made, and that will help you build Product Judgment in knowing how to design an easy-to-use product. It won’t help you make roadmap calls.
Building a great product crosses many domains. From what problems the product solves, to how it solves them, to how it is priced and positioned in the market.
To build fully rounded Product Judgment, you need to cover the wide range. This is why when I run customer interviews, I always cover a lot of ground. When working at Google, I ran many hundreds of tactical research interviews on the usability of proposed new social products. But I always covered much more ground than that. I always covered desirability and perceptions, and always spent time understanding what other social products they used. I added time to interviews to ensure we were getting the full picture, to ensure we were maximizing the growth of our Product Judgment.
So there are two critical skills here:
- Knowing the limitations of when to move Product Judgment from one domain to another.
- Knowing how the domain specificity of Product Judgment sets your own limitations.
A final thing to understand about Product Judgment and domain is that it is also a moving target.
Understanding where and when Product Judgment needs to evolve, versus simply be transferred, is very important
With experienced leaders and practitioners, there is a danger of giving yesterday’s advice today. To avoid dogma, you must continually ask: what has changed? Things fall into three different buckets.
First, there are many things that never change, and they are rooted in human psychology. Things like our cognitive biases, how we interpret different shapes and forms. You can learn these in many good books (and you should), but to truly internalize them you need to see them in action, you need to see people applying them unconsciously when interacting with your product. But when you understand them, you can easily spot them in any product.
Second, there are things that change slowly. Internet software design patterns is one such area. Like all new technologies, where the early years see wild divergence in design patterns and then extreme convergence, the fundamentals of how internet software is designed hasn’t changed much since that convergence happened about 10 years ago. So this domain knowledge is very transferable. If you have observed hundreds and hundreds of people using internet software, you see the same design patterns work and fail time and again, and so can predict with great accuracy the fundamentals of where a new proposed design will be confusing to use or hard to understand. You can use this judgment to easily shortcut the design process. This is an important skill, and one that all the great design leaders have deliberately acquired. Assuming they are close to the target customer, they can look at a product and dissect what makes it work or not work.
Yet whilst the fundamentals of internet software design remain the same, much is still evolving, albeit slowly. We have new versions of operating systems every year, with new types of devices changing how people use technology. It is critical to avoid dogma.
Finally there are things that change fast. Things that quickly and fundamentally change how people think. Brand new types of products, new innovation from competitors, macro socio-economic changes that alter our perceived reality in the blink of an eye.
Sometimes it is easy to conflate best practices with judgment
Some things don’t need judgment. Judgment comes from experience but for many things there is a right and wrong way to do it. Some of these things are in the first category above – things that don’t change, but there are also things that evolve and what is right and wrong evolves also. Often these evolve from human behavior patterns that set over time, and the consequent solutions people expect. For example, you steer a car with a big giant wheel. The accelerator is to the right hand side of the brake. Same in software.
There is a right and wrong time to use checkboxes and radio buttons. There is a right and wrong way to design buttons and links. There are right and wrong ways to use variables in font like size, colour, contrast, etc. This idea is one of the benefits of a formal education in designing and building software. From Design to Engineering, there are common best practices that don’t require judgment – they are objectively better or worse – and education teaches them to you. Of course you can also teach yourself, by reading and practicing, and many great PMs, Designers and Engineers are self-taught.
Domain specificity is the reason Researchers don’t always automatically have the strongest Product Judgment
One could conclude from what they have read here thus far that Researchers, as the people talking directly to customers most often, must have the strongest Product Judgment. This can be true, and I’ve seen it firsthand. When it is true it is an incredible asset for a team. As an ex Researcher, I always encourage Researchers to understand their personal domain limitations, and broaden their thinking to become the best version of themselves.
Unfortunately, this breadth of domain knowledge in Researchers is often not the case. Despite being the people talking to customers most often, many Researchers don’t have strong Product Judgment in other domains, and so find it hard to translate direct customer observations into product decisions. It is one thing to talk to customers and build judgment, it is another to apply it.
Researchers often stop short of applying it. They hand over the insight, they don’t own the impact of the insight. Perhaps this is for good reason – maybe nobody asked them to help design the product, or invited them to actively participate in a scoping workshop. But this is wrong in my opinion – they should be invited, and if not they should invite themselves and prove their value.
This is why I encourage people and teams to blur their roles and responsibilities, so they can learn how to apply what they are learning. For the same reasons I want PMs and Designers talking directly to customers (and not outsourcing it to Researchers), I want Researchers sketching solutions, talking to Engineers about how to build them, and building fully rounded Product Judgment by applying what they have observed.
The roles people play
People with strong Product Judgment are very good at saying when they don’t have that judgment, because they deeply understand how it is accumulated
They can look at a proposal and say “This is not something I have built judgment for because it is not something I have directly talked to customers about”. They know their own limits. They typically defer to the people in the room who have built that judgment, or they recommend that we get talking to customers asap to start to build that judgment.
As I have grown in my career, from working as a Designer, Researcher, and PM to running a full Product org, I find myself in this position more and more. As time goes on, I move into higher levels of strategy, and like almost all other leaders and execs, I talk to customers less and less. This is not good, and something I work on improving periodically. But leaders and execs are desperately time poor, and so while one should never get too far from your customers, it is important to defer to those who do have strong judgment in a domain because – and only because – they have talked to customers.
An experienced leader with deep Product Judgment can often look at something and know what to do, but find it hard to explain
They just know this is the right option, and they are often correct. This is because their brain is using lessons learned from years of direct customer interaction, synthesizing it all subconsciously, and shortcutting to the recommendation. As has been popularized by Thinking, Fast and Slow, this is how our brain works.
This is the most challenging of situations, as less experienced staff may not understand where this judgment is coming from, and simply see it as uninformed dogma. It is also unfortunately true that leaders with poor Product Judgment exhibit this exact same behavior, but from a place of false confidence and ego, rather than lessons learned from customer interaction.
This is tricky to disambiguate, but you can usually tell the difference by the leaders’ interest in talking to customers, past experience with talking to customers, and insistence that everyone talks to customers. You can often identify strong/weak Product Judgment by using interest in talking to customers as a proxy.
Many people think they have strong Product Judgment but don’t
The opposite of an experienced leader who knows their own limits is someone who doesn’t understand any of this, and believes they have strong Product Judgment because it is some kind of innate skill. They see other leaders do it well, and imitate them, creating a cargo cult. This is a very common problem in tech, and the cause of many failed startups.
This ranges from naive junior employees with a false sense of confidence (who have yet to be humbled by seeing their product fail) to senior leaders with an even greater false sense of confidence (who attribute their product successes to their own judgment rather than actual causes such as timing, luck or the judgment of others in their teams).
I witnessed the latter many times in my career. And many times as a Researcher/Designer/PM I tried to use indirect means, such as research reports, or video clips, to persuade people their judgment was poor and that the product would fail. I now realize the only fast way to have persuaded those people was to get them observing customers firsthand.
Conclusion: How to build strong product judgment
In summary
- Product Judgment isn’t an obscure concept. It exists, it is verifiable, and it is learned through direct experiences with customers.
- You can increase the speed at which you build Product Judgment by talking to, and observing, more and more customers.
- Building a great product crosses many domains, and Product Judgment is domain specific. The domain might be the industry the product is in, or the type of problem it solves. Or the domain might be the different areas that make up a full product, from how it is designed, to how it is priced and positioned in the market.
- Product Judgment is transferable across similar domains, but not across different domains. To build fully rounded Product Judgment, you need to cover the wide range by directing talking to customers about all the domains.
Finally, you may wonder how fast you can actually build strong product judgment. For people reading who want strong Product Judgment, but do not talk to customers, because they didn’t realize the importance, or feel too shy, or think it a pain to set it up, or think it someone else’s job, you may be asking what it takes. How many customers do I need to talk to before I build strong judgment?
On the one hand, this is the wrong question. It is a never-ending journey. To continue to have strong Product Judgment you must never stop. Things change. Technology evolves. Industries change. Competitors get better. You must never stop your direct experiences with your customers. You must keep talking to them about your product, about what you have shipped, about real usage, and how it too is changing.
You must keep checking your Product Judgment and refining it. When you apply it, and decisions get made, you must have the desire to go back and check those decisions. Were they correct i.e. was your judgment correct? If you got it wrong, learn why it was wrong and how you might improve it.
On the other hand there are some answers. To build strong Product Judgment in a specific domain, the answer is at least over 100 customers. For true proficiency, built over years, you’re looking at talking directly to many hundreds if not over 1,000 customers. This may seem daunting but if you build it into your working life they add up fast. Here are some examples.
Early in my career I worked for 2.5 years in a UX consultancy called Flow. We did rapid research and design projects. I worked on things like redesigning The Guardian website, BBC websites, transport and tourism websites, many financial products. We turned things around fast. In these projects we often did 5/10/20 person evaluative research studies. During my time at Flow, I estimate that I personally interviewed and observed more than 500 people using internet software. An average of ~3 people a week, but usually clustered into 5-6 people in one day. Sometimes we did 15-20 interviews in one week for the same project. It was tiring but worth it.
So what did I get from that? A lot of Product Judgment around how people use software, how they interpret UI, and how that overlaps with human cognitive biases. This is a really valuable foundation for anyone building software products. That helped me build intuition for software that is easy to use, and enabled me to easily spot software or design proposals that would be hard to use across most industries.
I then worked at Google for 4 years, and during that time interviewed around 800 people using Google products, using competitor products, or interacting with design proposals and prototypes. At Google I would often cluster interviews and as at Flow, go hard. So I would do 5-6 people each day for 3-4 days in a row. This was one way to dramatically fast track Product Judgment on a specific area. Again it was tiring but worth it.
At both Flow and Google I used in person interviews. I would often meet people in their own environment, their home, school, workplace. Someone’s own environment is the richest place to learn. It is also the most time consuming, and so I also used two other options. Having them come to our office meant I could stack 5-6 interviews into one day, or doing the interview over video enabled the same. All three of these options are good.
After Google I worked at Facebook for 2.5 years and have spent the last 7 years at Intercom. Thanks to Flow and Google, and shared domains, I was fast at building Product Judgment at Facebook and Intercom. In the early years at Intercom I talked to many of our customers to understand the domains that were new to me and got to a place where I felt my judgment was strong.
You can do this too. You simply need to start and the numbers add up fast. Most satisfying, you will start to feel your judgment grow. You get really good at pattern matching, you see things you have seen before, you can start to accurately predict customer behavior. It is a very rewarding endeavor.
To build strong Product Judgment, you need to fall in love with talking to customers. Get past the inhibitions you have today. I’ve seen many people do this over the years, and fast track their ability and career.
