🧠 Community Wisdom: From take-homes to vibe-coding product interviews, most impactful tactics to validate a start…
👋 Hello and welcome to this week’s edition of ✨ Community Wisdom ✨ a subscriber-only email, delivered every Saturday, highlighting the most helpful conversations in our members-only Slack community.
A big thank-you to this month’s community sponsor, Merge. Merge makes integrating with your customers’ software vendors easy. Add hundreds of integrations to your B2B SaaS products, with unified APIs for file storage, ticketing, CRM, ATS, HRIS, and accounting. AI companies love Merge because it delivers instant, secure access to clean customer data that powers their solutions. Stop wasting countless engineering hours staying ahead of breaking API changes and digging through a backlog of integration support tickets. Merge handles the full integration lifecycle so you can focus on what matters most: building your core product! ✨ Upcoming community meetups ✨Upcoming community-organized meetups—click the city name to RSVP:
Can’t find your city and want to host one? Just DM @Riya in our Slack. It takes 10 minutes (i.e. pick a date and location), and you get to meet awesome people from our community. Learn more here. 🎙️ New podcast episodes this weekAnthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): YouTube // Apple // Spotify  Why Uber’s CPO delivers food on weekends | Sachin Kansal: YouTube // Apple // Spotify  The exact AI playbook (using MCPs, custom GPTs, Granola) that saved ElevenLabs $100K+ and helps them ship daily | Luke Harries (Head of Growth): YouTube // Spotify // Apple  💥 Top threads this week1. From take-homes to vibe-coding product interviews
Adam Fishman: A few thoughts on this and also based on discussion with other senior people hiring product managers: I don’t mind take-home exercises. It’s a buyer’s market right now for companies, and we should be encouraging the use of ChatGPT. How they respond to follow-ups on the assignment and show original thinking (where ChatGPT will fall short due to lack of context) is most important here. I think vibe coding or vibe analysis is a perfectly acceptable alternative—I don’t know if enough PMs know how to do it, but the kind you want to hire probably do. Suraj Pabba: From a candidate’s perspective, I personally hate take-home exercises mainly because working full-time and recruiting can be so time-consuming that you never get to show your true creative self. I love the vibe coding challenge, though. I think it’s the perfect opportunity to show creativity and being a builder. I just read an article today about how PMs need to shift to being a product creator instead, and interviewing might be the best opportunity to showcase that. Andrew: I think the idea of doing it as a vibe coding exercise tells you one skill but really depends on what you are looking for. They might have made the coolest prototype, but do they know why they made that prototype, and why did they make that decision over this, can they research a problem, can they speak to users or can they just prompt a tool? Matt Gonzales: In a past role, we used to do a live exercise. Format TLDR: 45-minute interview to talk through the exercise (two parts: strategy, tactical with pre-set questions we shared 24 hours before, so no studying or memorizing required. Just familiarity). No more than 2 hours of the candidate summarizing what was discussed (no new research needed), and usually this was much faster. One round of async feedback (we comment on their write-ups and ask questions, they respond once), fin. The 45-minute live discussion, at least in our view, helped us get the same outcome from an exercise but in a more collaborative, less stressful way. Candidate and interview both picked 2 topics, decided on one during the call. Interviewer took notes and facilitated the discussion while screen sharing, so it was truly collaborative. Topics were never relevant to the business, so it was neutral ground. We were able to see how the candidate thought strategically and tactically, saw how they dealt with collaboration and challenging of their ideas (both live and async), saw how they communicated live and async, etc. As someone who’s been a candidate myself, I really dislike the vague, open-ended, and “here you go, swim” formats. I have a strong bias (and love) for this live + async format I’ve used in the past because, at least in my view, it’s more enjoyable for both interviewer and candidate. Shane J: One concern about a vibe coding exercise is that the AI can be so hit-or-miss that, for a single sample per candidate, you will probably end up with a lot more noise than signal (lucky vs. skilled). Though if you could split it into something like 4-5 mini vibe exercises, I feel like the signal will start to show through? (Not sure what 10-minute vibe tests would look like, though—but I would probably switch test-interview providers if someone has figured this out at scale.) Miroslav Pavelek: I quite often asked if I could use ChatGPT for the take-home exercise, and usually they have been fine with that. I liked them because I treated them like the product discovery—what can I learn about the company by doing this? Do I actually want a job where I would do things like these? Adam White: I love this idea, Mike. I’d prefer it to be done synchronously, so we could go back and forth together. Mike Halpert: @Adam White, agreed. I was thinking the candidate has an hour to do research, put their thoughts together, form a vision and then start the prototype. Then in the presentation, it would be more of an active collaboration. Adam White: I would welcome this style with open arms. Much higher signal for both parties than “Tell me about a time when you overcame a challenge.” Cathy Tyrell-Knights: As a candidate, I would welcome the opportunity to do a vibe coding exercise instead of take-homes that take heaps of time and effort for minimal returns. I’m not at the level yet that I can create a rapid prototype in tools (working on it!), but I can easily create something using pen and paper, virtual whiteboard, or even just talking it through with the interviewer. I have done it often when I needed to illustrate a point to our designers and prompt discussion. 2. Narrowing focus vs. expanding breadth in your product strategy
Eric Metelka: Do you have qualitative and quantitative data that can support your arguments? Your users are your best reference. Tim Balzer: I think the customer/user angle is already well supported in our internal data and feedback. I’m hoping to complement that with a broader industry perspective, ideally some examples or citable patterns with recognizable brands that reinforce the case for focus. Eric Metelka: Sounds like you’re looking for case studies. You could share more details about what you want to support to see if we have good references. Tim Balzer: Yes, case studies are exactly what I’m after! I’m building a case for narrowing our product focus to go deeper on our core offering. I’m looking for examples of companies (ideally SaaS) that succeeded by focusing deeply before expanding, or that struggled because they spread too thin too early. Anything that supports the value of depth over breadth would be super-helpful. Anecdotally, I recall hearing Shopify, Figma, Atlassian, Amazon, etc. adopting the deep focus until you get it right before expanding strategy, but I’m having trouble finding specific articles, podcasts, etc. that are good reference points for my audience. James Conway: How about vertical SaaS? Something like Toast or Veeva. Joshua Herzig-Marx: At some point, though, all companies expand. Toast is now building consumer apps, making loans, etc. I don’t have any strong, citable sources. I wonder if Lennybot can help. @Lennybot! My colleague is looking for citable case examples of where companies “narrowed” their product strategy and, instead of pursuing a broad strategy, went back to their core competencies. Is this something that one of the guests on the podcast talked about? Is there an article about this? Lennybot: @Joshua Herzig-Marx Here are some strong, citable examples of companies that narrowed their product strategy and focused deeply on their core competencies before expanding: 1. Dropbox: Dropbox is a textbook example of narrowing focus. At one point, they ventured into products like Carousel (a photo-sharing app) and Mailbox (an email client). However, Drew Houston, Dropbox’s CEO, made the tough decision to kill these products and double down on their core productivity tools. This decision was driven by the realization that 80% of their paying subscribers were using Dropbox for work, not for consumer-focused features like photo sharing. While the decision initially led to negative press and internal challenges, it ultimately allowed Dropbox to strengthen its core offering and position itself as a leader in cloud storage and productivity [7][8]. 2. Airbnb: Airbnb also provides a great example. They experimented with expanding into areas like transportation and experiences but realized that their core strength lay in providing unique, home-based accommodations. By refocusing on their core offering—homes—they were able to solidify their position as a market leader. This decision was critical during the pandemic when they had to streamline operations and focus on what made them special: authentic, home-based stays [3][6]. 3. LinkedIn: LinkedIn went through a phase where their product portfolio became cluttered with too many features and initiatives. They eventually rationalized their offerings, focusing on their core value propositions like professional networking and job search. This focus helped them optimize their product portfolio and improve the user experience [6]. 4. Toast: Toast started as a vertical SaaS company focused on restaurant point-of-sale systems. They stayed deeply focused on this niche, building a robust product tailored to the unique needs of restaurants before expanding into areas like consumer apps and financial services. Their initial focus allowed them to dominate the restaurant tech space before broadening their scope. These examples highlight the value of depth over breadth, especially in the early stages of a company’s growth. Narrowing focus often allows companies to build a stronger foundation, better serve their core customers, and achieve sustainable growth before expanding into adjacent markets. If you’re looking for more detailed references or articles, I’d recommend diving into Drew Houston’s interviews (like the one on my podcast) for Dropbox’s story, or Airbnb’s public communications during the pandemic for their strategic shift. Let me know if you’d like me to dig deeper into any of these! Joshua Herzig-Marx: Well, that was really nice! John Koht: Funny enough, 37Signals is a great example of this. They used to have four products. They killed all to focus on Basecamp. It’s worked well for them. 3. Most impactful tactics to validate a startup idea
Michael Thomas: Run 5-7 problem discovery interviews: Focus only on past behavior—ask about the last time they faced [problem], what they did, what it cost them, and how they decided what to do next. Don’t pitch. Just listen.
Joshua Herzig-Marx: Before answering the question, I’d ask:
In general, I think testing demand is most important for consumer startups, and testing your ability to convert early customers is most important for B2B startups, but I think it can vary a ton beyond that. But yeah—go talk to people. Have you read The Mom Test? Anuj Adhiya: “Learn before you build” is the mantra. Pretotyping is one great way to do that: pretotyping.org Joshua Herzig-Marx: I love The Right It, but I don’t think it’s the motion that every successful founder followed (even if I think it’s the right one for most founders). I’ve been thinking about this chart for a few weeks. Anuj Adhiya: I like it primarily for its thrust of doing extremely cheap experiments with quick learning loops focused on learning the right thing at the right time. Michael Thomas: @Joshua Herzig-Marx, do you think some successful founders may have gotten there through sheer hustle or brute force? Joshua Herzig-Marx: No—I don’t think hustle or brute force ever gets you there. I think they may have had some niche insight, or they were building off an observation and deep market understanding from a past role. Or they’d developed and then monetized a community or a public voice. Or to The Right It’s point, they validated the market and demand and their own ability to convert within it. Michael Thomas: I agree with that: sometimes that insight came from experience, sometimes it was discovered through direct observation. Theo: Thanks for the responses! Would love more specific examples and anecdotes. From my own experience—at one startup I was at, we would just cold call our early customers. Not advisable for most companies, but it was extremely effective! Ashley Rolfmore: A mental framing I like is that you are converting:
If you have a path for both, and cash flow to cover that journey, you’re golden. 1 is for you—when asking this question, typically you have some insight, some observation, even some technology, with commercial viability. First you check with who would buy/use the thing—that’s covered above by previous posters. But also, you need to check your competitors, and that’s not just competitors in the explicit vertical doing a very specific thing, that’s anything your potential buyer/user can do instead. You also need to work out buying motivations—there’s often little benefit in being the best if your buyer buys based on reputation and tradition. 2 is very much in the innovation space, and if you’re addressing a need that your potential customers don’t realise they have yet, you have to do that work to make that need more obvious to them somehow. There’s been some failed products which if launched, say, 5 or 10 years later, likely would have done better (for example, a lot of what Google Wave did can now be done in the Apple Notes app (!)). Re: your request for anecdotes:
4. Should growth PMs be the highest-paid product people?
Greg Docter: Interesting question, never thought about it this way. I think the answer is kinda “yes”—I’m thinking about enterprise sales and how much those folks get paid. Miguel Hernandez: No, at least in my experience. Engineering and product tend to make more money than marketing. Sales, on the other hand, can rival engineering and product. Miroslav Pavelek: I believe they are similar. For each good enterprise sales, there is also some niche engineer getting paid similarly huuuge amounts. Miles: I think Nikita is talking about a future state, so I wouldn’t expect it to reflect compensation today. I think if there’s provable value, money would flow there. Joshua Herzig-Marx: Economic theory says your employees are paid at the marginal incremental value they deliver, discounted by your ease in filling the role. So in theory, as product building becomes easier, you’ll expect to see fewer people in the “tech” half of the company, but they’ll continue to be well compensated. Each engineer will be able to deliver more marginal value (because they’ll be coding more efficiently), but there may also be more competition for that role (since there are fewer engineering jobs overall), but you might also really want to hold on to the engineers you have because you haven’t yet invested in onboarding. I don’t know where the math ends up, but probably you’re not seeing lower wages. When a “new” kind of product management emerges (e.g. growth a few years ago, AI today), those PMs tend to command a bit higher salaries because it’s harder to find them. Why do sales people and growth leaders (especially in GM roles) tend to command even higher salaries? Because you’re looking at target comp numbers, a greater percentage of their salary is performance-based, so if they don’t excel they may end up making substantially less. Ashwin: Marketers are actually paid well below engineers, and in some cases they are on par with designers, but in other cases they are not. In fact, between the major functions of marketing, sales, customer success, engineering, and product, they are probably on the lowest tier. Some growth product managers can make more money than other product management roles if the industry they’re in really needs them, like advertising, for instance. Michael Thomas: This connects to a pattern I see everywhere: pay is based on what we can measure easily (technical skills, traditional scarcity), rather than what we should measure (systematic momentum building). Same dynamic as so many project prioritizations—easy to prioritize delivery metrics over adoption and value. The AI shift forces the question: do we even know how to identify the people who systematically create compound value vs. those who just execute well in measurable ways? Daniel Salians: Influencers. Distribution brings 80% of revenue. Building product is 80% harder. Marc Dupuis: On a per-hour basis, I think good marketers are starting to easily surpass devs, especially considering that more and more companies are going overseas for dev talent. Personally, I’m not comparing apples to apples, since most of our marketing talent is agencies, which will always cost more, but it’s becoming pretty obvious to me that the value is hardly just in the product anymore; distribution is where the magic really happens. Harrison Telyan: Depends on stage, market, and leverage. Early stage? Engineers usually top the charts. Growth stage or post-PMF? A great marketer who can move the needle on revenue often earns just as much as—if not more than—engineers or designers. That said, designers with strong product sense and storytelling chops are increasingly closing that gap—especially in roles that blend brand, product, and growth. (It’s one of the reasons we built NUMI—to help designers flex beyond the UI and drive real business impact.) So yeah, comp follows leverage. Be the one who drives outcomes, not just output. 🤓 Top finds
😂 Meme of the week
Have a fulfilling and productive week 🙏 If anyone in your life would benefit from this newsletter or community, consider giving them a gift subscription 💞 There are group discounts, gift options, and referral bonuses available. Sincerely, Kiyani 👋 Invite your friends and earn rewardsIf you enjoy Lenny's Newsletter, share it with your friends and earn rewards when they subscribe.
|
Similar newsletters
There are other similar shared emails that you might be interested in:
- 🧠 Community Wisdom: Essential reading for early-stage founders, balancing unlimited SaaS plans with profitability…
- 🧠 Community Wisdom: Finding your voice on established teams, focus tips for ADHD brains, how to best use Granola,…
- 🧠 Community Wisdom: Designing great performance review templates, balancing oversight and autonomy with junior te…