🧠 Community Wisdom: Why effort-based estimation is broken, connecting Bolt/Lovable to existing apps, removing per…
👋 Hello and welcome to this week’s edition of ✨ Community Wisdom ✨ a subscriber-only email, delivered every Saturday, highlighting the most helpful conversations in our members-only Slack community.
A big thank-you to this month’s community sponsor, Merge. Merge makes integrating with your customers’ software vendors easy. Add hundreds of integrations to your B2B SaaS products, with unified APIs for file storage, ticketing, CRM, ATS, HRIS, and accounting. AI companies love Merge because it delivers instant, secure access to clean customer data that powers their solutions. Stop wasting countless engineering hours staying ahead of breaking API changes and digging through a backlog of integration support tickets. Merge handles the full integration lifecycle so you can focus on what matters most: building your core product! ✨ Upcoming community meetups ✨Upcoming community-organized meetups—click the city name to RSVP:
Can’t find your city and want to host one? Just DM @Riya in our Slack. It takes 10 minutes (i.e. pick a date and location), and you get to meet awesome people from our community. Learn more here. 🎙️ New podcast episodes this weekGrowth tactics from OpenAI and Stripe’s first marketer | Krithika Shankarraman: YouTube // Apple // Spotify  A 3-step AI coding workflow for solo founders | Ryan Carson (5x founder): YouTube // Spotify // Apple  💥 Top threads this week1. Why effort-based estimation is broken
Daniel Bartholomae: Not really controversial from my perspective: Estimation has mostly been proven by studies to be a total waste, as the prediction value is really low. So far, I could confirm that at every company where I checked it: The estimated story points had almost no correlation with the time a task actually took. Only exception I’ve seen is clearly repeatable work, e.g. a dev shop that sets up WordPress pages for small companies and builds basically the same page over and over again. Magnus Blicker: I’ve more or less moved away from estimates entirely within my team. It does help that I’ve been working on the same product with the same team for a long time and I generally understand the codebase quite well, so most of the time I can infer quite well whether something is a 1-day, 1-week or 1-month thing, which is more than enough precision for me to prioritize. Jeff: Complexity and risk are still helpful, but story points and velocity aren’t. I’ve seen them become very gamified in some orgs, and they offer little to no value in my experience. Daniel Bartholomae: Good point: The discussion is helpful, and even story points can theoretically help (I’ve worked with teams that estimated but had a strict rule that none of the estimates ever get written down or communicated outside of the planning session). It is easy to get in a situation where someone without a clue about estimation then tries to use the numbers for something they are not usable for. Jeff: I should add, we use the complexity/risk for assumption management too. @Daniel Bartholomae, all the damn time: “It is easy to get in a situation where someone without a clue about estimation then tries to use the numbers for something they are not usable for.” Law: There are whole industry frameworks (e.g. SAFe) that feel set up around estimation and orchestration in an attempt to give certainty to people who have a need to know how well things are going, and companies that get paid via delivery of velocity. Eric Metelka: I have never found estimating individual stories/tickets valuable. I’ve witnessed scrum masters try to make it fun, but even in the best circumstances it’s not worth the hour of investment, because it’s always wrong. What I care about is that projects don’t balloon in scope. Whether that means a small project stays small or that a large project doesn’t become an extra-large. And it’s ok if there’s unanticipated complexity, as long as it’s communicated and the consequences agreed to. Not staying on top of this is where I have experienced teams going off track. Miroslav Pavelek: The more experienced I am, the more I avoid estimates. If you want to keep some kind of predictability, try something like Shape Up—sit with a team for a few days with the goal to see what you can achieve in up to 6 weeks. Set a goal, think about it, and try to do it. Don’t be afraid to cut (minor) scope along the way. Joshua Herzig-Marx: It’s worth remembering that the “E” in ICE/RICE is for effort, not estimation. But this is also a reason why it’s much, much, much easier to prioritize problems than solutions. Prioritize the problem, then work with the team to figure out the fastest way to test if you can solve the problem and if solving the problem is, in fact, valuable. Krishna AG: Genuine question: In case any of you are selling to enterprise, markets in which legacy businesses dominate (like banks, insurance, etc.), how do you manage not having timelines? There are times when there’s a multi-million deal on the line which can be signed if you promise something by sometime. Steve Bergen: Never found the estimates themselves useful (or correct). What I’ve had success doing in the past is bringing in a senior eng (or someone who’s very familiar with a codebase) to break down an entire solution into a feasibility evaluation with the team at a high level (i.e. “Can this work be completed in 2 weeks?”). Then the conversations that typically arise will highlight the riskiest elements, which we then prioritize first. I find that when this exercise happens, teams are more likely to feel comfortable committing to a deadline. However, this does render task-based velocity tracking useless. Eric Metelka: For customer commitments in B2B, I give a loose estimate, such as the quarter we plan to ship in. Also, keep in mind this doesn’t mean forgoing a bigger project plan of number of engineering weeks, which gives a rough sense of time. This becomes an internal date that I make sure everyone is aware of and committed to. I then always always pad that number externally and also give that loose range when needing to set expectations. Daniel Bartholomae: We’re selling mainly to enterprise, though only a few banks, and our customer is the marketing department, not IT. There we don’t really need roadmaps: Either we already have what the customer needs to make the deal or we don’t. If we do, no need for a roadmap. If we don’t, then we do the roadmap planning the other way around: When does the feature need to be finished so that the deal can close? Then the question becomes one of investment: Can we justify putting a full dev team on this topic for that amount of time just to close this customer? If yes, we then put devs and customers in close contact and let them develop the feature together in Agile fashion, so no need for a roadmap either. Jeff: @Krishna AG, my answer is going to really depend on how your organization sells (e.g. is the Sales roadmap selling and asking for that “one small feature” to get a deal done) and how your org thinks about Product Development in general. This is why Marty Cagan’s books focus on the organizational level of change. Most Product Managers and even leaders can’t effect a broad, sweeping change that is necessary. It’s not that you don’t have timelines, it’s the level of granularity and how you communicate them. What has worked for me is to use a roadmap that outlines the outcomes we’re driving (e.g. reduced effort and error rates in Feature Area X for a specific persona). I use a now/next/later roadmap, and I say things like “We expect to deliver the first version of this in Q2” without committing to specific features. Customers (and your customer-facing teams) will press you for details. This never goes away, even in a company where I successfully eliminated roadmaps entirely. As you approach Q2 (in this case), I start to become more specific about what we’re delivering, as our confidence increases. Note that I’ve done this in a startup where our customers (banks) couldn’t take releases at the velocity we were shipping, so we bundled them into quarterly releases for them, allowing their change management teams to digest the changes. I’ve also done this in a more legacy enterprise where customers had previously been trained to buy from us via roadmap selling. In some cases, our roadmap was driving internal transformation (e.g. stack consolidation or new tool rollout/integration), so there was a legitimate and necessary reason for date coordination. It’s a delicate act, and to be brutally honest, it may not be worth staking your career on it to fight this fight. Once a company and its customers become addicted to roadmaps, it’s really hard to break that habit. Again, how your organization thinks about Product really dictates your success rates here. Ashwin: Effort estimations are very valid and I’m going through this right now. For example, we are transitioning to a whole micro-services-based front-end architecture, and a lack of estimations from some of the developers does not allow me enough visibility or line of sight into putting together the right roadmap. Daniel Bartholomae: @Ashwin, have you run a statistical analysis on the correlation between the estimates you get and the time tickets really take? If not, I would really recommend it, as it has been eye-opening for me. Otherwise, you just trust estimates without knowing whether they turned out even remotely realistic. Ashley Rolfmore: Effort estimations:
There is one final point—estimations are a multifaceted tool, and the one need that can get lost is the “oh, we need to work out what doing a thing might cost”. Estimates are a proxy for forecasting cost. If you don’t have a good answer on how you allocate and predict spend as a company, it can manifest in heavy estimate bureaucracy. 2. Connecting Bolt/Lovable to existing apps
Jordan Clark: Alternatively, you could use a tool like UX Pilot or Magic Patterns to prototype a feature to pass off to design and engineering. UX Pilot allows you to send to Figma too, so it would be a good handoff or the ability to make a high-fidelity prototype in Figma. I can’t recall if Magic Patterns has that capability. Teddy: I went at this from the other direction with a simple website last week. I had Lovable create everything, and then used their GitHub repo sync capability to be able to make my own edits. It worked really well for me—I imagine there’s a way to connect it to a repo that already has an existing app. Here’s a writeup of my experience, though I think all you really would care about is exploring the GitHub repo sync in more detail. Michelle: @Adriel Lubarsky, is it that you want like a prototype that looks very much like your current project, and then you want to play and add new features on? Adriel Lubarsky: @Jordan Clark, when would you use UX Pilot vs. bolt vs. the other ones? What does UX Pilot do that the others don’t? @Michelle, yep! That’s right. I want to design new features that are based on something as close to to our real app as possible. That way, they make sense in the flow, and its easier to pass off to eng. Michelle: Oh, there’s a much easier way! Adriel Lubarsky: Do tell... Michelle: When I first started vibe coding, as practice I’d find an easy product and replicate it. As in take screenshots, get Claude.ai to describe the screenshot, then drop both into v0.dev. For your initial prompt to create the app:
Then you just build up step-by-step like that. Adriel Lubarsky: Ahhh, that’s clever as a start! That makes sense, I’m going to try that. Jordan Clark: UX Pilot won’t build you a functional product, it’s simply a design tool. So if you want a visual prototype quick, it can build out your page flows. If you want a functioning product someone could use, then use Bolt, etc. Blake Moody: Just yesterday after a fun conversation with an LLM to create a product brief/requirements doc, I then asked it to give me an output that I could paste into Figma Make, so that I could create an interactive prototype based off these ideas. Blown away. I pasted an image during the conversation of our app including the navigation bar on the left and the identity and top nav... It’s close enough to clearly get the idea. Priya Mathew Badger: Magic Patterns can do the Figma export too. I wrote a bit about how I’ve been using it/why here. 3. Removing bias from product decisions
Mayank: Ideally you should evaluate the problem, not the feature, like what % of your users (or the ones you reached out to) have this problem. Features inherently have confirmation bias, and they evolve as you understand the problem more by talking to users. After it goes live, talk to users if the problem is solved for them. Finally, track your key metrics that you wanted to improve before and after the release. Shatha: Then maybe allow me to rephrase the question: How can you neutrally evaluate the problem without confirmation bias that you start working on proving it is the problem that we need to solve now? Samuel Barney (sambarney.com): Well, starting by asking the who, what, where, and why around the problem space.
Samuel Barney: Start quantifying the problem space, that’s pretty safe/neutral for unbiased evaluation. Andrew: Another good way of thinking before the presentation is the classic five why’s method: if something is important, why, because xxx, why, and just keep “why” until you get to the root and most fundamental reason, then build your presentation back up. If you can’t articulate why it is valuable or important past the first or second why, then chances are you need more information. Viet Dao: To add to @Andrew’s great callout to the 5 why’s, desirability is important but not the only consideration. What will probably matter most to leadership will be around the viability points that @Mayank and @Samuel Barney listed around the size of the opportunity for your business and alignment to your company goals. Always cater to your audience—if it includes engineering leaders, make sure you mention feasibility, and if it includes design leaders, then callouts to usability/desirability, etc. Myles Sutholt: All the above. Also, one thing I like to do is think about what ways the feature could be proven to be a failure. Like: If I was really eager to, how could I prove that this was a bad idea? This has two benefits.
4. Best analytics tools for under 1M events
Eric Metelka: I don’t think this differs much based on B2B vs. B2C. It’s still Amplitude, Mixpanel, PostHog for product analytics. Ashley Rolfmore: I liked Amplitude for funnels, and Mixpanel for investigating individual user paths (when you find someone falls off a funnel, where did they go instead?). We use plausible.io, right now—simple, but I don’t recommend it for actual product work. Kavir: @Eric Metelka, thanks! Any recommendation between the three you mentioned? I’m leaning towards PostHog because it also has other features and a good roadmap. @Ashley Rolfmore, between the two—Mixpanel and Amplitude—would you have any preference? I’ve used both as well. But would love an opinion. Andrew Draper: FWIW, in the last 3 years I’ve been a paid user of Amplitude, Heap, Plausible and PostHog. I love Plausible on my personal site, but, like @Ashley Rolfmore said, it’s terrible for anything else. Heap was my favorite for years but was the most expensive and after their acquisition seemed to get a little bloated. Amplitude was my least favorite, the UI seemed dated and clunky and it felt like work to use. PostHog is what I’m using now, there’s a definite learning curve compared to the others, but it’s really good once you get the hang of it—plus for less than a million events it’s free/dirt cheap. Both Heap and PostHog also automatically track events, so if you want to check on something it’s a matter of creating the action and you immediately have insight. Mixpanel, Amplitude, etc. you need to create the event and wait for data. Eric Metelka: My personal preference is Amplitude, but I haven’t been hands-on with PostHog. Justin Goff: Might be overkill at your size (in which case there are plenty of good options listed above), but I wouldn’t rule out a warehouse-native approach with something like RudderStack + (data warehouse) + Mitzu. Up-front costs tend to be higher, but you get a lot more flexibility in terms of how you model your data, which can be huge when it comes to custom metrics that draw on a range of data sources (like event tracking, CRM, email marketing, etc.). Also, hot take: I’d avoid anything that’s client-side-only (which rules out most auto-capture). That’s a whole different rabbit hole to go down, though. Eric Metelka: Having led product at a data-warehouse-native product for the last 2.5 years, I don’t think data-warehouse-native works for product analytics. A number of failed startups in the space and poor UX. For product analytics, you want something that works really fast to get answers, which is why events-driven is better. alexa: PostHog! Andrew Draper: This has been my experience. I had access to Snowflake, Qlik, etc., and as good as they are, for product analytics they’re definitely not my preference (great for reporting to the rest of the exec team, the board, etc., but for product decisions not so much). Justin Goff: I’m not sure I agree—there are some good product-focused tools emerging in the warehouse-native analytics space. I’ve had a lot of success with Mitzu in particular. (Speaking as a longtime Mixpanel user, I’ve had no trouble at all learning Mitzu—it can even do quite a few things Mixpanel can’t, at a fraction of the cost.) It’s still true that something like Mixpanel or PostHog will be the best bet for a lot of companies (and maybe for you, @Kavir!). But if you’re in a situation where a warehouse-native approach makes sense (or if, like me, you’re in an industry where warehouse-native is basically a must-have for data privacy reasons), you can still have nice things. Eric Metelka: I haven’t tried Mitzu, but both Netspring and Qubit failed, and even Amplitude’s warehouse-native offering hasn’t been a success. There are a number of advantages to warehouse-native, working with privacy constraints top amongst them, but I would want to learn how Mitzu is doing something different here compared to others who haven’t been successful. Ashley Rolfmore: Re: Amplitude vs. Mixpanel. I actually did a side-by-side comparison in my last role! We used GTM (Google Tag Manager, not go-to-market) to funnel events to both to compare. Picked up a few things:
Joshua Herzig-Marx: I see much more variation in my own skill and comfort with the tool than I do between the major players. I do really want to echo Justin’s hot take to focus on back-end events. More specifically:
Marc Dupuis: Our stack right now:
A few extra sources I tap into occasionally:
Jordan Clark: What’s your budget and the team’s bandwidth to implement? This may guide your best solution more than the total events. Melissa: @Andrew Draper, I think Amplitude does automatically capture events with autocapture now (https://amplitude.com/docs/data/autocapture). Curious if anyone has extensively used it. Joshua Herzig-Marx: I think autocapture is dangerous and unnecessary. Most implementations I’ve seen need fewer front-end events and more back-end events. Melissa: Curious your thoughts on why it’s dangerous. For early-stage startups, anyway, autocapture can be super-beneficial since resources are very limited. Back-end events are important too, but I see this usually owned by engineering teams. Joshua Herzig-Marx: I think this is always owned by the cross-functional team! Too many events makes it too costly to separate value from not-value. Your product can’t see itself. Melissa: Sorry, just to help clarify, but by ‘owned’ I mean implemented and monitored by the dev team. Maybe I’ve just been lucky, but the platform eng is usually the one who monitors for platform stability, and as the PM, I don’t tell them ‘how’ to do this, just to ensure that it’s captured. Front-end success metrics are usually ones the PMs will define, as it’s more specific to a success metric for a certain product spec/initiative. I’m definitely not saying the back-end metrics are not important. I’ve just worked with a team that also knew that they were important. Joshua Herzig-Marx: (I generally treat tracking as a non-functional and non-specified requirement. Being able to track is considered part of building, and analytics in test environments is part of proper QA.) Andrew Draper: Just curious (not arguing), but is this maybe a misunderstanding of how autocapture works? It’s definitely not a replacement for defining your metrics beforehand, and there’s not a massive list of events unless you define them, the benefit being that when you define what you want, the data is already there. Granted, most of my experience is $0-10m-revenue startups, and I’m sure it’s different in a lot of scenarios I’ve not been privy to, but I wouldn’t select a product analytics tool that didn’t have it. Melissa: Possible, and same experience here, with $0-10m-revenue startups. Joshua Herzig-Marx: I may have had only bad experiences that ended with hundreds of meaningless events! But also: front-end events are only capturing a fraction of users these days. Andrew Draper: Agreed (and wouldn’t ever not also have metrics that come directly from the actual data)—there is a fix to track all front-end users, though, even ones using “privacy browsers.” If you add a rewrite for the JS snippet to appear from your own domain, it’ll capture their events too. Melissa: I can see how capturing meaningless events can be annoying. I guess it depends on who’s looking at the data. Some certainly can analyze the data and compile a skewed story. I prefer autocapture for early-stage startups more so for time saving. Justin Goff: Counterpoint: Auto-capture for early startups can be extra-risky because you don’t have enough traffic to just lose 20% of your data to ad blockers. That leads to reduced velocity in a bunch of ways, some of them subtle (read: dangerous). Given that modern event tracking tools make it trivially easy to add server-side implementation, I’m pretty skeptical of the idea that auto-capture is a net plus in terms of velocity over time spans any longer than a few weeks. Justin Goff: Counter-counterpoint: if you’re, like, a marketing team and the best you can do is browbeat an engineer into spending an hour setting up auto-capture on your website … that’s definitely better than nothing. Melissa: Back-end data can only capture so much. You’re not going to see behavior and intent that well. Back end is for absolute data (account creations, revenue, database entries, etc.). Front end is for behavioral %. To your point, though, it is important to understand the limitations (both front and back end). Even self-hosted analytics like Matomo will get blocked if the user is running an ad blocker. Your best bet to avoid ad blockers would be to create your own tracking, but most early-stage startups don’t have the bandwidth to do that. Andrew Draper: I didn’t realize this was not widely known, but you can use third-party scripts and get through ad blockers. These instructions are for Plausible but work for any third-party script (give this URL to ChatGPT, tell it your particular environment and it’ll give you exact instructions to do the same thing): https://plausible.io/docs/proxy/guides/netlify Justin Goff: I think we (myself included) might be mixing up three different arguments. Namely:
Addendum: proxying is definitely a thing! It’s a thing you should be doing if you’re using client-side events at all, full stop. But proxying requires extra engineering work that you need to budget for when you’re evaluating whether to rely on auto-capture (and in my experience, a lot of people don’t). Ashley Rolfmore: Re: bypassing ad blockers—I had to turn off all of this to get the software (on the comparison earlier) to work in China. It did initially work very slowly, then they blocked the site completely. Unblocked when we switched client-side tracking off. Ymmv, though! Disclaimer—all my observations assume you are working with a nice taxonomy of events already—had to blat them and start again at least twice over the past few roles. Also want to add that the original ask was <1M events per month, so imo you’re focusing on qualitative insights more—i.e. what potential funnels are there rather than how many thousands of people failed a hyper-specific funnel. This makes the ad blocker aspect less relevant, unless your user cohorts use them extensively. 🤓 Top finds
😂 Meme of the week
Have a fulfilling and productive week 🙏 If anyone in your life would benefit from this newsletter or community, consider giving them a gift subscription 💞 There are group discounts, gift options, and referral bonuses available. Sincerely, Kiyani 👋 Invite your friends and earn rewardsIf you enjoy Lenny's Newsletter, share it with your friends and earn rewards when they subscribe.
|
Similar newsletters
There are other similar shared emails that you might be interested in:
- 🧠 Community Wisdom: Essential reading for early-stage founders, balancing unlimited SaaS plans with profitability…
- 🧠 Community Wisdom: Finding your voice on established teams, focus tips for ADHD brains, how to best use Granola,…
- 🧠 Community Wisdom: Designing great performance review templates, balancing oversight and autonomy with junior te…