📱 Issue #105: The Science Behind Successful Products
Andres Glusman delves into the science behind building successful apps and harnessing the principles of thoughtful experimentation to revolutionize the product development process.
💬 Welcome to issue #105 of Between the Lines
Good morning & Happy Thanksgiving! 🦃
This week, Claremont growth expert and DoWhatWorks founder, Andres Glusman, delves into the science behind building successful apps, the principles of thoughtful experimentation, and harnessing innovative tools to revolutionize the product development process. It’s a Claremont world out there. 👇
~ Josh, Miles, Pat
👤 Community Spotlight: Revealing The Science Behind Successful Apps w/ Andres Glusman
Claremont alum Andres Glusman (PO' 96) is a product/growth leader and the co-founder and CEO of DoWhatWorks. With a background in behavioral economics and early involvement in the Internet, Andres has a deep understanding of online marketing. After co-founding the Performance Marketing Division at i-traffic, the world's first online marketing agency, he spent fifteen years at MeetUp.com as Director, VP for Product, and Chief Strategy Officer before founding DoWhatWorks, a startup aimed at helping growth, marketing, and product teams accelerate their conversion rates.
Andres recently joined Chris Saad and Yaniv Bernstein from The Startup Podcast to delve into the science behind building successful apps. In this insightful conversation, they explore the principles of thoughtful experimentation and share valuable suggestions for harnessing innovative tools to revolutionize the product development process.
Product Experiments - Revealing The Science Behind Successful Apps
I think when you talk about experiments, people probably imagine people in lab coats mixing chemicals and Hadron colliders. In the context of tech startups, Silicon Valley-style startups, can you define what exactly is an experiment? What does it look like?
So in the context of a website in particular, or email, or any kind of experience, an experiment really involves systematically varying the experience across random segments of an audience at the same time to be able to understand how the presence of A, B, or C affects people's ultimate behavior.
So in the simplest of terms, does a button that says ‘Click Here’ work better than a button that says ‘Click Now’? As an example, methodologically, it borrows from the world of science and that you have a control group and you have an effective group, and half the people, for example, will see ‘Click Here’ while half of them will see ‘Click Now.’ You'll run it for some period of time until you reach statistical significance. Then after that, you'll basically see that one of those two variants had a better outcome and ultimately go down and double down on that outcome.
The reason why you might do that is because if you don't run things as experiments, you're not able to really get any kind of causality. And you're not able to really learn what, if anything, that you do makes a difference. And so in the early years of MeetUp, when we were running a lot of different things, we had a really good product sense. We were able to do a lot well. But when it came to optimizing and driving growth and trying to change people's behaviors, which let's face it, MeetUp was the ultimate behavior change, going from an anonymous internet to actually meeting strangers on the internet. You can't get more behavioral change than that.
When we tried to change parts of the website to motivate behavior and get more people to meet up, more often than not, we would just guess. And those guesses not only wouldn't make a difference, but we'd spend months trying to figure out what happened, and we wouldn't actually learn. We wouldn't even get better every time we launched them. That didn't work.
So when you run things as experiments, the upside is that you get lift when it works. When it doesn't work, you get learning. And that's why for people like me who love science, who are very curious, and who want to gravitate towards things that work, the more you can figure out that your ideas are actually backed by fact, which in fact, are the things that approximate the truth of what does move a behavior, the more you can do those things and the more likely it is that you can design an experience that produces the outcomes you want. Get more people to buy, get more people to lose weight, get more people to sign up, whatever it is that you're optimizing for. By running your changes as an experiment, you can learn. And by learning, you get closer and closer to the truth.
When is the right time for startups to run an experiment? Where along the spectrums in terms of the product market fit journey, in terms of the granularity of these things, when should people be thinking about these experiments?
It would be tempting for someone like me, who's in love with experiments to say, oh, you should experiment all the time. Actually, one of the mistakes I made in my career was almost falling too much in love with it, being a man with a hammer, and seeing everything as a nail.
Experiments are amazing at driving learning. Experiments are great at driving growth and conversion improvements. However, there's a real material cost that comes from running experiments, and there's one condition that you need in order for the benefits to outweigh the costs.
At its most simple level, when you're in the earliest stages of a startup, and you want to run an AB test on your website, and your website has eight people visiting a day, it will take you about a year to get enough traffic through that site in order to get statistical significance, in order to be able to know that variant a was in fact better than the variant B. And so, in some regards, until you reach a certain volume, it's a complete waste of your time, because it would just take you too long to get the answer. So when you're really, really, really small and you just don't have the traffic volume to be able to do it, then there's no sense in trying.
The other time when an experiment is not worth doing is when the downside risk is actually quite low. If you are changing your terms of service page, there's really no real reason to run an experiment there, because being wrong doesn't really hurt you that much. Being right doesn't really help you that much either. Then there's not a whole lot of change that's going to happen…..
“I listen to these podcasts, and it sounds like everyone runs experiments, and they get these massive wins. They just quadrupled the size of their revenue. Everyone talks about the checkbox winning, and this thing winning, or that winning, but what nobody gets up on stage and talks about is that 80% of experiments that people run, do not move the needle.” - Andres Glusman
Watch the full episode on The Startup Podcast for Andres’ masterclass in app evolution. 👇
💼 Who’s Hiring?: Drinks & Sydecar
DRINKS, co-founded by Claremont grad and CEO Zac Brandenberg (CMC '99), connects retailers, brands, and marketplaces to the $250B alcohol market. The company optimizes alcohol e-commerce with its patented DRINKS IQ technology, which empowers retailers to leverage AI-based predictive merchandising and marketing to personalize customer experiences. They’ve been recognized several times by Built In and Forbes as a top startup employer, and they’re hiring for several open positions:
Sydecar is a frictionless deal execution platform for venture capitalists. By pioneering a standardized approach to private investing, they enable capital allocators to identify and back entrepreneurs who are changing the world. Sydecar’s platform reliably handles back-office operations, automating banking, compliance, contracts, tax, and reporting so that customers can focus on making deals and building relationships. Claremont grad and VC David Meister (CMC) is the co-founder and CBO, and he’s hiring for several open roles:
Check out the other ~5,000 open jobs at 400+ Claremont-affiliated companies here on our Storyboard. Plus, create a profile and enter your preferences to get alerted to new job postings relevant to you, be they the 1,000+ remote jobs, 100+ internships, or 40+ part-time positions available. We’ve published research that shows that Claremont-founded companies that disproportionately hire Claremont talent outperform — so pay attention, Claremonsters!
If any of these roles catch your eye 👀 , apply and mention Between the Lines. Or, if you are an employer looking to hire tip-top Claremont talent, fill out this form to have your jobs featured.
🗣️ Conversations on the Interwebz:
This week’s top read 🔥
The Economist invited Claremont business leader and LanzaTech CEO Jennifer Holmgren (HMC ‘81) to contribute a piece to their World Ahead 2024 issue. She wrote about LanzaTech’s vision of a circular carbon economy, the carbon recycling technologies that will enable this, and why industry, governments, and consumers must collaborate to make it a reality.
This week’s must-watch 📺
Claremont graduate Emily Mischel (PI ‘14) was featured in the Makers Bar podcast, where she talked about her career path, her time at the Cedars-Sinai Accelerator, and how her startup is radically expanding access to pulmonary rehab. Emily is the co-founder and CEO of FlexTogether – a comprehensive telehealth practice on a mission to expand access to respiratory rehab for millions of chronic lung disease patients.
This week’s Claremont financing 💸
Congratulations to Claremont entrepreneur and CEO Ajay Shah (HMC ‘06) and his company on their recent $84M Series C round led by Norwest Venture Partners. Ajay is the co-founder of CytoVale – a medical diagnostics company focused on providing a more rapid and insightful way to diagnose fast-moving and immune-mediated diseases.
Congratulations to Claremont alum Cassie Borish (HMC ‘11) and co-founder Gilman Callsen on completing their $1M+ Pre-Seed funding. Their startup, Rho Impact, provides data and models that contextualize the environmental impact of climate technologies using software and expert services. Preston-Werner Ventures, founded by Claremont serial entrepreneur and GitHub co-founder Tom Preston-Werner (HMC ‘99), also participated in this round.
Congratulations to VC and Claremont grad Andrew Sheppard (PO ‘99) on closing their second seed fund of ~$60M focused on investing in games and digital entertainment. Andrew is the Managing Director of Transcend Fund, an early-stage VC firm that partners with some of the most ambitious and talented founders in the game industry.
This week’s top listen 🎧
Nobel laureate and Claremont alum Jennifer Doudna (PO) sat down with Marketplace Tech and discussed how using gene-editing technology to modify human microbiomes, potentially with the help of AI, could eliminate some chronic diseases such as asthma. Jennifer is an award-winning biologist who pioneered CRISPR gene editing and made several other significant discoveries in biochemistry and genetics. She is also a recipient of the Gravity Award from the University of Cambridge.
Retail expert and Claremont entrepreneur Alexei Agratchev (CMC ‘99) shared his deepest thoughts on all things retail on the Retailistic podcast with Deborah Weinswig. He talked about selling tech into retail, what it takes to run stores, the trends in shrink, and also a discussion of how AI will impact the industry. Alexei is the co-founder and CEO of the brick-and-mortar retail analytics company RetailNext.
Everything else you need to know💡
Claremont entrepreneur and Vesta CEO Julian Buckner (CMC ‘14) recently announced their acquisition of two furniture rental companies: Furnish and Feather. The three companies will operate as separate brands under one banner, “Showroom,” which will continue seeking additional acquisitions. Julian is the founder and CEO of Showroom Inc. and Vesta, a luxury furniture rental company and the largest home staging firm in the country.
Can’t get enough of Between the Lines? Follow and connect with us on Twitter!
🍽️ BTL Snacks:
💡 What To Know Before Migrating To Cloud….. The cloud has proven to be a transformational innovation. Without proper management, however, it can also become a burden. In a recent article, Laserfiche CEO and Claremont business leader Karl Chan (HMC ‘89) shared the five things he wished everyone knew before deciding to migrate to the cloud. Karl discussed how inefficient cloud use can also result in more complexity, less accessible information and tools, and reduced security.
⚠️ Expert Thoughts On The AI Executive Order….. President Biden just signed an executive order that the White House claims is the most "sweeping action ever taken" to protect Americans from AI, but Reality Defender CEO and Claremont tech expert Ben Colman (CMC) said the measure has dangerous limitations. Ben was recently interviewed by Fox News Digital, where he mentioned how the government should not implicitly trust unreliable safety measures offered by tech leaders.
🪄 The 2-Second Malcolm Gladwell Trick….. Claremont grad and best-selling author, Steve Strauss (CGU ‘83), recently wrote an article for Inc. Magazine on how the first two seconds with a potential customer is all it takes for any business to create a lasting first impression. Steve shared his analysis of Malcolm Gladwell’s NYT best-selling book, Blink, and a few ways businesses can make a great first impression in the digital age.
Feedback? We love to hear it. Hit us with an email. 👊🏼