❄️ Issue #93: Making Snowflake Queries Smarter and Cheaper
Dremio's unicorn Claremont founder announces his new company's Query Engineering Platform for Snowflake and shares the inspiration behind his 25-year career in the data world
💬 Welcome to issue #93 of Between the Lines
Good morning & happy Thursday. Historically, the Claremont Colleges have already produced ~27 unicorns, one of which is Dremio – the data lake platform co-founded by Claremont grad and serial entrepreneur Jacques Nadeau. Dremio reached a valuation of $2B last year and was dubbed by Silicon Valley analysts as the first new unicorn of 2021.
This week, Jacques shares the exciting announcement about his new company, Sundeck, coming out of stealth. He lets us in on what he and his team have been working on for the past year and how his company’s new Query Engineering Platform gives the everyday data worker more control and creativity over their data.
It’s a Claremont world out there. 👇
~ Josh, Miles, Pat
📢 👥 Community Voices: Making Snowflake Queries Smarter and Cheaper with Sundeck
Author: Jacques Nadeau
Claremont grad Jacques Nadeau (PO) has worked in the commercial and open-source data worlds for 25+ years. After Claremont, he assumed different engineering leadership roles for Quigo, Offermatica, and aQuantive before starting his search engine startup YapMap. Jacques also led MapR’s distributed systems team and co-created The Apache Software Foundation before joining Dremio as co-founder and CTO in 2015. Dremio is the original creator of Apache Arrow and is a high-performance SQL (data) lakehouse that helps to accelerate BI and Analytics directly on cloud data lake storage. Under his leadership, the company reached a $1B+ valuation in 2021.
After six years of overseeing Dremio’s entire information technology department as CTO, Jacques decided to step down and start his new company, Sundeck. Jacques and his team at Sundeck have been building a new product to enhance users’ experiences with Snowflake and create a better way to work with existing cloud data warehouses. They recently came out of stealth. Check out the full announcement here and read about his recent interview with Snowflake below:
What inspires you as a founder?
Empowerment. We all want to feel empowered. This is true in both our personal and professional spheres. As a founder (and OSS creator), I’ve generally looked for new ways to empower people in their daily “data” life. A key theme of this has been democratization of database internals and the belief that powerful data tools should be available to the everyday data worker, not just those with PhDs in database internals.
Apache Arrow came out of the realization that in-memory processing and transfer should be something everyone can access. Apache Calcite, another project I’ve been involved in for the last decade, is focused on democratizing query optimization and transformation. Substrait—a newer open source project I recently co-created—focuses on standardizing query plans and relational algebra.
How does this translate into what Sundeck does?
I’ve had several chances to build systems that influence how queries are expressed, interpreted, and executed through sophisticated query flows. These flows help organizations block query anti-patterns, mitigate security and privacy risks, enable routing to optimize performance and/or cost, and can enrich user interaction to make data consumption easier and less error-prone. The systems that enable this “query engineering” fundamentally change the breadth of tools data engineers and analysts have to get their job done.
Unfortunately, this level of power has historically been something only the most resource-rich data organizations can build and maintain. Sundeck looks to provide this kind of powerful platform to all data workers in much the same way Apache Arrow helped data scientists and data engineers access high-performance in-memory computing.
How can a query engineering platform help Snowflake users?
Sundeck is a query engineering platform that allows analysts, data engineers, and DBAs to influence where, how, and what queries run on Snowflake.
Sundeck allows any Snowflake user to create a new query flow, which can then be configured to execute additional operations before and/or after a query is executed in Snowflake. The simple concept is very powerful in practice. For example:
Hate it when a new user mistakenly does a query on a large fact table without limiting the set of results they return? Just define a Sundeck reject pre-hook that uses our QLIKE query-matching technology to identify specific suboptimal query patterns.
Want to shutdown warehouses immediately off-hours once the last running query completes? Define a Sundeck SQL post-hook that examines current load and time of day to suspend idle warehouses.
Want to implement tighter cost controls per user so each can run a maximum of $50 of queries each day? Implement a Sundeck SQL post-hook that collects query activity and records that in a Snowflake table, which is then consulted in a SQL pre-hook to reject excessive consumers (unless a manager overrides).
Want to automatically monitor dbt model-processing telemetry and reroute individual model executions to avoid excessive spilling, slow completion times, or warehouse inefficiency? Create a Sundeck SQL pre-hook that uses the model being created to look up historical trends of target warehouse size and the current warehouse load to route your operation intelligently.
All of these patterns are hard to achieve without a query engineering platform, but become trivial once you have one. It allows all Snowflake users to better leverage their investment and create query flows that can do previously impossible things.
What are you personally excited about next?
I’m super excited to be making Sundeck publicly (and freely) available for Snowflake users today. We’ve found that the patterns it introduces both enrich and expand the ways Snowflake can be used, and we hope other Snowflake users find the same.
I’m also very excited about Snowflake’s continued push into Data Cloud-managed computing. Snowpark was a great first foray and initial step. Snowflake Native Apps and Streamlit are exciting but still just scratching the surface of the expressibility we have in the broader domain of software development. With data as the backbone of many modern applications, Snowflake has an opportunity to fundamentally invert the way we build applications, allowing organizations to own control of their operational and reporting data while leveraging modern technologies in a secure way. We’ve been saying “bring the processing to the data” for years. With the right tooling, we can achieve that at the application level—and celebrate the demise of both ETL and reverse ETL!
Long term, the shift the entire industry is going through as we figure out new ways to apply and leverage powerful LLMs is exciting. It’s already drastically changing how we work with data. We’re just in our infancy around the application of these tools and how they are best leveraged. Chat is a great generic killer app, but merging our thinking with the acceleration of modern machine learning in a more streamlined and iterative way is going to fundamentally reshape our world. That’s daunting and awesome.
Read the full interview to learn more about how Sundeck can help Snowflake users perform previously impossible tasks.👇
💼 Who’s Hiring?: Social Cipher & Speakeasy
Vanessa Castañeda Gill (CMC' 18) is the CEO and Co-founder of Social Cipher – a social-emotional learning platform that connects neurodivergent youth and their advocates in an immersive virtual world. Last year Social Cipher raised a Pre-Seed round led by Difference Partners. The LEGO Foundation also recently selected the company to join a new cohort for their Play for All Accelerator, a program that accelerates products, services, and experiences that recognize neurodivergent children's strengths and nurture their skills. Vanessa is looking for a Digital Marketing Manager to join their growing marketing team:
Speakeasy is a developer-first API Ops platform building Zero Ops APIs and helping other companies deliver Stripe-quality API experiences to their customers. Their platform takes the heavy lifting out of providing a great API experience by integrating directly into your GitOps and propagating changes. Claremont grad and previous LiveRamp Director of Engineering Sagar Batchu (HMC '15) co-founded the company. Check out our BTL Interview with Sagar if you want to learn more about him. They’re currently hiring founding team members:
Check out the other ~5,000 open jobs at 400+ Claremont-affiliated companies here on our Storyboard. Plus, create a profile and enter your preferences to get alerted to new job postings relevant to you, be they the 1,000+ remote jobs, 100+ internships, or 40+ part-time positions available. We’ve published research that shows that Claremont-founded companies that disproportionately hire Claremont talent outperform — so pay attention, Claremonsters!
If any of these roles catch your eye 👀 , apply and mention Between the Lines. Or, if you are an employer looking to hire tip-top Claremont talent, fill out this form to have your jobs featured.
🗣️ Conversations on the Interwebz:
This week’s must-watch 📺
Claremont grad and Forbes 30 Under 30 awardee Varun Puri (CMC ‘16) took the stage at TEDxStGeorge to deliver the first-ever TEDx Talk written by AI. He discussed how anyone can now use generative AI-powered speech coaching to write and give compelling talks and shared his company’s vision for the future of AI in communication. Varun is the Co-founder of Yoodli – an AI-powered speech coaching platform.
This week’s Claremont financing 💸
Congratulations to Claremont grad Roger Howe (HMC ‘79) and ProbiusDx on their recent Series B raise from Zoic Capital. Roger is the Co-founder and Technology Advisor of Probius – a deep-tech company bridging the data gap between biology and AI. Probius is making biochemical data easier to access and use with AI workflows, making it faster to find detailed disease insights.
Congratulations to Claremont entrepreneur Eric Youngstrom (PI' 91) and his company, Onramp Funds, on their recent raise. Eric is the CEO and Founder of Onramp – an innovative financing platform built for e-commerce entrepreneurs. The company’s successful raise brings its total capital to ~$46M overall.
This week’s top listen 🎧
Claremont entrepreneur and Product Growth Expert Andres Glusman (PO' 96) joined the One Knight In Product podcast to talk about the pros and cons of split-testing, how it can accelerate product growth, and whether it yields results for businesses. Andres is the Co-Founder and CEO of DoWhatWorks.io – a startup helping growth, marketing, and product teams accelerate their conversion rates. He’s also been running tests online since the late 1990s.
Claremont entrepreneur Bobby Tzekin (PO) is the CEO and Co-Founder of Wisetack – the leading Pay Over Time platform for in-person services. He was recently on the Payments On Fire podcast to explore the intersection of three significant payment trends: helping small businesses modernize their payment acceptance options, embedded finance, and the consumer lending phenomenon known as Buy Now, Pay Later.
Everything else you need to know💡
Claremont tech entrepreneur Ben Colman (CMC) was invited back to CNBC’s ‘Last Call’ to break down how to spot an A.I. generated photo after a fake picture of an explosion outside the Pentagon went viral on Twitter and impacted the US markets. Ben’s company, Reality Defender, is a deepfake detection platform enterprises use to flag fake users and fraudulent content in real-time. He also recently wrote about Twitter's crowdsourced attempts to tackle deepfakes.
Can’t get enough of Between the Lines? Follow and connect with us on Twitter!
🍽️ BTL Snacks:
🌱 Bringing Climate-Smart Seedlings To US Markets….. Earlier this year, Claremont grad Maddie Hall (CMC' 14) and her company, Living Carbon, announced the successful large-scale planting of their climate-smart seedlings in the US. In her blog, Maddie talks about her journey with Living Carbon – how they raised a successful Series A to start developing their first carbon projects, her experiences navigating the USDA’s plant regulatory process, and some promising progress updates on their photosynthesis-enhanced trees.
❌ The Mistake In Your Career Development Plan….. Most professionals usually think they can rely on others (their employer and manager) to manage their careers. Claremont grad and career development expert Mollie Amkraut Mueller (CMC' 10) shares why this is one of the most common mistakes in career development. She talks about the three reasons why professionals often get this wrong and shares her insights into how career development will change over the next five years because of AI. Mollie is the Co-founder of a career design platform company – Crew.
🎯 A Challenge For The Biotech Community….. Biotech is ideally suited to solving planetary-scale challenges because of its two extremes: precision and scalability. But there are several reasons why it hasn’t been harnessed fully to address climate change. In his recent article, Claremont grad and Homeworld Collective Founding Co-Director, Daniel Goodwin (HMC ‘08), discussed the challenges involved in engineering and deploying biotech and shared a few thought-provoking statements to challenge the biotech community.
Feedback? We love to hear it. Hit us with an email. 👊🏼