Editorial note: This blog is based on a live interview between Snowflake’s Josh Klahr, Front’s Théo Gantzer, and Omni’s Jamie Davidson. We’ve lightly edited for length and clarity, but you can watch the full, unscripted conversation here.
New products and features are introduced to the ‘modern data stack’ every day. They’re supposed to unlock new capabilities and finally deliver on the promise of data. And yet, many teams still struggle with challenges left over from the past decade: maintaining data quality and accuracy, making data accessible to every team, and creating a single source of truth across teams and tools.
So why is it still so hard to get the fundamentals of self-service analytics “right”? And what can we do about it?
Recently, Omni co-founder Jamie Davidson sat down with Josh Klahr, VP of Product Management at Snowflake, and Théo Gantzer, Data Analytics Manager at Front, to discuss the challenges many data teams face today and what can finally be done to address them.
Read the highlights from our conversation below, or click the links to jump to a specific topic. To see how our team is tackling some of the topics discussed during this session (geospatial analysis, collaborative data modeling, etc.), check out our weekly Engineering demos.
Meet the data experts #
Jamie: I’m the co-founder of Omni, a BI platform that blends the best of flexible, easy ad-hoc analysis (in whatever way you like) on top of the foundation of a semantic model. I’m here with two experts to talk about how BI has evolved.
Josh: I’ve been leading product management for Snowflake’s core data warehouse product for the past few years, but I’ve worked in data for a long time. I got my start at Yahoo running what may have been the first data Hadoop pipelines, and building Oracle Data Marts using MicroStrategy and Tableau. So, lots of experience—both highs and lows—trying to run BI and analytics on large amounts of data.
Théo: I manage Front’s central Data and Analytics team, which is responsible for foundational data as well as go-to-market analytics and insights. I was the first dedicated data hire at Front and have been here for almost eight years, so I’ve been a part of the development of our data stack from the start.
Advancements in analytics #
Jamie: Josh, you’re at the center of the data ecosystem today, building one of the foremost data products. What are some of the biggest improvements among data tools that you’ve seen?
Josh: There’s been a lot of advancement in the scale and capabilities of analytics platforms—things like separating compute from storage, storing huge amounts of data, instantly bringing online compute clusters to analyze that data, and scaling up and down to match demand. We’ve seen customers analyzing semi-structured and even unstructured data, and wanting to work with that data through a traditional analytics interface, using SQL, or, in the case of semi-structured data, using BI tools. And there’s excitement about what’s possible at the intersection of these and the advancements in AI. For example, use cases like text-to-SQL and AI modalities, like chatbots and agents, are broadening the types of interfaces for working with data and the types of personas that can work with data.
Challenge: Data quality and team velocity #
Josh: You can’t really benefit from advancements in analytics or AI or confidently answer questions without a foundation of high-quality data. I think of quality in two dimensions. The first is the quality of the data you’re capturing. Do you have missing records? Do you have gaps in your data? And the second is the quality of semantics. When someone asks a question about a column or an attribute in the database, do you know what you’re getting? How? What’s the logic behind that? Whether it’s small data or big data, structured or unstructured, building a high-quality semantic understanding of your data remains a challenge today.
Théo: Good data is definitely important – and it’s also hard. At Front, our product changes rapidly and the Data team has to keep up. When talking about data, we need to speak the same language so we can iterate rapidly and reduce the turnaround time between turning an idea into a dashboard. That’s part of what we were looking for in a new BI tool: something that would make dashboarding and exploring more nimble so we could quickly analyze and iterate.
Josh: What Théo said really struck a chord with me. When I was working in a large internal data group at another company, we spent so much time preparing for each product launch—getting the pipelines, data model, and dashboards ready. But despite all of our prep work, we’d find that as soon as the product launched, we got something wrong. People would have questions that we hadn’t built the semantics for, and the cycle of responding to these requests and follow-ups (users wanting to ask another question, add a new dimension, slice and dice the data differently, etc.) required getting that back into the pipeline, updating the semantic model, and projecting it back out onto new dashboards. The process really slowed down the business.
Jamie: No Data team wants to go to the Product team and say “No, you can’t launch this great new feature because we don’t have the logging set up, or we don’t have the reporting set up yet.” No company wants to run that way. And now, with the rise of semi-structured or unstructured data, people can log huge amounts of data without necessarily knowing ahead of time what’s going to be interesting or the questions you’re going to want to ask. It will continue evolving.
Challenge: Single source of truth #
Théo: There are a lot of data tools out there that can solve problems, but having more tools can create a problem in itself. When you add another analytics tool to your stack, you can end up with more challenges around data reconciliation. And the only real workaround is talking to a lot of people until you get the right answer.
That’s okay for a smaller team, but we knew it would be a problem at our current scale. We started using dbt to address a lot of this, but then we needed a better integration with our BI tool to let us define the source of truth in our dbt layer and then reflect that back in the BI layer. So, we went looking for a BI tool that could serve as a single source of truth for data and allow us to have consistent metrics and definitions to increase trust in data and reduce inefficiencies.
Josh: Like Théo described, a lot of companies will have a BI tool that analysts use, another tool that self-service teams will use, and then people using Excel. I like to say that spreadsheets are the oldest BI tool in the world and will never go away. Even within Snowflake, sometimes we look at a dashboard that’s within our Snowsite experience, sometimes we look at a spreadsheet that someone put together, and sometimes the Sales team uses Tableau. We have spent a bunch of our time just trying to figure out why the numbers don’t match and re-running the calculations.
Customers definitely want to have some curated layer for analytics, and while I think there are opportunities to do that, there are still challenges to getting there and then maintaining and keeping it up to date.
Jamie: I’m curious—how have you seen customers work around some of these BI limitations?
Josh: The short answer is lots of different ways. We see increased interest in customers trying to leverage some kind of curated data in the form of tables and views that intersect with an enterprise data catalog. Basically, they’re trying to figure out how to make it so the data catalog owns some semantics and persisted tables and use that layer as the source of standardized views—maybe powered by dbt, maybe using dbt as a metrics layer. It's a very common approach that helps with discoverability and governance, but not with agility. In the time it takes to make the upfront investment and curate the data, something will change. But again, keeping it updated is the real challenge and that can slow things down.
Like you, Jamie, I’ve been a big fan of semantic layers for a long time, and I’m seeing a resurgence in interest. I think that’s partially been fueled by the excitement around AI—there’s very clear data and research that shows that a text-to-SQL experience on a raw schema compared to a curated semantic layer is night and day in terms of accuracy. So a lot of customers are trying to figure out how to come up with a semantic layer that provides some additional projections on top of those underlying database objects and also has some level of flexibility. And they’re often doing it in part because they have a bunch of tools.
Promising early solutions #
Jamie: Given all the challenges you’ve mentioned, Josh, how do you drive consistency? Have you seen workflows or mechanisms effectively pull this data together?
Josh: That’s one of the things that got us excited about Omni. For context, when my team has a business question, we go to our resident data scientist (for us, it’s Louis—hi Louis!). Louis prepares the data, and then we review it, which usually inspires more questions. Because Louis gives us the SQL, we can make tweaks, like adding new calculations or a new group by. So we would get our answers, but the knowledge that came from our additional analysis would be lost.
Omni allows for collaborative work on the semantic layer with experts on both the data and the business sides, so you don’t lose out on any of that knowledge. The business expert and the data expert are able to jointly build out and evolve the data model over time. And this also helps with velocity because you can turn things around as quickly as the product evolves.
Théo: What Josh said really resonates. With Omni, you have definitions sitting on the top of the data, and you expose data to your users. Some users will be able to interact with the data through their own slicing and dicing, and some of them will be able to reflect that back through more accurate definitions. That’s something that got me excited about Omni. We don’t have one trickledown definition that goes one direction, it goes both ways, and Omni allows you to do that without compromising on speed.
Jamie: Théo, can you tell us a bit more about how your workflow has changed since Front started using Omni?
Théo: Since Omni allows users to operate out of a spreadsheet-like interface, you can query the data without having to write SQL (or a tool-specific language like LookML). That’s benefitted us within the Data team, but we’re also getting traction with people on other teams and some of them are already becoming experts. The fact that we have all teams operating out of the same tool is really helpful.
Jamie: Josh, we’re super excited about our partnership with Snowflake and its investment in Omni. Can you talk about what makes Snowflake and Omni a good fit?
Josh: Part of it was the aligned vision of broadening access to analytics. Having all these different interfaces—drag-and-drop, dashboard consumption, the spreadsheet-like experience—it expands the personas that can consume data. We also believe strongly that semantic layers are a great way to democratize data in a way that satisfies governance requirements and doesn’t lock down self-service for other teams. Having that model in the middle is super important.
There’s definitely compatibility in what we think effective modern data architecture for analytics looks like: a multi-user analytics interface, a shared semantic model that provides value and governance, and then a data platform (like Snowflake) that gives you the scale, governance, and reliability you need to run your analytics.
Jamie: Having been in the industry for a long time, our team has tracked the many advancements in analytics and we’ve baked them into Omni. We’re always asking: how do we take advantage of everything that Snowflake is doing—how it’s gotten faster and cheaper, how it’s better for unstructured, geological data, etc.? We’re betting that Snowflake will continue to improve over time, and we want to make sure we build a platform that makes the most of those improvements.
For the full unscripted interview, watch the video. If you want to learn more about self-service analytics with Omni, we’d love to hear from you.