That’s why it’s important is because it helps you stay proactive and maintain a lot of trust in your system. And over the past decade with the rise of tools like Datadog, we have the visibility so that your team can be proactive and get ahead of breakages. The only way that you knew that something went wrong in your system was degraded or broken performance for your users, and that is not acceptable. I’d never say I didn’t do this, but a lot of people did do this. Make a rails app, put on an ECT box, put a heartbeat check there and call it a day. So, let’s go back 10 years ago, or 20 years ago when it was more common to deploy software systems, without any sort of telemetry. And it costs trust in the people who rely on the systems that you develop. And because it’s difficult to understand, it costs a lot of time for you, a time that is hard to get back. Kevin Hu 00:03:07 It’s necessary because systems are complicated that as software engineers, we have so many systems working independently of each other, interacting with each other, that when something goes wrong, which it inevitably will, it’s very, very time consuming to understand what the implications of that incident might be and what the root cause might be. Priyanka Raghavan 00:02:54 Why do you think it is necessary to have a view of the system, the centralized view, which everyone seems to be striving towards? Why is that necessary? And it really descends from the Physical Science discipline of control theory, where there was a concept called the Controllability of a system that given the inputs, can you manipulate and understand the state of that system? Well, the mathematical dual, the corresponding concept is, given the output of a system, can you infer the state of that system? So that is the rigorous definition from which our more colloquial definition is derived. And that is the colloquial definition that we use in data observability and what software observability / DevOps observability tools like Datadog and Signal Effects and Splunk have developed. Kevin Hu 00:02:06 Observability is the degree of visibility you have into your system. How would you define observability in your words? The first thing I wanted to ask you is something basic, but let’s start from the top. Priyanka Raghavan 00:01:48 So let’s jump into observability and some definitions before we get into data observability. Kevin Hu 00:01:21 I think you did a great job with the introduction and we’ll touch on this during the show, but I would love to start by saying data teams have so much to learn from software teams, that if you have a data team at your company, chances are that a lot of the best practices that you have developed as an engineer could also help them deploy more effective and more resilient data for your stakeholders internally. Is there anything else you would like listeners to know about yourself before we get into the show? So hopefully I can make them proud today for such a pleasure to be here. I’m a long-time listener of SE Radio and everyone on my team also is a listener. Kevin Hu 00:01:04 Such a pleasure to talk with you today. Kevin has written many articles on data observability in a variety of popular, as well as scientific publications. Prior to this, he researched the intersection of machine learning and data science at MIT, where he earned a PhD. It’s a data observability startup, which focuses on helping teams find and fix data-quality problems. Today, listeners will be treated to the topic of data observability, and to lead us through this we have with us our guest Kevin Hu, who’s the co-founder and CEO at Metaplane. This is Priyanka Raghavan for Software Engineering Radio. Priyanka Raghavan 00:00:16 Hello everyone. To suggest improvements in the text, please contact and include the episode number and URL. This transcript was automatically generated. Transcript brought to you by IEEE Software magazine. Episode 457: Jeffery D Smith on DevOps Anti Patterns.Episode 472: Liran Haimovitch on Handling Customer Issues.Episode 487: Davide Bedin on Dapr Distributed Application Runtime.Episode 455: Jamie Riedesel on Software Telemetry.Kevin Hu – Data observability and why it matters From there, the discussion turns to tooling, what a good data engineer should look for in data observability tools, Metaplane’s offerings, and challenges in the area and how the field might evolve to solve them. Starting from basics such as defining terms and weighing key differences and similarities between software and data observability, the episode explores components of data observability, biases in data algorithms, and how to deal with missing data. Kevin Hu, CEO and co-founder of the startup Metaplane, chatted with SE Radio’s Priyanka Raghavan about data observability.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |