Data Observability- The Next Monitoring Frontier

From Clouded Judgement by Jamin Ball — read the original.

Notes

• It’s no secret that companies of all sizes are collecting more and more data with each passing day. Snowflake’s 162% net revenue retention, and the fact that more data has been created and captured in the last 2 years than every year prior combined gives us concrete data points on the sheer scale of data growth.
• This has given rise to an entirely new problem for data teams to manage - monitoring the quality of their data and data pipelines.
• The concept of monitoring key businesses process has existed for quite some time. Network teams use Solarwinds, Riverbed or Cisco. Application teams use New Relic, AppDynamics or SignalFX. Infrastructure teams use Datadog, Dynatrace or Elastic. And recently we’ve seen all of these monitoring “point solutions” merge together to offer a single observability platform. Companies like Datadog now have solutions for network, application and infrastructure monitoring, and have also layered in logging.
• Historically data teams haven’t needed anything to monitor their pipelines for two main reasons. First, the pipelines themselves simply weren’t complex enough. Data originated from one place, was moved to one database, and it was very easy to find anomalies or errors.
• Second, data wasn’t (yet) mission critical. It wasn’t being used as frequently to make important strategic decisions, and it wasn’t used “in production.”
• But what happens if that data is somehow “broken” at some point from when it’s created to when it’s fed into the model. Now you’re training models on incorrect, or incomplete data! These businesses may start underwriting unprofitable policies. And the hardest part of all this? You many not have even noticed you had a data quality issue until months later when issues started showing up in your P&L. DataOps
• 95% of the time the outputs of the charts you’re preparing are normal. But every so often something changes DRAMATICALLY. Something just looks “off.” You’re then left scrambling to figure out if something fundamental in the business really changed, or if there’s a data quality issue somewhere “upstream” in your pipeline and the results in your dashboard are artificially skewed.
• Sometimes jobs fail to run and you’re just missing data. Sometimes data is transformed incorrectly. There are many ways data can go “wrong.”
• At the end of the day, if you can’t trust your data, how can you use it! I believe the best solutions will use machine learning to do anomaly detection on data quality issues, and tie in alerting and root cause analysis functionalities to quickly remedy issues that arise. Rules based frameworks simply won’t scale.
• n addition to Monte Carlo there are a number of other startups in this space. Bigeye (formerly Toro Data), Soda Data, Datakin, Datafold, Atlan and Superconductive (behind open source project Great Expectations) are all amazing businesses attacking the core issue of data reliability / observability. Fishtown Analytics also offers a data reliability solution.