What is Observability and how to unleash the power of data analytics with NRQL (I'm a Full Stack Observability Practitioner)
This article was originally published on Dev.to
Observability refers to the extent to which you can understand the internal state or condition of a complex system based solely on knowledge of its external outputs. In simpler terms, it's about gaining insights into what's happening inside a system by examining the data it generates, such as logs, metrics, and traces.
When a system is highly observable, you can quickly and accurately trace performance issues back to their root causes without additional testing or coding effort. This concept has become increasingly critical as cloud-native environments grow more complex and distributed.
You can use this to:
What's the difference between Observability and Monitoring?
OBSERVABILITY:
Definition: Observability refers to the ability to understand a complex system's internal state based on its external outputs. It allows you to gain insights into what's happening inside the system by analyzing data such as logs, metrics, and traces.
Observability is:
MONITORING:
Definition: Monitoring involves assessing the health of a system by collecting and analyzing aggregate data based on predefined metrics and logs.
Monitoring is:
Observability and monitoring are similar but are not the same. Observability enables understanding, while monitoring focuses on tracking specific metrics.
Data observability is crucial in today's data-driven landscape, and this you can easily do with New Relic:
Understanding Data Health:
Definition: Data observability involves monitoring, managing, and maintaining data to ensure its quality, availability, and reliability across processes, systems, and pipelines.
Benefits: It provides a deep understanding of data health, allowing teams to identify, troubleshoot, and resolve issues in near-real time.
Use Cases: Data observability is essential for modern data teams using data for insights, machine learning, and innovation.
Challenges with Bad Data
Statistics:
Impact: Bad data can lead to significant consequences, such as stock price drops and lost revenue.
Detection Difficulty: Unlike application failures, bad data often goes unnoticed until it's too late.
Defense Mechanism: Data observability acts as a defense, ensuring complete, accurate, and timely data delivery.
Components of Data Observability:
Examples of some New Relic Query Language (NRQL) techniques that can enhance your data analysis and provide valuable insights:
Standard Deviation with stddev():
The stddev() function measures the variation or dispersion within a set of values. It helps you understand reported values beyond averages.
Example: To compare the standard deviation of transaction response time ("duration") for the last day to the previous day:
SELECT stddev(duration) FROM Transaction SINCE 24 hours ago COMPARE WITH 24 hours ago TIMESERIES
Bucketing with FACET buckets():
SELECT average(duration) FROM Transaction SINCE 12 hours ago FACET buckets(databaseCallCount, 400, 10)
Advanced Math Functions:
SHOW EVENT TYPES
SHOW ATTRIBUTES
Regex Filtering with RLIKE:
SELECT * FROM PageView WHERE userAgent RLIKE 'Chrome|Firefox'
Nested Aggregation and Subqueries:
Have fun with these:
Unique users
How many unique user sessions did you have in the last week?
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago
Unique user trends
Were your unique user sessions up or down last week compared to the week before?
SELECT uniqueCount(session) FROM PageView SINCE 1 week ago COMPARE WITH 1 week ago
Pageview trends
How can I graph the number of unique users yesterday compared to the day before?
SELECT count(*) FROM PageView SINCE 1 day ago COMPARE WITH 1 day ago TIMESERIES AUTO
OS version
How many of your mobile users are on the latest OS version?
SELECT uniqueCount(uuid) FROM MobileSession FACET osVersion SINCE 7 days ago
Key account Apdex
What is the Apdex score for a particularly important customer? If you have defined some custom attributes, you can query to monitor how this customer experiences your app from a performance standpoint:
SELECT apdex(duration, t: 0.4) FROM Transaction WHERE customerName='ReallyImportantCustomer' SINCE 1 day ago
Check my article on OpenTelemetry, and some exciting NRQL to go with it. Meanwhile you can hire me to help you or your organization with your Observability technique and strategy.
Your Full Stack Observability Practitioner!