Nobl9 tracks service-level objects for faster reliability response

00:00  
Hi everybody, welcome to DEMO, the show where companies come in and they show us their latest products and services. Today, I'm joined by Brian Singer. He is the co-founder and chief product officer at Nobl9. Welcome to the show. So what is Nobl9, and what are you going to show us today here on the show?
 
00:14
So Nobl9 is a reliability management platform, and we use a concept called service level objectives to help companies be more proactive about building reliability into their products. We're going to show you a little bit about our product today, talk about how it works, and talk about how service level objectives work as a mechanism to essentially build more reliable software.
 
00:37
OK, who is this mainly designed for within the company, but also the different types of companies that would benefit from this?
 
00:44
So primarily for anybody that has a stake in the reliability of the product. And there's a few different groups that are typically using Nobl9. First and foremost is the developers who are supporting products and services, digital services, as well as site reliability engineers that are responsible for uptime and maintaining the reliability for end customers, as well as product managers who are really concerned with the tradeoffs between spending time addressing tech debt and building new features. And of course, because reliability is so core to many businesses today, we have executives that are interested in the sort of reports that we're producing as well.
 
01:24
Now, when I was looking at your site, you have a couple of terms that you use a lot through the demo, so what is a service level objective? I think you also use the term error budgeting, which is different from a regular budget, right?
 
01:35
That’s correct. So a service level objective is basically just a construct that allows one to say, I want something to happen some percentage of the time. In the case of software, it might be I want this API to respond in under two seconds 99% of the time. And it helps us build in tolerances into these services. So the error budget. If I want something to happen 99% of the time, then I'm okay with it not happening 1% of the time, and that's effectively my error budget. And for a service level objective, what we do is we define a time frame over which we're calculating this, usually seven or 28 days, and then we define that objective how often we want that thing to happen, and the error budget is effectively then the remainder, okay? And that gives us some very interesting data about how that service is performing. Because if you think about it, if something fails a little bit over the course of a month such that it's not perceptible by users, and we eat that 1% of error budget, that's probably fine. If that 1% of error budget is effectively gone in the course of an hour, that probably tells us that we have an outage.
 
02:53
So the problems that you're solving with the Nobl9 platform, what's the big problem, and why should companies kind of care about this? What would they be doing if they didn't have this platform?
 
03:07
So it's twofold. A lot of companies recognize the benefits of having service level objectives, but actually putting them in place and managing them across a big enterprise is a pretty big challenge, because the telemetry comes in many different shapes and sizes. So one thing that Nobl9 does is it normalizes all your existing telemetry down into these error budgets, and it does it in a way that's developer friendly and effective. The other thing that we do is help you get more insight from your existing telemetry, because you're looking at it through the lens of error budgets, and we're providing a lot of user experiences that we'll show in the demo that make it really easy to understand what's going on with the service level objectives. So if a company doesn't have a solution like Nobl9, they might be able to calculate a few SLOs and error budgets, but it's really hard to have a full SLO program and reliability culture.
 
04:04
Let's get into the demo and then show us some of the key features.
 
04:07
So as you can see here, I have actually some SLOs that I'm calculating in one of our demo organizations. And I'm going to show a few different things right here. We're looking at actually the latency of a API, and that API is called ingest and this SLO. So I mentioned that the SLO is how often a thing happens, and then you have to define what that thing is. So in this particular case, we want this API to return in under 300 milliseconds, 50% of the time, and we want it to return in under 500 milliseconds, 95% of the time, and under two seconds, 99% of the time. Okay, so that's one really cool feature of SLOs, is that as the tolerance gets wider, you want it. And it's basically capturing my tail latencies here.
 
05:04
And there's some calculations that a company has to make before like customers will get upset if they reach that certain threshold, right?
 
05:11
Exactly. And the cool thing about SLOs is that you can set them on every layer of your infrastructure. So if I know that this API needs to return in 300 milliseconds? Well, I know that my networking layer has to be able to route that request within 50 milliseconds, for example. So you're using them at sort of every level to build out reliability.
 
05:30
And in this case, you know, I'm going to assume green means good, and red means bad.
 
05:35
So you can see that. So I have these three thresholds that I've set, and they each have different targets. So even though, you know, this is sort of my tightest and this is my loosest, I actually have more error budget here, because the target is, is only 50% Yeah, right. And you can see, actually, in the error budget remaining, I have almost 300 hours, because over the course of 28 days, 50% is a lot of time, whereas here, the target is 99% and over the course of this 28 days or 30, I think this is set to 30 days, I've already consumed all my error budget, so I can actually drill into this SLO and get a lot more detail about what's going on. One of the cool things about Nobl9 is that when you create an SLO with multiple objectives, which is pretty common in this practice, you can set one of them to be the primary objective, the one that you're really focused on, and you want people to focus on as to whether you're meeting your reliability objectives. And here I can see I haven't set one, but in this case, I'll set it as the fast one that we're on right now.
 
One of the neat things about the Nobl9 platform is what we call a calendar-aligned SLO, which means that we're starting to calculate the error budget on the first of the month, and then at the end of every month we're resetting. You can also have SLOs that we call rolling, which basically is a rolling window that will constantly update, so as you have bad points roll off, you'll get more points. This SLO is more useful if you're trying to track an SLA for a customer like you say, oh, over the course of a calendar month, I need to be 99% available. And it really helps you do that. So you can see the requirements we have for that SLO. You can see actually the query, this is the actual indicator that we're testing against, that hypothesis, that it'll be less than 300 milliseconds, and it's basically how long an HTTP request to this takes to this ingest service.
 
One of the neat things about Nobl9 is that we support bringing in telemetry from a really wide array of providers. So this particular SLO is created, I think it's a Prometheus provider that's basically giving us the data. But we support Datadog, we support New Relic, we support pretty much all of the different observability systems that are out there. So we can see here, with this objective, you know, our target’s 50% we're at almost 90% we're looking good, nothing to really be worried about right here, I can see, actually, you know, this is the threshold that we're setting, and most of the time we're under it, but some of the time we are actually burning a little bit of error budget, and that's actually happening right now.
 
08:32
So this will tell you when something does not hit that threshold exactly?
 
08:36
In real time, we're calculating whether we are successfully hitting that threshold with this indicator, and you can see that we're actually not burning all that much error budget, even though we've had a little latency spike, yeah, and that's just because the target here is so low, yeah? Now, if we flip over and we look at the red ones, yeah, we're going to go to one of the red ones. Now to this, this poor objective, you can see that right? This is the same indicator. And even though the threshold that we're trying to hit is looser, it's 200 milliseconds we're at, we're actually burning error budget at such a rate that we'll exhaust it in a fifth of the time period. So that actually probably is enough if we had an alert configured on this SLO to alert us. And that's one of the cool things about SLOs, is that the error budget-based alerting is a really good indication when something is happening in the system that's going to cause customer pain, as opposed to your typical threshold based alerting, where it's noisy, it flips on and off, you might get paged, and you're not sure if it's something that's going to really impact.
 
Let's look at this ingest error rate SLO, and I noticed this actually has an alert that's firing right now. And I'll drill into this one here, and we see that we have this alert policy, which is called Fast Burn and this, this, we set this to a high severity, and we can see here that this triggered for this particular SLO here. And we can see, basically the condition that caused this to trigger. We're at 1,000x burn rate in this SLO. That's an indication that something is wrong. One of the cool things about Nobl9 is that you can create alert policies and then attach them to multiple SLOs. So this particular policy is basically saying, if the average error budget burn rate, that's the rate at which I'm consuming, that error budget is greater than 20x so 1x burn would mean I would exhaust all my error budget at the end of the period. 20x means I'm going to exhaust it in a 20th of the period, right? That's typically a pretty high burn rate based on an alerting window of five minutes. We want it to trigger this alert. When this alert’s triggered, we can actually add, and I don't have any added into this because I didn't want it to actually page me. But we can actually add an alert method, right? I can basically say, send this to PagerDuty, send this to JIRA, that kind of thing. So that's a really good way to look at sort of what's going on.
 
11:10
Once you see some of these things that are that are triggering high burn rates, can you dig into the details, or do you have to then go to the other monitoring system to find out what the reason?
 
11:19
So typically you'll be looking at maybe your dependencies in Nobl9 and seeing what other things are actually burning error budget right now. And we have a great dashboard service health by burn rate, right? So this basically shows you everything that's burning error budget right now. Typically, if you're burning error budget, it's probably because of something that you did to the service or something that's wrong with the dependency. So in this case, I see this notification success rate, SLO is burning. I see a couple of these other SLOs are burning that might be telling me something about what's actually going wrong. Now sometimes you actually have to go into the log data, right? And one of the cool things about SLOs is that they typically give you the jumping off point to do that, and within Nobl9, like we'll give you a link to go analyze the SLO that's actually failing.
 
12:12
So I wouldn't just want to set such a low bar that I'm hitting all of the thresholds, because that's not really going to make any good business sense. You do have to have to have some of these, yeah, these increasing bars so that you really know if you're causing some pain for customers.
 
12:27
And that's a great segue. One of the tools that we have in Nobl9 is a tool that lets you analyze the indicators and determine how reliable they should be. So if I go into this one, for example, just throw numbers out in the air. So you know that might be a conversation with a product manager to say, hey, when a customer is using this, what do you expect it to do? One of the SLOs we have within Nobl9, for example, is chart data latency. A lot of things in the platform can affect how long it takes us to actually pull the chart data in, right? It could be the load balancer that the user is coming in through. It could be the connection to our internal time series database, right? So it's not necessarily that's going to tell me exactly what's wrong, but that's definitely something that our engineers want to understand if there's, if there's an issue, right? So what we're doing here, actually, is we're pulling in some historical data and we're analyzing it. We're basically telling whoever is creating the SLO look, the P99 so basically, the 99th percentile of these requests is about just under a second. You know, point eight, nine seconds. So if you set your threshold to one and the target to 99% you'll be just about good on the error button, right? And so typically, what you'll do is you'll go back and say, Okay, let's look at some periods where we know we were successful, and maybe where we know we had an incident, and find that threshold where, you know we're sort of representing the ideal customer experience.
 
14:08
So how is this relatively easy for an IT team to set up, you know, if they've already got existing monitoring systems, and to deploy your system on top of it?
 
14:20
Relatively easy, very straightforward, and I mentioned that we support a lot of sources of data today, so whether you just have your data in Amazon CloudWatch, or you're using Prometheus, or you're using some of these other tools, they're pretty straightforward integrations. We can actually put an agent behind the firewall so that data is not exposed to the internet, or we can connect directly to a SaaS provider to gather it.
 
14:43
And you offer a free trial of this platform?
 
14:47
Yes, we offer a free trial. If you're interested in trying out the software, reach out to us and we'll get you set up.
 
14:52
Because there's a lot of other features that we didn't get into. But where can people go for more details on the platform?
 
14:58
So if you go to Nobl9.com, we'd be happy to give you a full demo. We also have a demo environment that is newly launched, actually this environment, so you can actually sign up and just start, you know, sort of perusing through the different features and functions of the platform.
 
15:12
So you can play with it without making anything to go haywire, right? All right. Brian Singer, thanks for being on the show and thanks for the demo.
 
15:20
Great. Thanks for having me, Keith.
 
15:22
That's all the time we have for today's episode. Be sure to like the video, subscribe to the channel and add any thoughts you have below. Join us every week for new episodes of DEMO. I'm Keith Shaw, thanks for watching.