I’ve spent a little more time than I would like studying severity models in vulnerability scanners. One question that comes up occasionally is Qualys severity vs CVSS, or any of its competitors, for that matter. Let’s take a look at how Qualys severity compares against the industry standard.
Qualys’ severity score predates CVSS, which is now the industry standard. The two have similarities but there are various reasons for choosing one over the other.
First activate firewall rules that are relevant to non-domain profiles in order to allow traffic for File and Print Sharing and Remote Administration. Then for each activated rule, add the scanner appliance IP address so that the scanner appliance traffic can reach the host. Step 1: Allow 'File and Print Sharing' traffic. A community of security professionals discussing IT security and compliance topics and collaborating with peers. Hi, We are testing the Qualys Technology Add-on (TA) for Splunk v 1.1.0 within our Dev Splunk environment v 6.4.2. At the moment, the Dev environment does not have external internet access to make API calls to Qualys. However, we are seeing a few errors and was wondering if any of them relate to the.
Why Qualys has its own severity score
File And Print Services Access Denied Qualys
CVSS, the industry standard, was only launched in 2004. Vulnerability scanners had existed for about a decade at that point. Nessus came along in 1995, and Qualys came into being around 1999. With no standard, each vendor took its own shot at rating the severity of the vulnerabilities they found. And, frankly, I think their models had as much to do with marketing as they did with security.
A few years ago I heard an interview with Steve Katz, the first ever CISO. He said to report in terms of high, medium, and low. It cuts down on hyperbole, and you don’t really have to explain it. Everyone understands the difference between high, medium, and low.
That’s good security but some would say not the best marketing. Fear sells. Qualys went with Urgent-Critical-Serious-Medium-Minimal. Rapid7 went with Critical, Severe, and Moderate. Tenable went with Critical-High-Medium-Low.
Qualys severity levels explained
Qualys uses a numeric scale of 1-5, ranging from Minimal to Urgent. Higher numbers are worse. I get asked by non-security types why that is and if we can change it. That’s not configurable. I can’t say every vulnerability scanner in history has used a scale where lowest is best, but the surviving big three do, and so does CVSS. I explain it by saying sports statistics aren’t consistent either. In baseball, hitters want high batting averages but pitchers want low earned run averages. Leading the league in touchdowns in football is good, but leading the league in turnovers is bad. In hockey, leading the league in penalty minutes might be both.
- 5, Urgent: An attacker can easily gain control of the system, including full read and write access to files and/or remote code execution.
- 4, Critical: Intruders can possibly gain control of the system, or leak sensitive information.
- 3, Serious: Intruders can gain access to specific information stored on the host and/or deny services or misuse them, such as mail relaying.
- 2, Medium: Attackers may be able to collect sensitive information from the host, such as the precise version of software installed. This type of vulnerability can be used to find additional, more severe vulnerabilities.
- 1, Minimal: Attackers can collect information about the host, such as open ports or, services, and may be able to use this information to find other vulnerabilities.
Problems with the Qualys severity levels
Most of us know that serious isn’t as bad as critical. Critical condition means you’re closer to dying than being in serious condition. But the difference between critical and urgent is much more fuzzy. You don’t go to urgent care when your life is in danger. That’s what hospitals are for. Frankly, if you think someone might be in serious condition, you go to the ER, not urgent care. Urgent care is for when you need stitches, or you’re sick and you can’t get in to see your regular doctor.
Qualys terminology frankly makes almost everything sound like a big deal. And when everything, or almost everything, is an emergency, nothing’s an emergency. I like the CVSS terminology better. It’s more intellectually honest.
I pushed patches for a living for nearly a decade under those everything-is-an-emergency circumstances. The policies conflicted with each other so much I had nothing but a deadline and a passing score of 100 percent to go on. I paid zero attention to severity. It was all based on what I could reboot and when. To anyone who was watching, it looked like I was fixing things in random order. I got it done, fixing 800,000 vulnerabilities along the way, but I don’t think anyone was ever happy with exactly how I did it.
CVSS severity levels explained
CVSS uses a numeric scale of 1-10, ranging from Low to Critical. Higher numbers are worse. The numbers are a little fuzzier but the terminology makes a bit more sense.
- 9-10, Critical: An attacker can easily gain control of the system, including full read and write access to files and/or remote code execution.
- 7-8.9, High: Intruders can possibly gain control of the system, or leak sensitive information.
- 4-6.9, Medium: Intruders can gain access to specific information stored on the host and/or deny services or misuse them, such as mail relaying.
- 3.9 and below, Low: Attackers may be able to collect sensitive information from the host, such as the precise version of software installed. This type of vulnerability can be used to find additional, more severe vulnerabilities.
File And Print Access Qualys
CVSS aims for more numeric precision while settling for broader terminology. But frankly I think the terminology ends up being more precise too. I don’t have to explain the scale with CVSS, you get it.
Interestingly, I find even when people use the Qualys numeric scale, they tend to use the CVSS terminology of Critical, High, Medium, and Low. Almost everyone has to look up what a Qualys SEV 1 vulnerability means.
When to use Qualys vs CVSS severity scores
CVSS is the newer standard, and it’s an open standard. No matter what scanner you use, the same vulnerability will have the same CVSS score. Two companies that use different scanners can agree to have the same policy, as long as they’re using CVSS to measure the results. If your policy states you have to fix Urgent or Severe vulnerabilities in a given length of time, your policy only works with Qualys (in the case of Urgent) or Rapid7 (in the case of Severe). It’s not a good idea to tie your policies to a specific vendor when there’s an open standard available.
That said, some policies have existed longer than CVSS has. That’s why Qualys and its competitors have kept their own severities along with supporting CVSS. That way you can continue to comply with older policies, as long as you haven’t changed tools since then.
I like CVSS better because it’s an open standard, and in a pinch, when someone wants to know why a vulnerability is critical instead of high, I can find out. The major components that go into the CVSS scoring are there in the scan results, and if you want more detail, you can export the knowledge base via the API and get every number that factored into the equation.
What about risk-based approaches?
Both CVSS and Qualys severity are calculated when a vulnerability is released. And I’ve specialized in vulnerability management long enough to tell you not every vulnerability ends up being as bad as we think at first. Meanwhile, others end up not getting any attention at all, even though they deserve it. When they come at a rate of 200 per week, it’s impossible to get anyone to pay attention.
A risk-based approach, such as what Kenna pioneered, recalculates severity based on whether attackers actually use the vulnerability or not. CVE-2018-15473 is my favorite example. It leaks the usernames on a Linux or Unix system via SSH. That’s a Medium or a Low the way Qualys and CVSS measure. Kenna rates it a high, because attackers use it a lot. It’s not as big of a deal as remote code execution, at least, reliable remote code execution. It is a bigger deal than remote code execution that doesn’t work reliably.
Some risk intelligence services use the CVSS equation and just factor in new risk intelligence in the parts of the equation that pertain to complexity and exploit code maturity. Others use their own equations.
Qualys risk intelligence
Qualys doesn’t recalculate risk in its Threat Protection module, which is included in its new VMDR plan. That’s my biggest knock on Qualys Threat Protection and VMDR. It provides threat indicators, but doesn’t provide any easy way to export them so you can do any kind of analysis on them outside the tool. Qualys says they leave it up to you to decide what kinds of threats matter to you. In my experience people really like that, but getting people to tell you which ones matter to them and why is difficult. If they do have an opinion, it’s because of something that happened to them in the past, which may not be a good indicator of future problems.
I’m not sure any three words strike more fear into the hearts and minds of security analysts than the words “Qualys false positives.” Some number of false positives is unavoidable. But the perceived number of false positives is usually an order of magnitude larger than the real number of false positives. Here’s how to estimate how many you should have, how to investigate them, and break the gridlock.
Qualys accuracy
While I hear sysadmins say all the time that Qualys isn’t accurate, that doesn’t mean they’re right. Equifax was breached partly because its vulnerability scanner wasn’t finding everything. What did Equifax do? Initially management blamed the system administrator. That should be a cautionary tale. In the end, Equifax switched to Qualys, because they needed accuracy.
Home Depot is another example of a high-profile business that suffered a breach, then brought in Qualys.
If Qualys were inaccurate, businesses that desperately need to save face and prevent a recurrence wouldn’t be buying it.
Estimating the number of false positives in your environment
When people tell me they have false positives in their Qualys scans, I tell them I believe them. This surprises them. Then I tell them how many false positives I think they have. It’s usually a much lower number than they want to hear. No solution in this space is 100% accurate. Qualys claims 99.99966% accuracy.
That means we can estimate how many errors will be in your scan. Take the number of live hosts you have, multiply it by 50,000 (the approximate number of checks Qualys can do), then multiply that by .9999966 (that’s five nines and two sixes).
In an environment with 50,000 hosts, that means you can expect 8,500 errors. Now, that’s errors. That’s both false positives and false negatives. That sounds like a lot, but Qualys is conducting two and a half billion checks.
I typically find more false negatives, where Qualys misses a vulnerability, than false positives. Those are tougher to find, because you have to actually look for them.
Realistically, the last time I investigated the false positives in a network with 50,000 hosts, they had about 400 false positives. Most of those were on Cisco devices.
False positive gridlock
It was 2006. The sysadmin in charge of patching got promoted. I had experience patching, so the boss decided I’d replace him. So I shadowed him for a week or so before he moved on.
I learned a couple of things from him. First, the words “false positive” made problems go away. But I could tell from the results column of the CSV files the security analysts were sending us that not all of them were false positives. I also noticed he was spending a couple of hours a week defending supposed false positives that I could fix in about half an hour.
I knew I knew more about how Windows works than those security analysts did. But I didn’t like wasting time. And improving security while being honest seemed like a good idea. So I decided to look at that results column and quietly fix what I could. When I couldn’t fix it, I asked questions, and more often than not we were able to figure out what we needed to fix.
Month over month, I installed the new updates as they came out, and worked with the security team to fix the small percentage of updates that failed. And when my sysadmin career came to an end and I moved into security myself, I’d fixed about 800,000 vulnerabilities.
Improving Qualys accuracy
The main cause of scan inaccuracies is running scans without authentication. Without authentication, Qualys has to probe behavior, rather than just checking file versions. Checking file versions is much more accurate, and also easier on the system. If you’re getting lots of false positives, ensure Qualys has accounts with administrator-level access across your enterprise. This means in Active Directory, on your Linux and Unix hosts, your network equipment, and your databases. Many of the false positives I see are due to network scans of systems that host Oracle products. Oracle doesn’t always report the full version number via a port scan, so Qualys will flag systems as vulnerable based on the partial match. An authenticated scan gives Qualys the correct version number, and therefore, correct results.
Backported patches also can cause these kinds of issues. Qualys knows about backporting, but if, say, you’re running SSH 7.7, the backported fix will still report version 7.7, which Qualys will flag as vulnerable. It may flag it as potential rather than confirmed, but still, not what you want to see in the report. With an authenticated scan, Qualys can check the file, compare it against the backport, and report correctly.
Investigating and reporting Qualys false positives
Investigating and reporting Qualys false positives is pretty easy. Look at the results section of your scan. This tells you what Qualys found that it objected to. If it says a particular update is missing, then it found something at the operating system level that suggests the patch either was never deployed, or it failed badly enough that it never told the operating system it finished. This could mean it failed to update the registry on Windows systems, or failed to update the package repository on Linux systems.
Re-installing that update, or an update that supersedes that update, should correct that issue.
More frequently, you find a file or a registry key in the results section. That tells you Qualys found a file that’s not the version that came with the update in question. For whatever reason that file failed to update. It could be the file was locked, or that the operating system reverted the file after the fact, or a disk error caused the journaling filesystem to revert the file. For whatever reason, that file is out of date now, and it rendered the system vulnerable again. Rebooting sometimes clears the fault, if the file is locked and pending a restart to update. If that doesn’t work, uninstalling and reinstalling the update is usually the fastest way to clear the fault.
If you check the file in question and find it doesn’t match what’s in the Qualys results, rescan the machine. It could be the file updated between the time you scanned and the time you investigated. If it persists, then open a support ticket with Qualys. Your security analyst will know how to do that. Qualys will need the evidence plus scan results for just the particular host in question in PDF format.
If the file matches what’s in the Qualys result, Qualys is right. Period. It doesn’t matter what SCCM says. SCCM is right about 85 percent of the time, while Qualys is right 99.99966% of the time.