Purpose
- Compare NULLNESS-focused analysis performance across tools using
hyperfine.
- Keep scope to common nullness semantics rather than total rule count.
Methodology
- Benchmark metric: median runtime (
hyperfine JSON median).
- Parameters:
--warmup 1, --min-runs 5.
- Generated at (UTC):
2026-08-16T03:01:54Z.
Datasets
- Library: Guava
33.6.0-jre.
- Binary input: Maven Central JAR.
- Source input: Maven Central source JAR.
- Application: SonarQube
26.8.0.126808.
- Binary input: Maven Central
sonar-application ZIP.
- Source input: GitHub tag source archive.
| Tool |
Version |
Nullness scope used in this page |
| inspequte |
workspace build |
NULLNESS rule only |
| SpotBugs |
4.10.3 |
NP_* via include filter (Bug code=NP) |
| PMD |
7.14.0 |
null-related subset in category/java/errorprone.xml (cache=off) |
| Checker Framework |
3.52.0 |
NullnessChecker |
| NullAway |
0.13.8 |
Error Prone plugin (error_prone_core 2.50.0) |
Environment:
- OS: Linux
- Kernel: 6.17.0-1022-azure
- CPU: INTEL(R) XEON(R) PLATINUM 8573C
- Java: openjdk version "21.0.12" 2026-07-21 LTS
Results: Guava
| Tool |
Median |
Mean |
Min |
Max |
| inspequte |
0.293 s |
0.293 s |
0.283 s |
0.301 s |
| nullaway |
1.225 s |
1.220 s |
1.182 s |
1.240 s |
| checker-framework |
1.849 s |
1.842 s |
1.792 s |
1.885 s |
| pmd |
4.690 s |
4.738 s |
4.593 s |
4.918 s |
| spotbugs |
21.347 s |
21.076 s |
18.038 s |
22.527 s |
Results: SonarQube
| Tool |
Median |
Mean |
Min |
Max |
| pmd |
11.380 s |
11.358 s |
11.113 s |
11.550 s |
| inspequte |
13.409 s |
13.439 s |
13.375 s |
13.552 s |
| spotbugs |
748.348 s |
749.397 s |
747.375 s |
752.560 s |
Caveats and Fairness
- Rule sets are aligned to nullness intent, not full one-to-one semantic equivalence.
- Source-oriented tools and bytecode-oriented tools have different execution models.
- For operational comparison, an order-of-magnitude speed difference is acceptable in this report.
- This page compares performance only, not detection quality or precision/recall.
Repro Command
bash scripts/bench-nullness-compare.sh --dataset all --min-runs 5 --warmup 1
bash scripts/render-performance-docs.sh