In this tutorial, we will explain how to properly perform a microbenchmarking of Java classes using Java Microbenchmark Harness(JMH).

What is Java Microbenchmark Harness?

JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

The problem that we want to solve

We want to compare the performance of AtomicLong and LongAdder classes using JMH.

Adding Gradle plugin and dependencies

We need to add a gradle plugin.

plugins {
	id "me.champeau.jmh" version "0.7.2"
}

Next we want gradle dependencies:

dependencies {
	jmh 'org.openjdk.jmh:jmh-core:1.37'
	jmh 'org.openjdk.jmh:jmh-generator-annprocess:1.37'
	jmh 'org.openjdk.jmh:jmh-generator-bytecode:1.37'
}

Important note: as you can see in the dependencies part we added dependencies in the jmh source set. If you open the plugin source code you will realize that jmh source set is created there. The plugin docs say: “Benchmark source files are expected to be found in the src/jmh directory:”

src/jmh
     |- java       : java sources for benchmarks
     |- resources  : resources for benchmarks

Creating our benchmark class

Inside src/jmh/java we want to create our SimpleBenchmark:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Slf4j
public class SimpleBenchmark {

    @Benchmark
    public long atomicLong() {
        AtomicLong atomicLong = new AtomicLong(0);
        //create threads that will increment atomicLong 10 milion times, execute those threads and measure result
        List<Thread> threads = IntStream.range(0, 10).mapToObj(i -> new Thread(() -> {
            IntStream.range(0, 10000000).forEach(i1 -> atomicLong.incrementAndGet());
        })).collect(Collectors.toList());
        threads.forEach(Thread::start);
        threads.forEach(thread -> {
            try {
                thread.join();
            } catch (InterruptedException e) {
                log.error(e.getMessage());
            }
        });
        return atomicLong.get();
    }

    @Benchmark
    public long longAdder() {
        LongAdder longAdder = new LongAdder();
        List<Thread> threads2 = IntStream.range(0, 10).mapToObj(i -> new Thread(() -> {
            IntStream.range(0, 10000000).forEach(i1 -> longAdder.increment());
        })).collect(Collectors.toList());
        threads2.forEach(Thread::start);
        threads2.forEach(thread -> {
            try {
                thread.join();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        });
        return longAdder.longValue();
    }

    public static void main(String[] args) throws Exception {
        org.openjdk.jmh.Main.main(args);
    }
}

We want to compare methods atomicLong and longAdder. In both methods, we created multiple threads that are just incrementing the counter ten million times. At the end, we want to compare the average time of both methods. Notice that JMH will run multiple iterations of both methods. Also, JMH will take care of a warm-up and other important benchmarking stuff.

Running the benchmark

We can run our benchmark using gradle jmh command. Here are the results on my machine:

Result "com.chess.SimpleBenchmark.atomicLong":
  2054.062 (99.9%) 80.334 ms/op [Average]
  (min, avg, max) = (1827.517, 2054.062, 2183.216), stdev = 107.243
  CI (99.9%): [1973.728, 2134.396] (assumes normal distribution)



Result "com.chess.SimpleBenchmark.longAdder":
  296.987 (99.9%) 31.222 ms/op [Average]
  (min, avg, max) = (273.579, 296.987, 453.291), stdev = 41.681
  CI (99.9%): [265.765, 328.209] (assumes normal distribution)

As you can see, the average execution time of the longAdder method is 297ms, for the atomicLong is 2054ms.

More customization of the benchmark

I won’t go deep into this topic. JMH is highly configurable. You can configure a number of iterations for every method, you can configure how to do warm-up, etc. I never needed some fancy config. Default did the job for me every single time. If you want to read more about performance optimization, I wrote more of my articles on that topic.

Further reading

You can find more about benchmarking good practice in this stackoverflow questions. Answers to that question also suggest a Google tool for benchmarking named caliper. I haven’t used caliper but heard from one of my colleagues that it is good.