Archive of posts with the tag Benchmarking

• Sep 17, 2024: Instrumentation-based Profiling on JVMs is Broken!

  Last year, we looked at how well sampling profilers work on top of the JVM. Unfortunately, they suffer from issues such as safepoint bias and may not correctly attribute observed run time to the correct methods because of the complexities introduced by inlining and other compiler optimizations.

• Sep 20, 2023: Don't Blindly Trust Your Java Profiler!

  How do we know on what to focus our attention when trying to optimize the performance of a program? I suspect at least some of us will reach for sampling profilers. They keep the direct impact on the program execution low, and collect stack traces every so often during the program execution. This gives us an approximate view of where a program spends its time. Though, this approximation as it turns out can be surprisingly unreliable.

• Jun 6, 2023: Squeezing a Little More Performance Out of Bytecode Interpreters

• Oct 10, 2022: The Cost of Safety in Java

  Overhead of Null Checks, Array Bounds, and Class Cast Exceptions in GraalVM Native Image

• Sep 30, 2021: How do we do Benchmarking?

  Impressions from Conversations with the Community

• Jul 7, 2020: Is This Noise, or Does This Mean Something? #benchmarking

  Do my performance measurements allow me to conclude anything at all?

• May 27, 2019: Generating an Artifact From a Benchmarking Setup as Part of CI

  Disclaimer: The artifact, for which I put this automation together, was rejected. I take this as a reminder that the technical bits still require good documentation to be useful.

• Oct 25, 2016: Cross-Language Compiler Benchmarking: Are We Fast Yet?

  Research on programming languages is often more fun when we can use our own languages. However, for research on performance optimizations that can be a trap. In the end, we need to argue that what we did is comparable to state-of-the-art language implementations. Ideally, we are able to show that our own little language is not just a research toy, but that it is, at least performance-wise, competitive with for instance Java or JavaScript VMs.

• Jun 25, 2016: Writing Papers with Completely Automated Data Processing

  One of the first things that I found problematic about paper writing was the manual processing and updating of numbers based on experiments. Ever since my master thesis, this felt like a unnecessary and error prone step.

• Sep 18, 2011: Using R to Understand Benchmarking Results

  Why R?

• Sep 5, 2009: How to use Pharo/Squeak from the Command-line

  Along the way to measure the performance of a Smalltalk implementation for commodity multi-core systems, I tried to use Pharo as a more convenient development platform. Well, and I failed in the first attempt…