Jekyll2026-06-16T12:52:33+02:00https://stefan-marr.de/feed/index.xmlStefan-Marr.depersonal and research notes
SSW@ECOOP’26: On Compilation, GC, Debugging, Benchmarking and Meta-Compilation2026-06-16T09:47:00+02:002026-06-16T09:47:00+02:00https://stefan-marr.de/2026/06/ssw-at-ecoop-compilers-gc-debugging-meta-compilationAt this year’s ECOOP, the Institute for System Software will be attending with the almost complete team
and we’re going to present on a variety of topics. Come and talk to us at ICOOOLPS, MPLR, the Demo Track, DEBT, ECOOP Academy, and at the poster session!
Below, a brief overview and when the talks are scheduled.
Monday, June 29, 2pm: AOT Meta-Compilation of Dynamic Languages
Christoph A. is going to present initial ideas of how to approach ahead-of-time meta-compilation
for dynamic languages. While some dynamic languages can already be compiled fairly successfully ahead of time, we would love to get the compiler for free, ideally from not much more than having to implement the interpreter.
Dynamically-typed languages rely on just-in-time (JIT) compilation for execution performance. Meta-compilation systems such as GraalVM's Truffle language implementation framework have reduced the effort needed of enabling JIT compilation to implementing an interpreter. But dynamic languages are increasingly used in scenarios where ahead-of-time (AOT) compilation would be preferable, for instance, for faster startup or to avoid the memory cost of JIT compilation. Therefore, we plan to extend meta-compilation systems to also support AOT compilation.
For successful AOT compilation of dynamically-typed languages, we need an extensive and robust type analysis. In this position paper, we present first ideas for a framework with an extensible core analysis that will enable us to extract type flow semantics from an interpreter implemented in a meta-compilation system.
To achieve the precision needed for fast machine code, we will need to include heuristic analyses. For this, we envision a plugin system that allows us to integrate various different heuristics into a singular unified analysis. Combining analyses in this way can produce results that are better than the sum of their parts.
While this is a very ambitious goal, given the complexity of compiling dynamic languages, we believe we can achieve better-than-interpreted performance for programs with reasonable behavior. Furthermore, to support the full language semantics we keep a general interpreter as a fallback.
Monday, June 29, 3pm: Pedagogical Annotations in a Debugger
Markus is going to present work on teaching programming by using debugging techniques.
Specifically, he will look at enriching program visualizations with explanations and interactive questions.
Educators frequently use trace-based debuggers for live classroom demonstrations. Yet, if a student’s attention drops during class, they have to fall back to watching recordings (providing a passive, non-interactive experience) or replaying the debugging session at home (lacking the instructor’s pedagogical context and verbal explanations). We introduce Pedagogical Execution Traces (PETs), a concept that enriches execution traces with explanations, highlights and interactive questions. In this work-in-progress idea paper, we present the conceptual foundation of PETs as interactive learning artifacts, showing their applicability within JavaWiz, an educational trace-based graphical debugger. We explore PET authoring design goals and outline ongoing work regarding collaborative debugging scenarios and leveraging Large Language Models (LLMs) for trace annotation.
Tuesday, June 30, 4pm: Supporting Different GCs in AOT-compiled Binaries
Thomas will present a fairly simple but effective approach to enable a single AOT-compiled binary for a Java program
to use different GCs. At the moment it supports HotSpot's G1 and a more basic generational GC.
Some language implementations combine garbage collection with ahead-of-time compilation to produce self-contained executables for managed-language programs. In these systems, one can typically choose a garbage collector (GC) only at build time. To use another GC, e.g., for better performance,one needs to build another executable.
In this paper, we present an approach for supporting multiple GCs in the same self-contained executable using unified barriers, object layout, object header, and dynamic dispatch. This enables developers to select a GC at run time. Additionally, isolates, i.e., lightweight virtual machine instances with separate collected heaps but within the same process, can now use different GCs alongside each other.
We evaluate our approach in GraalVM Native Image, supporting the Garbage First (G1) and the Serial GC in the same executable. Our evaluation on the DaCapo Chopin and Renaissance benchmarks shows that G1 has on average no performance change (min. −9 %, max. 14 %). Serial GC shows a peak performance regression of 11 % (min. −10 %, max. 33 %). We believe the simplicity of the approach and that one can now choose the GC at run time and on a per isolate basis make this overhead acceptable.
Tuesday, June 30, 5pm: Reducing Binary Size with Static Heuristics
Christoph P. will present his evaluation of how far one can get with basic static compiler heuristics,
when it comes to reducing the size of AOT-compiled Java binaries, while minimizing the impact on performance.
To compile, or not to compile, that is the question: When ’tis nobler to optimize for performance. Modern compilers have many different optimizations and optimization goals. A common one is to balance between peak performance and startup time. A new ahead-of-time compiled native executable that embeds a managed runtime tries to offer both, while solidifying the notion that everything should be compiled. However, the cost of an enlarged binary size raises the question whether it is beneficial to compile everything.
In this paper, we evaluate static heuristics from classical AOT compilers as well as other techniques based on our own observations. Our goal is to identify heuristics that work in a compilation-first environment and that allow us to reduce binary size while maintaining peak performance.
We compare the different policies in a closed-world hybrid execution system for Java, based on GraalVM Native Image, on a set of 5 DaCapo and 13 Renaissance benchmarks. We find that with the best combination of heuristics we can reduce binary size by 20% while slowing down average performance by only 4%, but avoiding the need for any run-time feedback or complex machine-learning-based approaches. The most promising combination for production use combines heuristics based on early returns, estimated CPU cycles, number of parameters, and whether a method is a static initializer.
Date TBC: A Debugger for Teaching Threads and Locks
Melissa is going to present a visual debugger designed for teaching threads and locks in Java.
Threads, locks, and their interaction can feel hard to explain, though, with the right representation
in a debugging tool, their dynamic interactions can become more understandable.
Programming novices often face difficulties understanding how multi-threading works. Visual debuggers such as JavaWiz can support beginners by providing dynamic visualizations of a program’s behavior, however, they usually only work for single-threaded programs. This paper presents ThreadViz, an extension of JavaWiz to support visualizing multi-threaded Java programs.
In ThreadViz, thread information is collected for visualization by using the Truffle Debug API.
Instead of real concurrency, threads are executed stepwise, allowing the user to determine the order of execution and preventing any unpredictable behavior. In the user interface, a unique color is associated with each thread to illustrate the effects of different synchronization mechanisms such as locking and indicate thread state changes. To conclude, application examples are presented to highlight the tool's capabilities.
Last but not least, I'll give a lecture on benchmarking. Modern hard- and software makes that quite a bit more complicated than what we would like it to be and I will show a bit how we approach it in practice.
Full Abstract
Modern systems are great! In many ways, they adapt to our software, and optimize it, despite us not really knowing what we are doing, and to a degree that would have been considered magic just a few decades ago.
Though, once we develop our own research ideas on top of these systems and want to make any argument about performance, all this “magic” makes it hard to understand what measurements mean. Worse yet, making sensible performance claims means we have to understand a good chunk of it. Is this benchmark 20% faster because of what I did, or did the CPU increase the clock frequency for the new but not for the old code? Did the JVM just trigger garbage collection? Did the just-in-time compiler slow down my code? What do you mean, “efficiency core”?
In this lecture, we will have a brief look at why benchmarking on modern systems is hard and what can go wrong. Then we will discuss a range of different research scenarios to get a better feeling of what we may need for our work. Since much of this work may involve gradually building up our own systems, we will also look at what it takes to build them based on reliable feedback.
In the second part, we will look at how we can turn the often chaotic scientific process, with all its trials and errors, into a “scientific engineering process” that enables us to try and try again. I’ll suggest a process that allows us to use the same setup that we use for developing our system to not just understand its performance, but also use it to run the experiments we may want for a scientific paper. I’ll demonstrate how to go from daily pull requests with continuous performance tracking to generating plots and statistics for direct inclusion in LaTeX.
]]>Programming Language Implementation: In Theory, We Understand. In Practice, We Wish We Would.2026-02-02T15:15:10+01:002026-02-02T15:15:10+01:00https://stefan-marr.de/2026/02/programming-language-implementation-in-theory-we-understand-in-practice-we-wish-we-wouldIt’s February! This means I have been at the JKU for four months.
Four months with teaching Compiler Construction and System Software,
lots of new responsibilities (most notably signing off on telephone bills and coffee orders…), many new colleagues, and new things to learn for me, not least because of the very motivated students and PhD students here.
And when I say motivated, yes, I am very surprised. While the attendance of my 8:30am Compiler Construction lectures was declining throughout the term as expected, the students absolutely aced their exam.
I suspect I will have to make it harder next year. Much harder… hmmm 🤔
Much of the good results can likely be attributed to the very extensive exercise sessions run by my colleagues throughout the semester.
At this point, I have to send a big thank you to everyone from the Institute for System Software, past and present.
It’s great to be part of such a team! You made my start very easy, and, well, it now gives me the time to think about my inaugural lecture.
What’s an inaugural lecture?
I have been in academia for almost two decades, but I have to admit, I don’t really remember being at an inaugural lecture.
According to Wikipedia, in the Germanic tradition an inaugural lecture (Antrittsvorlesung) is these days something of a celebration.
It’s a festive occasion for a new professor to present their field to a wider audience, possibly also presenting their research vision.
At the JKU, it indeed seems to be planned as a festive occasion, too.
On March 9th, 2026, starting at 4pm Prof. Bernhard Aichernig and I will give our Antrittsvorlesungen,
and you are cordially invited to attend.
Bernhard will start out by looking at the formal side of things, making the connection between proving correctness, testing systems in the context of where they are used, and learning models from observable data. My talk will narrow in on language implementations, but also look at how formal correctness is helping us there.
Unfortunately, provably-correct systems still elude us for many practical languages.
Even worse, we are at a point where we rarely understand what’s going on in enough detail to improve performance or perhaps fix certain rare bugs.
In Theory, We Understand. In Practice, We Wish We Would
Here’s the abstract of my talk:
Our world runs on software, but we understand it less and less. In practice, the complexity of modern
systems drains your phone’s battery faster, increases the cost of hosting applications, and consumes
unnecessary resources, for instance, in AI systems. All because we do not truly understand our
systems any longer. Still, at a basic level, we can fully understand how computers work, from
transistors to processors, machine language, all the way up to high-level programming languages.
The convenience of contemporary programming languages is however bought with complexity. Over
the last two decades, I admit, I added to that complexity. In the next two decades, I hope we can learn
to build programming languages in ways that we can prove to be correct, enable us to generate their
implementations automatically, and let systems select optimizations in a way that we can still
understand the implications for software running on top of it.
You may now wonder where to go from here. And that’s a very good question.
I have another month to figure that out, perhaps more… 😅
So, maybe see you in March?
Until then, suggestions, questions, and complaints, as usual on
Mastodon,
BlueSky, and
Twitter.
]]>Python, Is It Being Killed by Incremental Improvements?2026-01-20T14:02:15+01:002026-01-20T14:02:15+01:00https://stefan-marr.de/2026/01/python-killed-by-incremental-improvements-questionmarkOver the past years, two major players invested into the future of Python. Microsoft’s Faster CPython team has pushed ahead with impressive performance improvements for the CPython interpreter, which has gotten at least 2x faster since Python 3.9. They also have a baseline JIT compiler for CPython, too. At the same time, Meta is worked hard on making free-threaded Python a reality to bring classic shared-memory multithreading to Python, without being limited by the still standard Global Interpreter Lock, which prevents true parallelism.
Both projects deliver major improvements to Python, and the wider ecosystem. So, it’s all great, or is it?
In my talk talk on this topic at SPLASH, which is now online, I discussed some of the aspects the Python core developers and wider community seem to not regard with the same urgency as I would hope for. Concurrency makes me scared, and I strongly believe the Python ecosystem should be scared, too, or look forward to the 2030s being “Python’s Decade of Concurrency Bugs”.
In the talk, I started out reviewing some of the changes in observable language semantics between Python 3.9 and today and discuss their implications.
I previously discussed the changes around the global interpreter lock in my post on the changing “guarantees”. In the talk, I also use the example from a real bug report, to illustrate the semantic changes:
request_id = self._next_id
self._next_id += 1
It looks simple, but reveals quite profound differences between Python versions.
Since I have some old ideas lying around, I also propose a way forward.
In practice though, this isn’t a small well-defined engineering or research project.
So, I hope I can inspire some of you to follow me down the rabbit hole of Python’s free-threaded future.
Questions, pointers, and suggestions are always welcome, for instance, on
Mastodon,
BlueSky, or
Twitter.
Slides
]]>Benchmarking Language Implementations: Am I doing it right? Get Early Feedback!2025-11-17T19:00:24+01:002025-11-17T19:00:24+01:00https://stefan-marr.de/2025/11/experimental-setupsModern CPUs, operating systems, and software in general do lots of smart and hard-to-track optimizations, leading to warmup behavior, cache effects, profile pollution and other unexpected interactions.
For us engineers and scientists, whether in industry or academia, this unfortunately means that we may not fully understand the system on top of which we are trying to measure the performance impact of, for instance, an optimization, a new feature, a data structure, or even a bug fix.
Many of us even treat the hardware and software we run on top of as black boxes, relying on the scientific method to give us a good degree of confidence in the understanding of the performance results we are seeing.
Unfortunately, with the complexity of today’s systems, we can easily miss important confounding variables.
Did we account, e.g., for CPU frequency scaling, garbage collection, JIT compilation, and network latency correctly?
If not, this can lead us down the wrong, and possibly time-consuming path of implementing experiments that do not yield the results we are hoping for, or our experiments are too specific to allow us to draw general conclusions.
So, what’s the solution? What could a PhD student or industrial researcher do when planning for the next large project?
How about getting early feedback?
Get Early Feedback at a Language Implementation Workshop!
At the MoreVMs and VMIL workshop series, we introduced a new category of submissions last year:
Experimental Setups.
We solicited extended abstracts that focus on the experiments themselves before an implementation is completed.
This way, the experimental setup can receive feedback and guidance to improve the chances that the experiments lead to the desired outcomes. With early feedback, we can avoid common traps and pitfalls, share best practices, and deeper understanding of the systems we are using.
With the complexity of today’s systems, one person, or even one group, is not likely to think of all the issues that may be relevant. Instead of encountering these issues only in the review process after all experiments are done, we can share knowledge and ideas ahead of time, and hopefully improve the science!
So, if you think you may benefit from such feedback, please consider submitting an extended abstract describing your experimental goals and methodology. No results needed!
For sampling profilers, the so-called observer effect gets in the way: when we profile a program, the profiling itself can change the program’s performance behavior. This means we can’t simply increase the sampling frequency to get a more accurate profile, because the sampling causes inaccuracies.
So, how could we possibly know whether a profile correctly reflects an execution?
We could try to look at the code and estimate how long each bit takes, and then painstakingly compute what an accurate profile would be. Unfortunately, with the complexity of today’s processors and language runtimes, this would require a cycle-accurate simulator that needs to model everything, from the processor’s pipeline, over the cache hierarchy, to memory and storage.
While there are simulators that do this kind of thing, they are generally too slow to simulate a full JVM with JIT compilation for any interesting program within a practical amount of time.
This means that simulation is currently impractical, and it is impractical to determine what a ground truth would be.
So, what other approaches might there be to determine whether a profile is accurate?
In 2010, Mytkowicz et al. already checked whether Java profilers were actionable
by inserting computations at the Java bytecode level.
On today’s VMs, that’s unfortunately an approach that changes performance in fairly unpredictable ways, because it interacts with the compiler optimizations.
However, the idea to check whether a profiler accurately reflects the slowdown of a program is sound.
For example, an inaccurate profiler is less likely to correctly identify a change in the distribution of where a program spends its time.
Similarly, if we change the overall amount of time a program takes, without changing the distribution of where time is spent,
it may attribute run time to the wrong parts of a program.
We can detect both of these issues by accurately slowing down a program.
And, as you might know from the previous post,
we are able to slow down programs fairly accurately.
Figure 1 illustrates the idea with a stacked bar chart for a hypothetical distribution of run-time over three methods. This distribution should remain identical, independent of a slowdown observed by the program.
So, there’s a linear relation between the absolute time measured and a constant relation between the percentage of time per method, depending on the slowdown.
Figure 1: A stacked bar chart for a hypothetical program execution, showing the absolute time per method. A profiler should see the linear increase in run time taken by each method, but still report the same percentage of run time taken. If a profiler reports something else, we have found an inaccuracy.
With this slowdown approach, we can detect whether the profiler is accurate with respect to the predicted time increase.
I’ll leave all the technical details to the paper.
We can also slow down individual basic blocks accurately to make a particular method take more time.
As it turns out, this is a good litmus test for the accuracy of profilers,
and we find a number of examples where they fail to attribute the run time correctly.
Figure 2 shows an example for the Havlak benchmark.
The bar charts show how much change the four profilers detect after we slowed down
Vector.hasSome to the level indicated by the red dashed line.
In this particular example, async-profiler detects the change accurately.
JFR is probably within the margin of error.
However, JProfiler and YourKit are completely off. JProfiler likely can’t deal with inlining and attributes the change to the forEach method that calls hasSome.
YourKit does not seem to see the change at all.
Figure 2: Bar chart with the change in run time between the baseline and slowed-down version, for the top 5 methods of the Havlak benchmark.
The red dashed line indicates the expected change for the Vector.hasSome method. Only async-profiler and JFR come close to the expectation.
With this slowdown-based approach, we finally have a way to see how accurate sampling profilers are by approximating the ground truth profile. Since we can’t measure the ground truth directly, we found a way to sidestep a fundamental problem and found a reasonably practical solution.
The paper details how we implement our divining approach, i.e., how we slow down programs accurately.
It also has all the methodological details, research questions, benchmarking setup, and lots more numbers, especially in the appendix. So, please give it a read, and let us know what you think.
If you happen to attend the SPLASH conference,
Humphrey is presenting our work today and on Saturday.
Questions, pointers, and suggestions are always welcome, for instance, on
Mastodon,
BlueSky, or
Twitter.
Optimizing performance on top of modern runtime systems with just-in-time (JIT) compilation is a challenge for a wide range of applications from browser-based applications on mobile devices to large-scale server applications. Developers often rely on sampling-based profilers to understand where their code spends its time. Unfortunately, sampling of JIT-compiled programs can give inaccurate and sometimes unreliable results.
To assess accuracy of such profilers, we would ideally want to compare their results to a known ground truth. With the complexity of today’s software and hardware stacks, such ground truth is unfortunately not available. Instead, we propose a novel technique to approximate a ground truth by accurately slowing down a Java program at the machine-code level, preserving its optimization and compilation decisions as well as its execution behavior on modern CPUs.
Our experiments demonstrate that we can slow down benchmarks by a specific amount, which is a challenge because of the optimizations in modern CPUs, and we verified with hardware profiling that on a basic-block level, the slowdown is accurate for blocks that dominate the execution. With the benchmarks slowed down to specific speeds, we confirmed that async-profiler, JFR, JProfiler, and YourKit maintain original performance behavior and assign the same percentage of run time to methods. Additionally, we identify cases of inaccuracy caused by missing debug information, which prevents the correct identification of the relevant source code. Finally, we tested the accuracy of sampling profilers by approximating the ground truth by the slowing down of specific basic blocks and found large differences in accuracy between the profilers.
We believe, our slowdown-based approach is the first practical methodology to assess the accuracy of sampling profilers for JIT-compiling systems and will enable further work to improve the accuracy of profilers.
Divining Profiler Accuracy: An Approach to Approximate Profiler Accuracy Through Machine Code-Level Slowdown
H. Burchell,
S. Marr;
Proceedings of the ACM on Programming Languages,
OOPSLA'25,
ACM,
2025.
@article{Burchell:2025:Divining,
abstract = {Optimizing performance on top of modern runtime systems with just-in-time (JIT) compilation is a challenge for a wide range of applications from browser-based applications on mobile devices to large-scale server applications. Developers often rely on sampling-based profilers to understand where their code spends its time. Unfortunately, sampling of JIT-compiled programs can give inaccurate and sometimes unreliable results.
To assess accuracy of such profilers, we would ideally want to compare their results to a known ground truth. With the complexity of today's software and hardware stacks, such ground truth is unfortunately not available. Instead, we propose a novel technique to approximate a ground truth by accurately slowing down a Java program at the machine-code level, preserving its optimization and compilation decisions as well as its execution behavior on modern CPUs.
Our experiments demonstrate that we can slow down benchmarks by a specific amount, which is a challenge because of the optimizations in modern CPUs, and we verified with hardware profiling that on a basic-block level, the slowdown is accurate for blocks that dominate the execution. With the benchmarks slowed down to specific speeds, we confirmed that async-profiler, JFR, JProfiler, and YourKit maintain original performance behavior and assign the same percentage of run time to methods. Additionally, we identify cases of inaccuracy caused by missing debug information, which prevents the correct identification of the relevant source code. Finally, we tested the accuracy of sampling profilers by approximating the ground truth by the slowing down of specific basic blocks and found large differences in accuracy between the profilers.
We believe, our slowdown-based approach is the first practical methodology to assess the accuracy of sampling profilers for JIT-compiling systems and will enable further work to improve the accuracy of profilers.},
acceptancerate = {0.356},
appendix = {https://doi.org/10.5281/zenodo.16911348},
articleno = {402},
author = {Burchell, Humphrey and Marr, Stefan},
blog = {https://stefan-marr.de/2025/10/can-we-know-whether-a-profiler-is-accurate/},
doi = {10.1145/3763180},
issn = {2475-1421},
journal = {Proceedings of the ACM on Programming Languages},
keywords = {Accuracy GroundTruth Java MeMyPublication Profiling Sampling myown},
month = oct,
number = {OOPSLAB25},
numpages = {32},
pdf = {https://stefan-marr.de/downloads/oopsla25-burchell-marr-divining-profiler-accuracy.pdf},
publisher = {{ACM}},
series = {OOPSLA'25},
title = {{Divining Profiler Accuracy: An Approach to Approximate Profiler Accuracy Through Machine Code-Level Slowdown}},
year = {2025},
month_numeric = {10}
}
]]>First Day: A New Chapter at the JKU2025-10-01T08:02:19+02:002025-10-01T08:02:19+02:00https://stefan-marr.de/2025/10/first-day-at-jkuIt’s Wednesday. Is this important? It’s my first day in a new position. So, perhaps the real question is: what’s going to be important to me from now on?
Let’s get the titles out of the way first:
Today is my first day as Universitätsprofessor. That’s a full professor, chair, W3 Professor, gewoon hoogleraar, or similar. Yeah, there are lots of different names in different countries. It’s also my first day as the head of the Institute for System Software.
The term institute is used here for something that’s a research group in many other places.
This means I have the opportunity to work with a number of very smart people to offer university courses in the field of programming languages, compilers, and more broadly system software.
It also means I am asked to advise, mentor, and support others in their research journey, from taking their very first steps, up to becoming their own independent academics, and professors in their own right.
To me, this sounds fun. I am asked to help people learn, pursue knowledge, and develop their skills. Something I not only enjoy, but also find important to prepare the next generation to tackle the problems of our time.
However, this also means I reached the end of a journey.
That’s it. I am a full professor now, and I have convinced enough people that I am not entirely terrible at this job. Or so we all hope…
At this point, I already have to thank all the people at the JKU for the very warm welcome I received over the last few weeks. Particularly, thank you Peter, Herbert, Markus, and Karin, for all the support to get me started here! Similarly, I wouldn’t be here without my dear colleagues and mentors at Kent and in the wider programming language research community. You know who you are, I hope.
What Now?
With the new job and responsibilities, I need to think about what’s now important to me.
What follows isn’t a detailed plan.
I had already been asked to formulate one of those, and I’ll continue to work on realizing it.
Instead,
I wanted to think here a bit broader.
Teaching: Advocate for Fundamentals
Let’s start with teaching, since my first lectures will already be next week.
Our institute teaches various courses, including software development, compiler construction, advanced compiler construction, system software, dynamic compilation and run-time optimization, and principles of programming languages.
My impression from early discussions with colleagues is that I will need to work on making sure that we can keep teaching these fundamental topics in the future. While there seems to be a very strong push for AI everything, I remain to be convinced that this means that the fundamentals are any less important. On the contrary, it feels that we need to keep reminding people of classic techniques that are guaranteed to work, are correct, and efficient. So, when it comes to teaching, I think an important part of my job will be advocating for the fundamentals.
Of course, looking at the material I’ll teach this term on compiler construction and system software, perhaps I can adapt it in future years. Currently, 6 out of 13 compiler construction lectures are on parsing. This makes me want to work out what the most useful learning outcomes for such a course should be today.
Research: Take Risks and Pursue Problems Too Hard for Industry
Some people seem to advocate for exploring new things and expanding one’s horizon when reaching this career level.
Indeed, I have the chance to take risks, explore new research topics and communities, and ways of working.
If there’s a single tag line for the work I have in mind, it might be: improve language implementations to better enable old and new kinds of applications.
After all, I like to explore ideas that enable developers to make better use of computing systems.
This will take new ways of looking at problems.
For instance, with few exceptions, I have been shying away from very formal work in the past.
Though, a while ago I started dreaming of defining a new kind of high-level memory model, for which we may need a more formal approach in addition to building working prototypes.
Looking at today’s memory models, they seem too low-level for dynamic languages such as Python and Ruby. I already gave a few talks
about the background of this work and will also give one at SPLASH.
This will be a huge project, and a risky one. Not least because it’s unclear whether the language communities care enough about the issue until they start suffering from not having a memory model more notably.
And then there is interpreter performance, a topic I have been working on for a long time already.
Since I am now in a group with a long history in the area of compilers,
I would like to double down on generating fast interpreters.
Interpreters, the way we build them today, have a lot of headroom in terms of performance.
The classic ones, implemented in C/C++, and even more so, the ones on top of meta-compilation systems.
The work of Haoran Xu suggests that we can do much better.
Unfortunately, it’s a really hard problem, for various reasons.
Something that doesn’t fit into the short and mid-term priorities of most companies.
But we can chip away at it slowly and steadily, benefiting lots of programming languages in the process.
I’ll stop here for now. Seems like I do need to get on with the actual job…
somewhere in Science Park 3.
I am looking forward to starting to work with all my new colleagues at the JKU and seeing which new collaborations and cooperations we can begin.
If you’re a student and interested in a project, please see the Open Project’s page, where I will post more concrete project ideas in the future.
]]>How to Slow Down a Program? And Why it Can Be Useful.2025-08-27T12:20:04+02:002025-08-27T12:20:04+02:00https://stefan-marr.de/2025/08/how-to-slow-down-a-programMost research on programming language performance asks a variation of a single question: how can we make some specific program faster?
Sometimes we may even investigate how we can use less memory.
This means a lot of research focuses solely on reducing the amount of resources needed to achieve some computational goal.
So, why on earth might we be interested in slowing down programs then?
Slowing Down Programs is Surprisingly Useful!
Making programs slower can be useful to find race conditions,
to simulate speedups, and to assess how accurate profilers are.
To detect race conditions,
we may want to use an approach similar to fuzzing.
Instead of exploring a program’s implementation
by varying its input,
we can explore different instruction interleavings, thread or event schedules,
by slowing down program parts to change timings.
This approach allows us to identify concurrency bugs
and is used by CHESS, WAFFLE, and NACD.
The Coz profiler is an example of how slowing down programs can be used to simulate speedup.
With Coz, we can estimate whether an optimization is beneficial
before implementing it.
Coz simulates it by slowing down all other program parts.
The part we think might be optimizable stays at the same speed
it was before, but is now virtually sped up, which allows us to see
whether it gives enough of a benefit to justify a perhaps lengthy optimization project.
And, as mentioned before, we can also use it to assess how accurate profilers are.
Though, I’ll leave this for the next blog posts. :)
The current approaches to slowing down programs for these use cases are
rather coarse-grained though. Race detection often adapts the scheduler or uses, for example, APIs such as Thread.sleep().
Similarly, Coz pauses the execution of the other threads.
Work on measuring whether profilers give actionable results,
inserts bytecodes into Java programs to compute Fibonacci numbers.
By using more fine-grained slowdowns,
we think we could make race detection, speedup estimation, and profiler accuracy assessments more precise. Thus, we looked into inserting slowdown instructions into basic blocks.
Which x86 Instructions Allow us to Consistently Slow Down Basic Blocks?
Let’s assume we run on some x86 processor, and we are looking at programs
from the perspective of processors.
When running a benchmark like Towers,
the OpenJDK’s HotSpot JVM may compile it to x86 instructions like this:
This is one of the basic blocks produced by HotSpot’s C2 compiler.
For our purposes, it suffices to see that there are some memory accesses
with the mov instructions, and we end up checking whether the edx register
contains the value 1. If that’s not the case, we jump to Block 55.
Otherwise, execution continues in the next basic block.
A key property of a basic block is that there’s no control flow inside of it,
which means once it starts executing, all of its instructions will execute.
Though, how can we slow it down?
x86 has many many different instructions one could try to insert into the block,
which each will probably consume CPU cycles.
However, modern CPUs try to execute as many instructions
as possible at the same time using out-of-order execution.
This means, instructions in our basic block
that do not directly depend on each other
might be executed at the same time.
For instance, the first three mov instructions access neither the same register
nor memory location. This means the order in which they are executed here does not matter.
Though, which optimizations CPUs apply depends on the program and the specific CPU generation,
or rather microarchitecture.
To find suitable instructions to slow down basic blocks,
we experimented only on an Intel Core i5-10600 CPU,
which has the Comet Lake-S microarchitecture.
On other microarchitectures, things can be very different.
For the slowdown that we want,
we can use nop or mov regX, regX instructions on Comet Lake-S.
This mov would move the value from register X to itself, so basically does nothing.
These two instructions give us a slowdown
that is small enough to slow down most blocks accurately to a desired target speed,
and the slowdown seems to affect only the specific block it is meant for.
Our basic block from earlier would then perhaps end up with nop instructions
interleaved after each instruction.
In practice, the number of instructions we need to insert
depends on how much time a basic block takes in the program.
Though, for illustration, it might look like this:
We tried six different candidates, including a push-pop sequence, to get a better impression
of how Comet Lake-S deals with them.
For more details of how and what we tried, please have a look at our short paper below, which we will present at the VMIL workshop.
When inserting these instructions into basic blocks,
so that each individual basic block
takes about twice as much time as before,
we end up with a program that indeed is overall twice as slow, as one would hope.
Even better, when we look at the Towers benchmark with the async-profiler for HotSpot, and compare the proportions of run time it
attributes to each method, the slowed-down and the normal version
match almost perfectly, as illustrated below.
The same is not true for the other candidates we looked at.
Figure 1: A scatter plot per slowdown instruction with the median run-time percentage for the top six Java methods of Towers. The X=Y diagonal indicates that a method’s run‐time percentage remains the same with and without slowdown.
The paper has a few more details, including a more detailed analysis
of the slowdown each candidate introduces,
how precise the slowdown is for all basic blocks
in the benchmark, and whether it makes a difference when we put the slowdown
all at the beginning, interleaved, or at the end.
Of course, this work is merely a stepping stone to more interesting things,
which I will look at in a bit more detail in the next post.
Until then, the paper is linked below, and questions, pointers, and suggestions are welcome on
Mastodon,
BlueSky, or
Twitter.
Slowing down programs has surprisingly many use cases: it helps finding race conditions, enables speedup estimation, and allows us to assess a profiler’s accuracy. Yet, slowing down a program is complicated because today’s CPUs and runtime systems can optimize execution on the fly, making it challenging to preserve a program’s performance behavior to avoid introducing bias.
We evaluate six x86 instruction candidates for controlled and fine-grained slowdown including NOP, MOV, and PAUSE. We tested each candidate’s ability to achieve an overhead of 100%, to maintain the profiler-observable performance behavior, and whether slowdown placement within basic blocks influences results. On an Intel Core i5-10600, our experiments suggest that only NOP and MOV instructions are suitable. We believe these experiments can guide future research on advanced developer tooling that utilizes fine-granular slowdown at the machine-code level.
Evaluating Candidate Instructions for Reliable Program Slowdown at the Compiler Level: Towards Supporting Fine-Grained Slowdown for Advanced Developer Tooling
H. Burchell,
S. Marr;
In Proceedings of the 17th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages,
VMIL'25,
p. 8,
ACM,
2025.
@inproceedings{Burchell:2025:SlowCandidates,
abstract = {Slowing down programs has surprisingly many use cases: it helps finding race conditions, enables speedup estimation, and allows us to assess a profiler's accuracy. Yet, slowing down a program is complicated because today's CPUs and runtime systems can optimize execution on the fly, making it challenging to preserve a program's performance behavior to avoid introducing bias.
We evaluate six x86 instruction candidates for controlled and fine-grained slowdown including NOP, MOV, and PAUSE. We tested each candidate’s ability to achieve an overhead of 100%, to maintain the profiler-observable performance behavior, and whether slowdown placement within basic blocks influences results. On an Intel Core i5-10600, our experiments suggest that only NOP and MOV instructions are suitable. We believe these experiments can guide future research on advanced developer tooling that utilizes fine-granular slowdown at the machine-code level.},
author = {Burchell, Humphrey and Marr, Stefan},
blog = {https://stefan-marr.de/2025/08/how-to-slow-down-a-program/},
booktitle = {Proceedings of the 17th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Languages},
doi = {10.1145/3759548.3763374},
isbn = {979-8-4007-2164-9/2025/10},
keywords = {Benchmarking HotSpot ISA Instructions Java MeMyPublication assembly evaluation myown slowdown x86},
location = {Singapore},
month = oct,
pages = {8},
pdf = {https://stefan-marr.de/downloads/vmil25-burchell-marr-evaluating-candidate-instructions-for-reliable-program-slowdown-at-the-compiler-level.pdf},
publisher = {{ACM}},
series = {VMIL'25},
title = {{Evaluating Candidate Instructions for Reliable Program Slowdown at the Compiler Level: Towards Supporting Fine-Grained Slowdown for Advanced Developer Tooling}},
year = {2025},
month_numeric = {10}
}
]]>It’s Thursday, and My Last* Day at Kent2025-07-31T10:54:08+02:002025-07-31T10:54:08+02:00https://stefan-marr.de/2025/07/last-day-at-kentToday is the 31st of July 2025, and from tomorrow on I’ll be “between jobs”, or as Gen Z allegedly calls it, on a micro-retirement.
When I first came to Kent for my interview, I was thinking, I’ll do this one for practice.
I still had more than 2 years left on a research grant we just got, which promised to be lots of fun, but academic jobs for PL systems people are rare, even rarer these days.
But then I got the call from Richard Jones, offering me the position, and I never regretted taking him up on it.
Kent’s School of Computing was just growing its Programming Languages and Systems (PLAS) group and Richard, Simon Thompson, Andy King, Peter Rodgers, and many others at the School did a remarkable job in creating an environment and community that was truly supportive of young academics taking their first steps in a permanent academic post. Be it about wrestling with teaching duties, papers, reviews, reviewers, and of course grant writing. PLAS and the School of Computing was the right place for me.
Of course, many things changed since my start in October 2017. Perhaps most notably, Computing is now in the Kennedy building, a very nice space. But there was also that moment, where we, the young ones, became the “senior” ones. Mark, Laura, and Dominic grew well into their new roles and I can only hope that I passed on some of the extensive support I got, to the people who started after me.
There are many challenges ahead for my dear colleagues at Kent, but I hope, that enough of the spirit of support and community remains in the School, enabling PLAS and the next generation of academics to do great things.
Also a huge thank you to Kemi, Anna, and Janet for keeping the School afloat.
I’ll miss you all. Thanks for everything! And see you soon!
Most of PLAS in October 2023
* It’s a little more complicated than that, but for good reasons. Right, EPSRC? :)
]]>Instrumentation-based Profiling on JVMs is Broken!2024-09-17T11:08:10+02:002024-09-17T11:08:10+02:00https://stefan-marr.de/2024/09/instrumenation-based-profiling-on-jvms-is-brokenLast year, we looked at how well sampling profilers work on top of the JVM.
Unfortunately, they suffer from issues such as safepoint bias and may not correctly attribute observed run time to the correct methods because of the complexities introduced by inlining and other compiler optimizations.
After looking at sampling profilers, Humphrey started to investigate instrumentation-based profilers and found during his initial investigation that they were giving much more consistent numbers. Unfortunately, it became quickly clear that the state-of-the-art instrumentation-based profilers on the JVM also have major issues, which results in profiles that are not representative of production performance. Since profilers are supposed to help us identify performance issues, they fail at their one job.
When investigating them further, we found that they interact badly with inlining and other standard optimizations. Because the profilers we found instrument JVM bytecodes, they add a lot of extra code that compiler optimizations treat as any other application code.
While this does not strictly prevent optimizations such as inlining, the extra code interferes enough with the optimization that the observable behavior of a program with and without inlining is basically identical.
In practice, this means that instrumentation-based profilers on the JVM are easily portable, but they can’t effectively guide developers to the code that would benefit most from attention, which is their main purpose.
Profilers that do not capture production performance will misguide us!
While they can still identify the code that is activated most often, the interaction with optimizations means that developers see mostly unoptimized behavior.
With today’s highly optimizing compilers this is unfortunate, because we may end up optimizing code that the compiler normally would have optimized for us already, and we spend time on things that likely won’t make a difference in production.
In this arguably a little contrived example, we use some kind of framework,
for which we have actions that the framework applies for us.
This is probably a worst case for profilers that instrument bytecodes.
Here, the execute() methods would be identified as the most problematic
aspect. Though, they don’t do anything.
A just-in-time compiler like HotSpot’s C2,
would likely end up seeing a bimorphic call site to execute() and inline both methods.
And if the compiler heuristics are with us, it might even optimize out the empty loop
in the framework.
So, if we assume a sufficiently smart compiler, here our inefficient code,
that’s forced on us by a framework, is being taken care of by the compiler.
And a good profiler, would ideally guide us to the bubbleSortById(.) as being of interest.
Typically, we’d expect to get a good speedup here by switching to a more suitable
sort, especially since we implicitly assume there are many actions so that this code matters
in production.
To me this means, instrumentation-based profilers can only be a matter of last resort when sampling with its own flaws fails. They are just not useful enough as they are.
Can we do better than profilers that instrument bytecode?
At the time, Humphrey was quite in favor of instrumentation,
because it gives very consistent results.
So, he wanted to make the results of instrumentation-based profilers more realistic.
Inspired by the work of Basso et al.,
he built an instrumentation-based profiler into the Graal just-in-time compiler
that works more like classic instrumentation-based profilers for ahead-of-time-compiled language implementations.
Figure 1: Instrumentation-based profilers on the JVM typically insert instrumentation very early, before compilers optimize code.
In our profiler, instrumentation is inserted very late, to minimize interfering with optimizations.
Instead of inserting the instrumentation right when the bytecode is loaded,
for instance with an agent or some other form of bytecode rewriting,
we move the addition of instrumentation code to a much later part of the just-in-time compilation. Most importantly, we insert it only after inlining and most optimizations are performed.
To keep the prototype simple, we insert the probes right before it is turned into the lower level IR.
At this point, there are still a few optimizations to be performed, including instruction selection and register allocation. Though, in the grand scheme of things, these are minor.
How much better is it?
With his prototype, Humphrey managed to achieve not only much better performance than classic instrumentation-based profilers, but also minimize interference with optimizations.
For a rough idea of the overall performance impact of this approach,
let’s have a look at Figure 2:
Figure 12: Sampling-based profilers such as Async, Honest, JFR,
Perf, and YourKit (in sampling mode) have very low overhead, though suffer from safepoint bias
and only observe samples.
YourKit and JProfiler doing instrumentation introduce overhead of two orders of magnitudes
and lead to unrealistic results because of their impact on optimizations.
Bubo, our prototype, has much lower overhead, and does not interfere with optimizations.
With a few extra tricks briefly sketched in the paper, we get good attribution of where time is spent, even in the presence of inlining, reduce overhead, and benefit from the more precise results of instrumentation, because it does not have the same drawbacks of only occasionally obtaining data.
There’s one major open question though: what does a correct profile look like?
At the moment, we can’t assess whether our approach is correct.
Sampling profilers, as we saw last year, also do not agree on a single answer.
So, while we believe our approach is much better than classic instrumentation, we still need to find out how correct it is.
All results so far, and a few more technical details are in the paper linked below.
Questions, pointers, and suggestions are greatly appreciated
perhaps on
Mastodon or
Twitter @smarr.
Abstract
Profilers are crucial tools for identifying and improving application performance. However, for language implementations with just-in-time (JIT) compilation, e.g., for Java and JavaScript, instrumentation-based profilers can have significant overheads and report unrealistic results caused by the instrumentation.
In this paper, we examine state-of-the-art instrumentation-based profilers for Java to determine the realism of their results. We assess their overhead, the effect on compilation time, and the generated bytecode. We found that the profiler with the lowest overhead increased run time by 82x. Additionally, we investigate the realism of results by testing a profiler’s ability to detect whether inlining is enabled, which is an important compiler optimization. Our results document that instrumentation can alter program behavior so that performance observations are unrealistic, i.e., they do not reflect the performance of the uninstrumented program.
As a solution, we sketch late-compiler-phase-based instrumentation for just-in-time compilers, which gives us the precision of instrumentation-based profiling with an overhead that is multiple magnitudes lower than that of standard instrumentation-based profilers, with a median overhead of 23.3% (min. 1.4%, max. 464%). By inserting probes late in the compilation process, we avoid interfering with compiler optimizations, which yields more realistic results.
Towards Realistic Results for Instrumentation-Based Profilers for JIT-Compiled Systems
H. Burchell,
O. Larose,
S. Marr;
In Proceedings of the 21st ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes,
MPLR'24,
ACM,
2024.
@inproceedings{Burchell:2024:InstBased,
abstract = {Profilers are crucial tools for identifying and improving application performance. However, for language implementations with just-in-time (JIT) compilation, e.g., for Java and JavaScript, instrumentation-based profilers can have significant overheads and report unrealistic results caused by the instrumentation.
In this paper, we examine state-of-the-art instrumentation-based profilers for Java to determine the realism of their results. We assess their overhead, the effect on compilation time, and the generated bytecode. We found that the profiler with the lowest overhead increased run time by 82x. Additionally, we investigate the realism of results by testing a profiler’s ability to detect whether inlining is enabled, which is an important compiler optimization. Our results document that instrumentation can alter program behavior so that performance observations are unrealistic, i.e., they do not reflect the performance of the uninstrumented program.
As a solution, we sketch late-compiler-phase-based instrumentation for just-in-time compilers, which gives us the precision of instrumentation-based profiling with an overhead that is multiple magnitudes lower than that of standard instrumentation-based profilers, with a median overhead of 23.3% (min. 1.4%, max. 464%). By inserting probes late in the compilation process, we avoid interfering with compiler optimizations, which yields more realistic results.},
author = {Burchell, Humphrey and Larose, Octave and Marr, Stefan},
blog = {https://stefan-marr.de/2024/09/instrumenation-based-profiling-on-jvms-is-broken/},
booktitle = {Proceedings of the 21st ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes},
doi = {10.1145/3679007.3685058},
keywords = {Graal Instrumentation JVM Java MeMyPublication Optimization Profiler Profiling Sampling myown},
month = sep,
pdf = {https://stefan-marr.de/downloads/mplr24-burchell-et-al-towards-realistic-results-for-instrumentation-based-profilers-for-jit-compiled-systems.pdf},
publisher = {ACM},
series = {MPLR'24},
title = {{Towards Realistic Results for Instrumentation-Based Profilers for JIT-Compiled Systems}},
year = {2024},
month_numeric = {9}
}
]]>5 Reasons Why Box Plots are the Better Default Choice for Visualizing Performance2024-06-18T09:38:38+02:002024-06-18T09:38:38+02:00https://stefan-marr.de/2024/06/5-reasons-for-box-plots-as-defaultBox Plots, Or Better!
This post is motivated by discussions I have been having for, ehm, forever?
To encourage others to use good research practices and avoid bar charts, I’ll argue that people should use box plots as their go-to choice when presenting performance results. Of course, box plots aren’t a one-size-fits-all solution. However, I believe they should be the preferred choice for many standard situations. For some situations, more appropriate chart types should be chosen based on careful consideration.
Thus, box plots should be the default choice instead of the omnipresent bar chart. Or short: Box Plots, or Better!
When working on performance, I usually work with just-in-time compiling language runtimes, on which I would run various experiments that I want to compare.
For examples, check the papers of Humphrey, Octave, and Sophie (copies are here).
However, I believe the argument applies more generally beyond our own work.
Reason 1: Performance Measurements Are Samples from a Distribution
When we measure the performance of a system, we usually get a data point that has been influenced by many different factors.
This is independent of whether we measure wall-clock time, the number of executed instructions, or perhaps memory.
While we can control some factors and influence others, today’s systems are often too complex for us to fully understand them. For example, cache effects, thermal properties, as well as hard- and software interactions outside our control can change performance non-deterministically. In practice, we therefore often treat the system as a black box.11 I’d encourage people to dig deeper, but I’m aware that time does not always allow for it.
Treating it as a black box then of course requires us to repeat our experiments multiple times to be able to characterize the range of results that are to be expected. Statisticians would perhaps describe our measuring as “sampling a distribution”.
And this is the point where box plots come in. They are designed to be a convenient way to characterize distributions. Let’s assume we have an experiment A and B, and we have taken 50 measurements each. Figure 1 shows the results of our experiments as box plots.
Figure 1: Box plot comparing A and B, including annotations for the key elements of a box plot.
I annotated the box plot for A with some key elements, including the median, 25th, and 75th percentile. We also see the notion of an interquartile range, which tells us a bit about the shape of the result distribution and outliers, i.e., typically all measurements that are further from the 25th and 75th percentile than 1.5x the interquartile range.
With box plots, we have enough details to see that the two experiments behave differently in a number of ways.
The median lines tell us that A is usually faster than B. However, we also see that A is not always faster than B, because the results are further spread out. In the worst case, A takes 19 seconds, which is more than B’s worst case of 15 seconds. While the main half of all data points for both experiments don’t overlap, we see that a good chunk of A’s results still fall within what’s often not considered to be outliers, i.e., the range between the 75th percentile with 1.5x of the interquartile range added.
By looking at the figure and comparing these plots, I believe we can get a reasonable intuition of the performance tradeoffs of the two options.
Reason 3: Box Plots Give Enough Details
The above analysis of the results would not have been possible for instance with a classic bar chart
as shown in Figure 2.
Figure 2: Bar chart comparing A and B, showing the mean and the standard deviation as error bars.
Bar charts are often used to compare the performance of two or more systems or experiments.
However, they show only three values per bar, typically a chosen “measure of centrality”, and some form of “error”. Very common are here things like the arithmetic mean, geometric mean, harmonic mean, and perhaps the median. Each of these has different properties, and one has to carefully think about which one to use based on the type of data one is working with (or perhaps not). At this point one also still has to chose how to characterize measurement errors.
This means that bar charts are less standardized than box plots, and one has to be explicit about what is shown.
Figure 3: Bar chart comparing A and B, showing the median and 25th and 75th percentile.
To just give one example, Figure 3 is the same data but shows the median and the 25th and 75th percentile
instead of the mean and standard deviation.
Since we show different sets of statistics, our impression of the results somewhat changes.
Of course, this is the power of visualization and picking statistics.
We can draw attention to specific aspects of the data. Figure 3 would lead me to conclude that A is always better than B, while Figure 2 would make me wonder what the underlying data looks like
to understand how we got to the depicted standard deviation.
Compared to our box plot in Figure 1, the choice of statistics to show,
and the reduced number of details we see here can result in misleading others and ourselves.
Thus, I’d strongly argue that bar charts are neither a good default to represent data
during data analysis, nor when presenting the final insights in a paper. They show too few details, oversimplifying an often more complex story.
Reason 4: Box Plots Don’t Overwhelm With Details
Of course, we could also go in the other direction and choose a plot type
that shows much more detail.
Let’s start with Figure 4, which shows a violin plot. I selected here a version that
shows just the density distribution of our results. One could go and highlight specific statistics on it for clarity of course.
However, just looking at Figure 4, we get a more detailed look at how our measurements are distributed.
From this, we see very clearly that B’s results are grouped much tighter together, and at each end, i.e., at 9 and 15 seconds, there are outliers.
A on the other hand, is much more stretched out, though, a good chunk of the results are indeed roughly in the area indicated by the box plot previously. Though, what we see here also is that the area is wider and stretches from perhaps 8 to 15, only outside of which we likely have significantly fewer samples.
We did not see these details on Figure 1.
Figure 4: A violin plot to compare A and B showing the density distribution of the results.
For data analysis, this way of looking at the data is very helpful,
because it allows us to see the underlying distribution.
For reporting data in a paper, this might however be too detailed,
in the sense that it is not as easily interpretable visually and makes drawing
conclusions harder.
Figure 5: A combination of violin, box plot, and raw data. The mean is indicated as a red dot.
While not ideal for final reports, violin plots are useful during analysis.
Perhaps one wants to go even a step further and use a combination of violin and box plot
together with the raw data and the mean during analysis.
An example of this is shown in Figure 5.
While the plot is very busy and not suitable for a paper, I’d think,
it prevents us from jumping to conclusions based on data summaries.
If you’re analyzing your data in R, a package like ggstatsplot might be a good solution.
Reason 5: They Are Very Versatile
Box plots can be used for many different purposes, independent of the type of distribution of data one wants to visualize, for different types of experiments, and to represent experimental data, as well as data summaries.
Because box plots visualize selected “percentile statistics”, we can use them without having to adapt them for specific experiments or types of distributions. They are nonparametric, i.e., one does not have to select any parameters for specific input data. This is useful for performance evaluations, because we do not generally know what type of distribution we are dealing with, and samples are not generally independent, which makes the use of various other statistical tools more complicated.
Figure 6: Comparing experiments A, B, and C. Their data is drawn from different distributions, none of which are normal distributions. The density plot at the bottom characterizes the sample distributions more precisely.
Figure 6 shows box plots for three different distributions. Important here is that neither of these experiments gives normally distributed data. Nonetheless, we can use box plots to describe them more abstractly and see certain key details such as A being skewed to the left, B slightly less so but much more narrow, and C having outliers to the left,
with a small skew to the right.
So far, I have used examples where there was an experiment A and B, and perhaps C.
Though, often we may want to understand the relation of perhaps two variables.
This might be in the sense of scaling a computation over multiple processor cores. Figure 7 shows a box plot that visualizes data for such a hypothetical experiment.
Figure 7: Comparing A and B as they change for increasing values of a second variable from 1 to 20.
While one would often use line charts for such scaling experiments, box plots can be used here as well.
One can still see the rough shape of a line, but we do not lose sight of the distribution of our experimental data.
Arguably, Figure 7 is very busy though, and a line chart with a confidence interval or similar would look better
(Python, ggplot).
We can also see that box plots “scale” reasonably well themselves in the sense
that they work for data that is spread out as well as for data that is very closely grouped.
For example for B, we see the values at x-axis point 1 to be very narrowly together.
Similarly for A, we see at 20 that data is tightly grouped.
In either case, we still have the complete power of the box plot and can draw conclusions.
If we would now want to summarize these results, we can of course use box plots!
Figure 8: Summary of the data of Figure 7. A box plot of a box plot. In the upper half, it's a box plot over the medians. In the lower half, it's a box plot over all raw data.
Figure 8 shows a summary. The plot at the top uses the medians for each of the experiments
over the variable that went from 1 to 20. So, for A and B, we have 20 values each, and plot them as box plots.
Note, for this to be a valid statistic, technically the medians have to be derived from independent samples, so, you may need to consult your friendly neighborhood statistician.
In the bottom plot, I used the raw data of all experiments.
In a way this still “works”, and results in a very similar box plots in this case.
Though, here the meaning changes and of course whether you can do this with your data is something you need to ask your statistician about. I think, common wisdom in our field is to first normalize the data and then “bootstrap” it. This would give us bootstrapped medians etc. The median is then technically from a normal distribution of independent samples, and standard statistics are legal again.
Conclusion: Box Plots Answer Important Questions At A Glance. Use Box Plots, Or Better!
When it comes to writing academic papers, I do believe that box plots are a much better default choice for communicating performance results than bar charts are.
The key reasons for me are:
they are a concise representation of the result distribution
they allow a visual comparison of more than the most basic statistics
and thus, answer more questions than bar charts
but without making things too complicated
they are also more standardize, and thus, remain more readable when taken out of context
and they can be used sensibly for a wide range of use cases
So, for me box plots strike a good overall balance, which makes them a good standard choice for papers.
Though, as mentioned earlier, they are not a universally best choice either.
For data analysis, one would want more details, and for specific use cases
or types of data distributions, e.g., bi- or multi-modal distribution,
other types of plots are more suitable.
I can recommend this piece with many examples where other types of plots than box plots may be better choices.
For questions, comments, or suggestions, please find me on Twitter @smarr or Mastodon.
