Performance Results

Performance Results

This report was generated on 2023-06-15 14:44:48.

The results are usually normalized either to Java JDK8 C2 or Java JDK8 Int, depending on whether we are interested in best or interpreter performance.

For VMs with just-in-time compilation, we report results based on the measurements starting from iteration 100. While this does not guarantee that we measure “peak performance”, it gives us relatively stable results for most language implementations.

Overview

Just-in-Time Compiling Language Implementations

The following set of language implementations reaches the performance of Java on our set of benchmarks within a factor of 2 to 3 on average. To allow for a more detailed assessment of these implementations, we exclude slower ones from the following plot.

plot of chunk jit-overview
Interpreters

Looking at interpreter performance, the plot looks quite a bit different. Here we use Java JDK8 Int as baseline.

plot of chunk int-overview
First Iteration

Looking at interpreter performance, the plot looks quite a bit different. Here we use Java JDK8 Int as baseline.

plot of chunk first-overview
Java Performance

Interpreter plot of chunk java-overview-int

C2

plot of chunk java-overview-c2
JavaScript Performance

plot of chunk js-overviewplot of chunk js-overview

Python Performance
plot of chunk python-overview plot of chunk python-overview-int
Ruby Performance
plot of chunk ruby-overview
SOM Performance
plot of chunk som-overview
SOM Interpreters Performance
plot of chunk som-int-overview
SOM Optimizations Performance
plot of chunk som-opt-overview
Performance Overview Data

The following table contains the numerical representation of the results depicted above.

  Runtime Factor over Java
median_ratio min_ratio max_ratio
Java JDK17 C2 0.98 0.84 1.37
Java ?? C2 0.99 0.85 1.37
Java-Temurin-17-jit 0.99 0.85 1.35
Java-Temurin-8-jit 1.00 0.96 1.02
Java JDK8 C2 1.00 1.00 1.00
Java-Temurin-20-jit 1.01 0.78 1.36
GraalVM Native EE 22.3 1.36 0.53 1.90
TruffleSOM AST Hotspot EE base 1.98 0.88 3.99
TruffleRuby EE 22.3 Hotspot 2.13 0.91 4.23
Node.js 20.3 2.17 0.91 2.82
TruffleSOM BC Hotspot EE base 2.18 0.96 4.99
Bun 0.6.9 2.18 0.82 4.43
Graal.js EE 22.3 Hotspot 2.31 0.68 5.62
TruffleSOM AST Hotspot CE base 2.58 0.99 3.61
TruffleSOM BC Hotspot CE base 2.91 1.11 5.58
GraalPython EE 22.3 Hotspot 2.91 0.81 5.13
PySOM BC JIT 3.09 1.11 8.35
PySOM AST JIT 4.31 0.99 19.86
PyPy 3.8 7.3.9 4.53 1.04 12.25
Ruby 3.1.0 yjit 25.33 9.97 61.21
Java JDK20 Int 27.50 4.71 57.18
Node.js 20.3 jitless 35.68 9.97 81.27
Bun 0.6.9 jitless 37.33 5.84 48.44
Ruby 3.1.0 41.31 12.88 89.81
Java-Temurin-8-int 46.05 5.13 181.77
Java JDK8 Int 46.08 5.18 183.73
CPython 3.11 48.92 10.95 79.19
Java-Temurin-17-int 54.43 4.48 179.82
TruffleSOM AST Native Int EE uber 55.00 1.19 132.49
Java JDK17 Int 57.13 4.60 188.28
PySOM AST Int 102.15 18.99 138.87
TruffleSOM AST Native Int EE super 106.77 18.39 167.66
PySOM BC Int 109.26 20.78 170.66
TruffleSOM AST Native Int EE base 114.81 23.20 170.03
TruffleSOM BC Native Int EE base 183.96 33.74 268.33
PyPy 3.8 7.3.9 Int 227.02 58.02 377.57
Graal.js EE 22.3 Native Int 233.02 36.73 505.37
TruffleRuby EE 22.3 Native Int 358.05 50.22 608.99
ykSOM Int 600.83 110.45 802.59
GraalPython EE 22.3 Native Int 738.71 68.74 1289.49
SOM-RS BC Int 1670.33 420.26 3127.36
SOM++ Int 3118.74 620.67 6614.88
SOM-RS AST Int 3673.31 822.01 6033.10
CSOM Int 5430.41 1364.01 15498.90

Details for all Benchmarks

The following plots show results for each of the benchmarks.

With JIT Compiler

plot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarksplot of chunk jit-benchmarks

Truffle Impl. Only

plot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarksplot of chunk jit-truffle-benchmarks