SOM Performance Report

This document reports on the performance of various SOM implementations. SOM (Simple Object Machine) is a minimal Smalltalk meant for teaching and research on Virtual Machines and language implementation techniques.

This report was generated on 2017-07-04 13:58:46.

Benchmarks

These results are based on the Are We Fast Yet benchmark suite. The suite is designed to compare allow us to compare performance across languages. It contains 5 macrobenchmarks and 9 microbenchmarks.

Methodology

Depending on the peak performance of the language implementation, we measure 25, 85, 250, or 1000 iterations of the same benchmark within the same VM process. The details are configured here. Warmup runs, i.e., runs with obvious signs of compilation are discarded. Thus, results correspond to peak performance.

All results are normalized to Java 1.8.0_131.

Summary:

reports peak performance
on Are We Fast Yet benchmarks
results are normalized to Java 1.8

Results

Other language implementations are not necessarily reaching performance similar to Java on our benchmarks. The following plot include all of the implementations.

plot of chunk som-overview

The above plot shows the peak performance for the following implementations:

HotSpot C2: Java benchmarks run on Java 1.8.0_131. C2 is used as top-tier compiler.
SOMns, a Newspeak implementation derived from TruffleSOM. It runs on HotSpot with Graal Core as compiler for Truffle.
Node.js: JavaScript benchmarks run on Node.js 8.1.3, which uses Ignition+TurboFan as default.
TruffleSOM is a SOM implementation on top of Truffle. It runs on HotSpot with Graal Core as compiler for Truffle.
RTruffleSOM is a SOM implementation on top of RPython. It uses a Truffle-like AST interpreter design.
HotSpot int, Java benchmarks run on Java 1.8.0_131 with -Xint, i.e., purely interpreted.
SOM++ OMR, the SOM++ interpreter using the Eclipse OMR JIT compiler and garbage collector.
SOMns int, SOMns executed on top of HotSpot without Graal, i.e. only the Truffle interpreter without Truffle JIT compilation.
RTruffleSOM int, RTruffleSOM without RPython's meta-tracing JIT compiler.
SOM++ int, a SOM bytecode interpreter written in SOM++.
SOM int, a SOM bytecode interpreter written in Java, executed on HotSpot.

Performance Overview Data

The following table contains the numerical representation of the results depicted above.

	Run-time Factor over Java
	geomean	sd	min	max	median
HotSpot C2	1.00	0.00	1.00	1.00	1.00
SOMns	2.22	0.93	1.26	3.99	2.54
Node.js	2.79	2.78	1.13	11.79	2.35
TruffleSOM	3.07	3.40	1.24	14.87	2.90
RTruffleSOM	3.45	2.35	1.54	8.52	3.26
Pharo	7.54	3.85	3.39	18.95	7.42
JRubyJ8	13.14	7.87	3.15	30.65	16.39
HotSpot Int	37.51	24.46	10.79	98.95	40.84
SOM++ OMR	107.54	42.03	54.02	215.58	103.97
SOMns int	110.52	67.05	47.81	265.32	123.59
RTruffleSOM int	165.18	84.41	56.08	330.36	192.25
SOM++ int	415.28	203.43	159.42	917.76	459.37
SOM int	486.93	256.03	203.83	1038.49	534.87

Performance Overview Data, SOM++

	Run-time Factor over Java
	geomean	sd	min	max	median
SOM++ OMR	1.00	0.00	1.00	1.00	1.00
SOM++ int	3.86	1.45	1.77	6.70	3.80
SOM int	4.53	1.61	1.37	7.58	4.84

Benchmark Results

The following table contains the numerical results for all benchmarks.

		Run-time Factor over Java
	VM	mean	sd
Bounce	HotSpot C2	1.00	0.03
	SOMns	1.39	0.05
	Node.js	4.38	0.00
	TruffleSOM	1.62	0.02
	RTruffleSOM	1.54	0.01
	Pharo	5.48	0.01
	JRubyJ8	14.85	0.06
	HotSpot Int	22.76	0.25
	SOM++ OMR	96.42	0.76
	SOMns int	142.48	0.95
	RTruffleSOM int	131.96	0.60
	SOM++ int	561.79	0.21
	SOM int	686.25	4.61
CD	HotSpot C2	1.00	0.02
	SOMns	2.62	0.19
	Node.js	2.41	0.01
	TruffleSOM	3.67	0.03
	RTruffleSOM	8.04	0.06
	Pharo	NaN	NaN
	JRubyJ8	10.35	0.05
	HotSpot Int	51.38	0.23
	SOM++ OMR	130.68	0.20
	SOMns int	240.64	3.51
	RTruffleSOM int	269.16	0.87
	SOM++ int	516.53	0.67
	SOM int	619.46	3.34
DeltaBlue	HotSpot C2	1.00	0.04
	SOMns	2.47	0.24
	Node.js	6.08	0.30
	TruffleSOM	4.67	0.15
	RTruffleSOM	5.56	0.04
	Pharo	7.38	3.71
	JRubyJ8	24.37	0.69
	HotSpot Int	52.72	0.75
	SOM++ OMR	106.42	0.37
	SOMns int	125.30	3.60
	RTruffleSOM int	130.48	0.13
	SOM++ int	279.02	1.78
	SOM int	356.10	12.01
Havlak	HotSpot C2	1.00	0.04
	SOMns	3.02	0.10
	Node.js	11.79	0.43
	TruffleSOM	4.67	0.17
	RTruffleSOM	NaN	NaN
	Pharo	18.95	1.15
	JRubyJ8	18.89	0.12
	HotSpot Int	50.22	0.34
	SOM++ OMR	91.11	0.17
	SOMns int	134.04	0.90
	RTruffleSOM int	122.11	1.69
	SOM++ int	294.64	2.78
	SOM int	329.25	1.99
Json	HotSpot C2	1.00	0.02
	SOMns	1.55	0.02
	Node.js	1.55	0.00
	TruffleSOM	2.06	0.02
	RTruffleSOM	2.28	0.04
	Pharo	10.66	0.07
	JRubyJ8	5.08	0.02
	HotSpot Int	40.23	0.10
	SOM++ OMR	148.65	0.46
	SOMns int	47.81	0.57
	RTruffleSOM int	71.03	0.05
	SOM++ int	262.48	0.56
	SOM int	203.83	2.11
List	HotSpot C2	1.00	0.01
	SOMns	3.08	0.04
	Node.js	2.27	0.00
	TruffleSOM	3.04	0.02
	RTruffleSOM	8.52	0.05
	Pharo	9.78	0.02
	JRubyJ8	3.15	0.03
	HotSpot Int	39.74	0.01
	SOM++ OMR	215.58	0.15
	SOMns int	121.87	1.26
	RTruffleSOM int	241.13	0.55
	SOM++ int	704.04	0.66
	SOM int	908.45	2.01
Mandelbrot	HotSpot C2	1.00	0.00
	SOMns	1.41	0.03
	Node.js	1.13	0.00
	TruffleSOM	1.24	0.00
	RTruffleSOM	1.76	0.01
	Pharo	5.17	0.00
	JRubyJ8	8.75	0.05
	HotSpot Int	10.79	0.03
	SOM++ OMR	74.23	0.07
	SOMns int	63.39	1.46
	RTruffleSOM int	124.72	0.41
	SOM++ int	271.43	0.25
	SOM int	307.36	3.49
NBody	HotSpot C2	1.00	0.01
	SOMns	1.59	0.03
	Node.js	1.74	0.01
	TruffleSOM	1.78	0.00
	RTruffleSOM	2.02	0.02
	Pharo	9.10	0.00
	JRubyJ8	18.39	0.08
	HotSpot Int	98.95	0.33
	SOM++ OMR	84.33	0.07
	SOMns int	112.00	0.90
	RTruffleSOM int	204.71	0.22
	SOM++ int	397.28	0.29
	SOM int	415.89	3.01
Permute	HotSpot C2	1.00	0.00
	SOMns	3.89	0.08
	Node.js	2.23	0.00
	TruffleSOM	4.00	0.05
	RTruffleSOM	5.99	0.04
	Pharo	9.30	0.02
	JRubyJ8	30.65	0.13
	HotSpot Int	41.46	0.03
	SOM++ OMR	117.77	0.14
	SOMns int	164.82	1.16
	RTruffleSOM int	211.12	0.12
	SOM++ int	602.03	0.47
	SOM int	641.20	5.03
Queens	HotSpot C2	1.00	0.02
	SOMns	2.81	0.07
	Node.js	2.33	0.00
	TruffleSOM	2.72	0.02
	RTruffleSOM	4.53	0.02
	Pharo	5.46	0.00
	JRubyJ8	20.72	0.06
	HotSpot Int	21.36	0.08
	SOM++ OMR	78.91	0.08
	SOMns int	89.07	0.81
	RTruffleSOM int	200.67	0.20
	SOM++ int	498.43	0.29
	SOM int	524.78	4.60
Richards	HotSpot C2	1.00	0.00
	SOMns	2.76	0.01
	Node.js	4.87	0.01
	TruffleSOM	2.76	0.02
	RTruffleSOM	3.09	0.01
	Pharo	7.42	0.03
	JRubyJ8	17.93	1.23
	HotSpot Int	75.81	0.13
	SOM++ OMR	161.79	0.06
	SOMns int	265.32	2.73
	RTruffleSOM int	330.36	1.04
	SOM++ int	517.18	0.17
	SOM int	811.54	2.43
Sieve	HotSpot C2	1.00	0.02
	SOMns	1.57	0.02
	Node.js	2.37	0.00
	TruffleSOM	1.75	0.01
	RTruffleSOM	3.26	0.01
	Pharo	6.40	0.28
	JRubyJ8	22.42	0.05
	HotSpot Int	13.11	0.06
	SOM++ OMR	137.02	1.29
	SOMns int	54.17	0.74
	RTruffleSOM int	183.83	0.10
	SOM++ int	917.76	0.36
	SOM int	1038.49	11.69
Storage	HotSpot C2	1.00	0.02
	SOMns	1.26	0.02
	Node.js	1.97	0.01
	TruffleSOM	14.87	1.10
	RTruffleSOM	2.23	0.03
	Pharo	3.39	0.23
	JRubyJ8	10.05	0.03
	HotSpot Int	35.94	0.24
	SOM++ OMR	54.02	0.06
	SOMns int	49.33	1.11
	RTruffleSOM int	56.08	0.02
	SOM++ int	159.42	0.26
	SOM int	231.06	2.46
Towers	HotSpot C2	1.00	0.02
	SOMns	3.99	0.08
	Node.js	2.80	0.02
	TruffleSOM	4.76	0.05
	RTruffleSOM	3.40	0.04
	Pharo	8.38	0.01
	JRubyJ8	9.82	0.07
	HotSpot Int	65.29	0.10
	SOM++ OMR	101.51	0.19
	SOMns int	163.53	1.57
	RTruffleSOM int	315.23	0.47
	SOM++ int	420.30	0.34
	SOM int	544.95	5.29

Benchmark Results, SOM++

		Run-time Factor over Java
	VM	mean	sd
Bounce	SOM++ OMR	1.00	0.01
	SOM++ int	5.83	0.00
	SOM int	7.12	0.05
CD	SOM++ OMR	1.00	0.00
	SOM++ int	3.95	0.01
	SOM int	4.74	0.03
DeltaBlue	SOM++ OMR	1.00	0.00
	SOM++ int	2.62	0.02
	SOM int	3.35	0.11
Havlak	SOM++ OMR	1.00	0.00
	SOM++ int	3.23	0.03
	SOM int	3.61	0.02
Json	SOM++ OMR	1.00	0.00
	SOM++ int	1.77	0.00
	SOM int	1.37	0.01
List	SOM++ OMR	1.00	0.00
	SOM++ int	3.27	0.00
	SOM int	4.21	0.01
Mandelbrot	SOM++ OMR	1.00	0.00
	SOM++ int	3.66	0.00
	SOM int	4.14	0.05
NBody	SOM++ OMR	1.00	0.00
	SOM++ int	4.71	0.00
	SOM int	4.93	0.04
Permute	SOM++ OMR	1.00	0.00
	SOM++ int	5.11	0.00
	SOM int	5.44	0.04
Queens	SOM++ OMR	1.00	0.00
	SOM++ int	6.32	0.00
	SOM int	6.65	0.06
Richards	SOM++ OMR	1.00	0.00
	SOM++ int	3.20	0.00
	SOM int	5.02	0.02
Sieve	SOM++ OMR	1.00	0.01
	SOM++ int	6.70	0.00
	SOM int	7.58	0.09
Storage	SOM++ OMR	1.00	0.00
	SOM++ int	2.95	0.00
	SOM int	4.28	0.05
Towers	SOM++ OMR	1.00	0.00
	SOM++ int	4.14	0.00
	SOM int	5.37	0.05