Compared performance of Varnish Cache on x86_64 and aarch64

[Poul-Henning Kamp]

--------
Martin Grigorov writes:

> Any feedback and ideas how to tweak it (VCL or even patches) are very
> welcome!

First you need to tweak your benchmark setup.

   aarch64

	  Thread Stats   Avg      Stdev     Max   +/- Stdev
	    Latency   655.40us  798.70us  28.43ms   90.52%

Strictly speaking, you cannot rule out that the ARM machine
sends responses before it receives the request, because your
standard deviation is larger than your average.

In other words:  Those numbers tell us nothing.

If you want to do this comparison, and I would love for you to do so,
you really need to take the time it takes, and get your "noise" down.

Here is how you should do it:

	for machine in ARM, INTEL
		Reboot machine
		For i in (at least) 1-5:
			Run test for 5 minutes

If the results from the first run on each machine is very different
from the other four runs, you can disrecard it, as a startup/bootup
artifact.

Report the numbers for all the runs for both machines.

Make a plot of all those numbers, where you plot the reported
average +/- stddev as a line, and the max value as a dot/cross/box.

If you want to get fancy, you can do a Student's T test to tell
you if there is any real difference.  There's a program called
"ministat" which will do this for you.

Also:  I can highly recommend this book:

   http://www.larrygonick.com/titles/science/the-cartoon-guide-to-statistics/

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.