Benchmarks

MotoFit Benchmarks

I recently noticed that the reflectivity software package Motofit (http://motofit.sourceforge.net/) by Andrew Nelson (ANSTO, Australia) was benchmarked against other common reflectivity software.

Andrew provides model data against which Motofit, Parratt32, drydoc, and reflfit were compared. Below is a table of how well the various fitting routines available in yanera works with that same data, and same inital conditions.

On the left is a list of the constrained and unconstrained fitting methods. By "constrained", I mean whether one can set maximum and minimum limits for a parameter. In all cases, a parameter can be held fixed. We record the number of iterations, the execution time (in minutes:seconds), and the final chi-squared of fit for each method.

Layer models

Shown in the left table all methods work well to fit the data using the multi-layer model type that Parratt32 and Motofit uses. Except for the steepest descent method, all reach a solution in about a minute. Even 1000 iterations of simulated annealing reaches a reasonable chi-squared quickly. Most remarkable are the Levenberg-Marquardt algorithms which converge and write to output in less than 2 seconds.

Layer models

Yanera, like drydoc software, approximates interface roughness with small slabs when fitting using analytical expressions of the scattering density. I set up the model to fit using the equation (a+b*z) for the very thick layer, with a=10x10^-6 and b=0.001x10^-6 as inital conditions. We know that b should be 0 in the final fit.

In the cases tested, fitting is much slower. Levenberg-Marquardt algorithms work well to achieve resonable chi-squared in about 8 minutes of compute time. It should be noted that they converge very rapidly at first, but slow down after the first few iterations. This is a consequence of the slow approach of b to 0.

The Neadler-Mead simplex method iterates much faster (since is doesn't take derivatives), but converges much more slowly. The final result by simulated annealing is comparable to the layer model, but takes much longer. So far, only one method has failed show much convergence.

		Layer model				Function model
		Iterations	Time	Chi-Sq		Iterations	Time	Chi-Sq
UNCONSTRAINED
CONJUGATE_FR		500	01:12.93	1.353
CONJUGATE_PR		500	01:15.70	1.141
VECTOR_BFGS		500	01:10.88	1.255
STEEP_DESC		500	00:57.38	51.269
NMSIMPLEX		1500	00:07.03	1.145		1500	02:49.94	72.110
LMDR		12	00:01.20	1.106
LMSDR		16	00:01.64	1.106		13	07:03.45	1.550
CONSTRAINED
SIMAN		1000/50	01:38.38	2.375		1000/50	18:47.10	2.654
CLMDR		20	00:01.24	1.297
CLMSDR		22	00:01.47	1.297
OOL_SPG		500	00:20.36	1.596
OOL_GENCAN		18	01:07.04	1.106			> 20:00.00
LEVMAR		8	00:00.82	1.106		13	07:43.75	1.242
			Starting Chi-Sq	5378.800			Starting Chi-Sq	1978.9