Benchmarks

MotoFit Benchmarks

I recently noticed that the reflectivity software package Motofit (http://motofit.sourceforge.net/) by Andrew Nelson (ANSTO, Australia) was benchmarked against other common reflectivity software.

Andrew provides model data against which Motofit, Parratt32, drydoc, and reflfit were compared. Below is a table of how well the various fitting routines available in yanera works with that same data, and same inital conditions.

On the left is a list of the constrained and unconstrained fitting methods. By "constrained", I mean whether one can set maximum and minimum limits for a parameter. In all cases, a parameter can be held fixed. We record the number of iterations, the execution time (in minutes:seconds), and the final chi-squared of fit for each method.

Layer models

Shown in the left table all methods work well to fit the data using the multi-layer model type that Parratt32 and Motofit uses. Except for the steepest descent method, all reach a solution in about a minute. Even 1000 iterations of simulated annealing reaches a reasonable chi-squared quickly. Most remarkable are the Levenberg-Marquardt algorithms which converge and write to output in less than 2 seconds.

Layer models

Yanera, like drydoc software, approximates interface roughness with small slabs when fitting using analytical expressions of the scattering density. I set up the model to fit using the equation (a+b*z) for the very thick layer, with a=10x10^-6 and b=0.001x10^-6 as inital conditions. We know that b should be 0 in the final fit.

In the cases tested, fitting is much slower. Levenberg-Marquardt algorithms work well to achieve resonable chi-squared in about 8 minutes of compute time. It should be noted that they converge very rapidly at first, but slow down after the first few iterations. This is a consequence of the slow approach of b to 0.

The Neadler-Mead simplex method iterates much faster (since is doesn't take derivatives), but converges much more slowly. The final result by simulated annealing is comparable to the layer model, but takes much longer. So far, only one method has failed show much convergence.



Layer model


Function model












Iterations Time Chi-Sq
Iterations Time Chi-Sq
UNCONSTRAINED







CONJUGATE_FR
500 01:12.93 1.353



CONJUGATE_PR
500 01:15.70 1.141



VECTOR_BFGS
500 01:10.88 1.255



STEEP_DESC
500 00:57.38 51.269



NMSIMPLEX
1500 00:07.03 1.145
1500 02:49.94 72.110
LMDR
12 00:01.20 1.106



LMSDR
16 00:01.64 1.106
13 07:03.45 1.550
CONSTRAINED







SIMAN
1000/50 01:38.38 2.375
1000/50 18:47.10 2.654
CLMDR
20 00:01.24 1.297



CLMSDR
22 00:01.47 1.297



OOL_SPG
500 00:20.36 1.596



OOL_GENCAN
18 01:07.04 1.106

> 20:00.00
LEVMAR
8 00:00.82 1.106
13 07:43.75 1.242



Starting Chi-Sq 5378.800

Starting Chi-Sq 1978.9


Generated on Thu May 29 10:56:33 2008 by  doxygen 1.5.5