# Benchmarks

## MotoFit Benchmarks

I recently noticed that the reflectivity software package Motofit (http://motofit.sourceforge.net/) by Andrew Nelson (ANSTO, Australia) was benchmarked against other common reflectivity software.

Andrew provides model data against which Motofit, Parratt32, drydoc, and reflfit were compared. Below is a table of how well the various fitting routines available in yanera works with that same data, and same inital conditions.

On the left is a list of the constrained and unconstrained fitting methods. By "constrained", I mean whether one can set maximum and minimum limits for a parameter. In all cases, a parameter can be held fixed. We record the number of iterations, the execution time (in minutes:seconds), and the final chi-squared of fit for each method.

### Layer models

Shown in the left table all methods work well to fit the data using the multi-layer model type that Parratt32 and Motofit uses. Except for the steepest descent method, all reach a solution in about a minute. Even 1000 iterations of simulated annealing reaches a reasonable chi-squared quickly. Most remarkable are the Levenberg-Marquardt algorithms which converge and write to output in less than 2 seconds.

### Layer models

Yanera, like drydoc software, approximates interface roughness with small slabs when fitting using analytical expressions of the scattering density. I set up the model to fit using the equation (a+b*z) for the very thick layer, with a=10x10^-6 and b=0.001x10^-6 as inital conditions. We know that b should be 0 in the final fit.

In the cases tested, fitting is much slower. Levenberg-Marquardt algorithms work well to achieve resonable chi-squared in about 8 minutes of compute time. It should be noted that they converge very rapidly at first, but slow down after the first few iterations. This is a consequence of the slow approach of b to 0.

The Neadler-Mead simplex method iterates much faster (since is doesn't take derivatives), but converges much more slowly. The final result by simulated annealing is comparable to the layer model, but takes much longer. So far, only one method has failed show much convergence.

 Layer model Function model Iterations Time Chi-Sq Iterations Time Chi-Sq UNCONSTRAINED CONJUGATE_FR 500 01:12.93 1.353 CONJUGATE_PR 500 01:15.70 1.141 VECTOR_BFGS 500 01:10.88 1.255 STEEP_DESC 500 00:57.38 51.269 NMSIMPLEX 1500 00:07.03 1.145 1500 02:49.94 72.110 LMDR 12 00:01.20 1.106 LMSDR 16 00:01.64 1.106 13 07:03.45 1.550 CONSTRAINED SIMAN 1000/50 01:38.38 2.375 1000/50 18:47.10 2.654 CLMDR 20 00:01.24 1.297 CLMSDR 22 00:01.47 1.297 OOL_SPG 500 00:20.36 1.596 OOL_GENCAN 18 01:07.04 1.106 > 20:00.00 LEVMAR 8 00:00.82 1.106 13 07:43.75 1.242 Starting Chi-Sq 5378.800 Starting Chi-Sq 1978.9

Generated on Thu May 29 10:56:33 2008 by  1.5.5