Progress Report

May 23^rd, 2006

Overview

• Discussions about high-dimensional data and its corresponding normality tests, if any are suited for the task
• Noise and sensitivity experiments
• Implementation of a new NRR algorithm -- one which is related to congealing (an attempt at cloning it)
• Preparation for a TMI re-submission (re-registration, files, experiments)

Normality Tests

• Anderson-darling (AD) test for fit in Chi-squared distributions
• Apply to 37 images, 15 modes are used in evaluation
• Experiments with toy data by Carole
• Implemented the AD statistic, as well as KS
• Does not think it is worth the trouble, as the central limit theorem says that it is not worth doing
• No answers or opinions about the outcome, so might be worth setting aside for the time being
• Can do some writing and further experiments, but only if there is a point in doing so

Normality Tests - Conclusions

• Histograms/output depicted the distribution of values for data from a normal distribution, and from a flat distribution
• The conclusion is that so far, one could tell some (atypical) normal distributions were almost certainly not flat, but not vice versa
• Practically anything might as well be modelled as normal, but it is difficult to prove this
• Applying normality tests to small numbers of data points is a long process that has no eventual merits

Another Scheduled Task: Noise Experiments

• Accumulated sensitivity figures
• 4-dotted (i.e. 4 sample points) curves for Euclidean and shuffle distance
• One for Generalisation and one for Specificity
• Noise levels 0%, 2.5%, 5%, and 10%
• Aggregated sensitivity with the effect of noise shown

Sensitivity of Generalisation

Aggregated Sensitivity of Generalisation

Sensitivity Plots

• Can put error bars on these plots
• Error propagation makes this laborious, but shown is pretty much what we would ultimately get (albeit it is merely a rough estimate)
• Since the errors are quite large and there is no obvious pattern, I have not computed these properly
• Can do so if desired, but it requires quite a bit of work
• Assumed that when the plots are visually assesses we will conclude there is a trend
• In reality: no obvious trend can be spotted -- which is a bit surprising

Estimation of Error Bars

Lines represent estimated scale of error bars

Explanation About Graphs

• One graph is for Specificity and the other for Generalisation. Each contains one colour for Euclidean and one for shuffle, as before.
• Looking at the figure, they are rather sloppy as they don't explain what they contain
• On the right hand side it is sensitivity of Specificity and on the left it is sensitivity of Generalisation
• As before, the green lines, which represent the error bars, correspond to shuffle 5x5 (going from no noise on the left to 10% at the right column)
• The red lines are the error range for Euclidean
• No interesting conclusions can be arrived at
• Tried to see if a mistake was made, to no avail

Sensitivity Plots - A Sanity Check

• Done a cursory check followed by a more comprehensive sanity check on all of these
• Previously, when we looked at the plots, there didn't appear to be a pattern
• re-checked the numbers more carefully, hopefully to spot a mistake
• Looked at the numbers and the figures again. There appears to have been no mistake. Also see previous curves (before aggregation), as they affirm this.
• When summing these curves up, there is no apparent trend, not even if one excludes the first point, which behaves oddly

Sensitivity of Specificity

Aggregated Sensitivity of Specificity

Data from 2005 Experiments (IPMI)

• Compressed some files which Tim will find useful
• Also included a digest including his description of the experiments, collated from old E-mails
• 3 JPEG's just show the 1^st mode of the 3 models compared at the time
• The intent (experimental plan): carry out 3 evaluations for group-wise and pair-wise, as well as congealing, 5x5 shuffle, different numbers of modes, and as many repetitions as time permits

Early Stages of Evaluation Experiments

• Received a set of new results for pair/groupwise registration
• Ran the software on it in the same way as before and came up with some reminiscent results
• Start this right away (overnight) and repeated for as many choices of [number of modes] as time permitted

Early Glimpse at Assessment Results

With 2 modes used in assessment:

Pairwise experiment - Gen: 4.96257 se: 0.0831228 Spec: 3.68014 se: 0.00793536

Groupwise experiment mean-only (no shape) - Gen: 4.80943 se: 0.0846461 Spec: 3.5619 se: 0.00912405

Groupwise experiment shape (width=100) - Gen: 4.81031 se: 0.0845343 Spec: 3.56228 se: 0.00898879

Congealing experiment - Gen: 5.37891 se: 0.0896088 Spec: 4.04577 se: 0.00683385

Going in steps of 4 towards 20 to replicate the old figure. With steps of 2 there would not be enough time for completion though.

Reviews and Early Results

• Not familiar with this process of addressing the reviewers' comments
• However, I had a first (and very rough) go at it, just to hear if I am way off
• As we continue to receive results from the experiments (6 modes, up from just 2), it is possible to see that groupwise and MDL are the best NRR algorithms, followed by pairwise and congealing (which turns out to be considerably worse)
• Comparing NRR Algorithms: Here are the results accumulated so far, at the earlier stages

Specificity of the Various NRR Algorithms

Generalisation of the Various NRR Algorithms

Editing the Text

• Could/should I edit the latest TMI TeX? You (the Supervisor) are currently in hold of the latest version, so I worry that we might make conflicting edits
• It is not clear to me if you want to make all changes yourself
• Poised to begin if you permit this
• Tim made contributions and has the latest version of the text, with peripheral files, and graphics
• Not sure if that version contained all the figures, but I hope so
• Tim was sent all the files to write a paragraph or two about congealing, which he did

Latest Notes on Correspondence - Suggestions by Tim

• The response to reviewers is a good start
• However, it's better to be a bit more detailed about the changes
• Where appropriate (i.e. when not a very small change, or a very large one), include the new text (together with its section number)
• Ideally the reviewer should be able to read the document and be reasonably happy that you've responded sensibly without having to re-read the whole paper again

Latest Notes on Correspondence - Carole

• Usually one should do an itemised list. Start each item with a quote as to the bit you are addressing. You don't need to quote the whole review, just the relevant bit.
• Then what you have done to address it, referencing the line, paragraph and page in the new document
• I usually do the two parts in different fontweights, for example, put the quoted part on bold, and then your response in ordinary
• Best to quote the reviewers points in order
• If the second reviewer has made same point, put explanation in again

Latest Notes on Correspondence - Carole - Ctd.

• \section*{Reviewer One} - Section starts a section but without the numeric labelling
• begin{itemize} \item{Stuff} \item{ {\bf quote from reviewer \ldots } - Another bit
• Our response, see Page here, paragraph two, last line, third word.............}
• \item{More stuff} \end{itemize}

Congealing Experiments

• Tim ran a variety of variants on the experiments over the weekend, but always found that the congealing was poorer than the other methods
• Later came to discuss what we ought to put in the text about this
• Subsequently sent an alternative version of the congealing data
• The previous version used a sum(prob) form of objective function
• This new version uses a sum(log prob) form, which is the approximation recommended by Learned-Miller

Congealing Experiments - Ctd.

• Given that we have some time, I re-ran the experiments on the new data
• I am gradually finding out how this compares to the predecessor
• I use 1000 examples, but run it nice'd, which means that it takes about 2-3 hours per evaluation

Recent Suggestion

• It is not clear what to do, now that the congealing turns out to be worse than pairwise
• With the benefit of hindsight, we wish that we had run the different registration algorithms on the data for which we have ground truth
• Then we could have said (presumably) that the ground truth based approach gave the same answer regarding the relative merits of the different algorithms
• Might be too late to run the different registration algorithms on the MGH data. Could use overlap method, doing a rush job on running the overlap measures, but then it wouldn't matter what order the methods came in, as long as they came in the same order using overlap and Specificity

Recent Suggestion - Ctd.

• The only problem I can think of is that the latest of VXL is in the hands of Tim
• I will not manage to output the deformation field from the raw images
• Other than that, it is possible
• It will only be a matter of time, although the evaluation algorithms will be 3 times quicker to run (taking a third of the time to complete, to be more precise)
• The latest results from the congealing-type algorithm appear much better than their predecessors

Present Progress

• At present, we get as far as we can with new results, since the others just confuse the picture
• starting with 'sensible' number of modes (i.e. not at 2, working all the way up)
• Finished running it with 12 modes, more processed at this very moment (it seriously hinders work, so better off done overnight)

Early Results

• Below are the results that we get for the new(er) congealing-type NRR algorithm
• Using just the first 12 modes of the model
• The changes have improved that algorithm, judging by Generalisation and Specificity

Early Results - Ctd.

Gen: 4.33357 se: 0.0652538 Spec: 4.02085 se: 0.0103181



Compare with:



Pairwise - Gen: 4.28118 se: 0.0611423 Spec: 3.94977 se: 0.0111107

Groupwise 2 - Gen: 4.15031 se: 0.0587872 Spec: 3.87714 se: 0.011745

Groupwise 1 - Gen: 4.14852 se: 0.0599883 Spec: 3.86781 se: 0.011341

Old congealing - Gen: 4.5239 se: 0.0658246 Spec: 4.20466 se: 0.0105512

Further thoughts

• We know that the quality of pairwise registration of a group to a reference depends on the choice of reference
• Q: "Are the pairwise results we are using new ones?" A: "When I ran some quick evaluations on the new congealing version it was better than the previous one, but still significantly worse than the pairwise. The new one roughly halves the difference. We haven't played with the choice of reference image in the pairwise - I might have a chance to run a couple of different tests tonight, but it is a long shot - the congealing is so much worse that it's unlikely we'd manage to get a poorer result."

Further thoughts - Ctd.

• I think a better approach is to present the results as-is, as a demonstration of how we can compare different methods using the specificity - relating to suggested text in response to the 'self-fulfilling prophesy' argument.
• Q: Have you re-done the pairwise results we are using, or are we still using the old one? If these are new results, have we made sure that for 'illustrative purposes' we are using a bad reference? We can and probably should include a sentence to say 'of course if you used a different reference you might get a better or indeed worse result.

Further thoughts - Ctd.

• Fingers crossed the new version of congealing will demonstrate what we want, but just in case we need to ensure that the pairwise is as bad as it honestly can be
• Data we were using in pairwise NRR us new, so currently looking at what happens with the old pairwise data
• Data was previously evaluated in MATLAB, so algorithms and values vary slightly
• Preparation of this progress report prevented taking this further (namely C++/VXL evaluation), which can be done tonight
• Sent the reviews in Word format (from OpenOffice), as well as text fragments from Tim
• Paper re-written at the moment, as well as reply to the reviewers