Progress Report

November 15^th, 2005

About the Progress Report

• This progress report will be relatively short
• Principal emphasis to be put on the looming deadlines
• Further break-down into next steps of practical work
• A slight bit on organisational matters

General Overview

• Long time since the previous meeting
• In the midst of this cessation of meetings:
  • MIAS-Grid - registration assessment benchmarks
  • Worth pursuing once deadlines and other activities have abated
  • Submission of MIAS-IRC abstract (registration assessment) accepted as an oral presentation
  • Abstract on model-building accepted as an oral presentation as well
  • Submissions to CVPR 2006 and ISBI 2006 pending (the former was due yesterday)

MIAS-Grid: Diagram of Assessment Framework

Registration assessment benchmarking (larger-sized figure)

Notes on Overlap-based Evaluation

• There were some gaps in understanding of the meaning of alpha weighting
• To quote from MICCAI 2005:
  • "The weight alpha is introduced to respectively define the relative contribution of labels and subjects to the overlap measures. With alpha=1, all labels are implicitly weighted by their volume. This may not be desirable as smaller labels may represent a greater registration or segmentation challenge. Two alternative choices are to set alpha either to (i) the inverse mean volume of the current label pair to give all labels equal weighting or (ii) the inverse mean volume squared of the current label pair to weight by the inverse volume."

Overlap-based Assessment

• The assignments appear to reflect on a general behaviour, not a value. For MICCAI, the following were investigated:
  • Volume-weighting
  • Equal weighting
  • Inverse volume weighting
  • Union of the labels
  • Simple average

Overlap-based Assessment

• For the MIAS-IRC and ISBI submissions, the following were investigated:
  • Volume-weighting (w=0): The volume columns are the overlap for default (implicit) weighting. Larger labels have more weight in the accumulated overlap
  • Equal weighting (w=1): All labels treated equally
  • Inverse volume weighting (w=2): Unlike volume-weighting
  • Measure of complexity (w=3): The complexity of the labels

MIAS-IRC Presentation

• Progress made at the presentation already (graphics in particular)
• Not completely clear if UCL is to put together slides
• Contribution of text and material from UCL
• Two weeks remain until completion (November 29^th-30^th)
• UPDATE: Presentation to be split between Manchester and UCL

Face Model Evaluation

• We swiftly moved into investigation of face models
• Proper, well-understood perturbation was applied this time
• 16 sample points (degraded models) were used to plot the 'model degradation versus quality' figures
• The results were encouraging, but needed more instantiations for smoother curves
• Initial evaluation drop at the start - a problem that crept back into life

Face Models and Generalisation

Plot which shows the evaluation of progressively degraded face models - Generalisation as the quality criterion

Face Models and Specificity

Plot which shows the evaluation of progressively degraded face models - Specificity as the quality criterion

Generalisation Sensitivity - Face Models

Generalisation of the Specificity of progressively degraded face models

Specificity Sensitivity - Face Models

Sensitivity of the Specificity of progressively degraded face models

Notes on Sensitivity Plots - Face and Brain

• Without the sensitivity plots, it is difficult to see by eyesight which shuffle distance is best, though it is clear that Euclidean is worse than the rest
• Sensitivity plots for brain models look okay
• Dubious for the face
• Observe the effect of the 'dip' at the start
• We definitely ought to resolve it somehow

Sensitivity Plots - Generalisation and Specificity - Face

• In these plots we have 'inverse sensitivity', which implies that we cannot tell apart good models from worse ones when they verge optimality
• Thus sensitivity plots left out altogether from a CVPR submission
• Note that the intersection with 0 in the plots indicates at which point (pixel displacement) we can begin to evaluate the models
• Need to get to the bottom of it as soon as possible
• The fact that using 7x7 almost mends it might provide a clue

Sensitivity Plots - Face - Ctd.

• I re-checked the code this and can confirm that the re-sampling error (half-a-pixel offset) is there at the start
• We had some small 'dips' when handling brain data, but these got 'averaged away' since we had 10 instantiations
• Possibilities:
  • Evaluate using shuffle 11x11 (highly excessive) and see what happens
  • Start a second instantiation and see if results are improved. If so, continue to increase number of instantiations for further averaging

Submissions - Rudimentary Notes

• ISBI submission
  • Should bear the exact same title as the MIAS-IRC. It should be a direct extension of the MIAS-IRC abstract and ought to re-use text from the rejected MICCAI submission.
  • The ISBI submission had an outline already
  • Much of the work was already there

Submissions - Rudimentary Notes - Ctd.

• CVPR submission
  • Potentially the BMVC 2005 submission with modified results and some modified text
  • Possibly a reference to ground truth at the end
  • Unlikely/undesirable: Incorporate sensitivity figures despite the unwanted behaviour of face model evaluation (drop at the start)
  • Might just display the sensitivity plots for brain models?
  • Could still argue in our defence saying that we must have multiple instantiation for sensitivity to be sufficiently meaningful

ISBI 2006 Submission

• The first part contains the latest version of the MIAS-IRC abstract
• The second part was extracted from a MICCAI 2005 submission, from which we re-used some text
• The last part was the planned structure for the 4-page paper
• Need more text which discussed the overlap-based assessment method
• Hindered slightly due to illness

ISBI 2006 Submission - Ctd.

• There are a few things that have changed since the previous submission(s):
  • The perturbation method is much more principled
  • We have a comparison with ground-truth results
  • We have multiple instantiations
  • The graphs we produce make our points more explicit
• Feedback-driven:
  • Taken account of previous comments by MICCAI referees
  • Comment from reviewers of the MIAS-IRC abstract

CVPR 2006 Submission

• Attempt to re-use what we still have from the BMVC 2005 submission
• I included 2 figures among the new results at the end of the document
• These are based on a better-understood perturbation framework
• Evaluation of brain models taken from the MIAS-IRC/ISBI submission

Next Steps

• Work on a journals paper which summarises work on registration assessment and/or model evaluation
• Normalisation (among other possible extensions)
• Other conference deadlines lie far ahead (apart from ICPR)
• Form 11 - long overdue
• Continued work on thesis
• Thesis - possibly set draft submission date? Submission in chapters?