Progress Report

January 3^rd, 2006

Overview

• Evaluation based on the entropic graphs method
• Dicussion - method in theory and in practice
• Setsets and randomisation in entropy estimation
• Paper draft progress
• References management and 'housekeeping' chores
• Approaching talk, deadlines

Entropic Graphs-based Evaluation

• An error was corrected, which previously led to badly-scaled results
• Cause: brackets were closed at the wrong place
• The new curves look more or less the same as before
• Further work was done on isolating free parameters (e.g. alpha, constant) so that their effect can be studied
• The evaluation framework is much more modular
• Interfaces make experiment setup/arrangement trivial and invocation immediate

Various Steps to Follow

• No computer resources available throughout the holiday
• Remote access is still needed for brute-force-dependent studies
• Learn the effect of various parameters by repeating the experiments with different values
• Consider alterring:
  • Alpha value
  • Constant values
  • k as in kNN
• Discuss error estimation propagation

Coming Slides that Contain Figures

• Shown are some figures which we previously looked at
• The error in the entropy equation was eradicated
• Attempts were made to discover why we had gotten such high numbers
• Ultimately this led to the correct graphs (shown at the end)

Entropy Graphs #1

Entropy of Synthetics->Synthetics and Training->Synthetics

Entropy Graphs #2

Corrected figure (Y-scale changes)

Entropy Versus Generalisation and Specificity

• Still need to add the error bars to entropy
• Propagation of errors seems complex
• Getting more instantiations included in the analysis
• Curve are expected to be monotonic given more instantiations
• Data which was used for the ISBI submission seems to support this expectation

Entropic Measures Overlaid atop Gen. and Spec.

The curves shown beforehand and their relationship to Generalisation and Specificity (instantiation 1 only)

Survey of Methods

• When reversed, one can see how well-correlated entropy is to Gen. and Spec.
• These are the corrected plot, which show the entropy as a function of misregistration
• The analysis of the numbers is innately the same
• Normalisation included, which is a pro
• More flexibility available in this form of MST-based calculation, e.g. generalisation to kNN with weighting (see later).

Entropy Graphs #3

Entropic graph-based measures - revised, shown in a clearer way

Entropy Graphs #4

Training set to synthetic set - entropy shown in isolation

Improvements to the Assessment Framework

• Added two new options to the model/NRR evaluation framework
• Both option are easily controlled from outside the core of the program
• Option 1: Can make the sample of images for distance estimation random
• Option 2: Can use just a subset of a given pool of image rather than iterating over entire sets
• Clusters at the School of Computer Science down throughout the holiday
• Inability to run large experiments and produce useful results (yet)

Improved Entropy Measures

• Implemented additive power weighted kNN
• Rather than just minimal spanning trees (MST), consider multiple near nodes
• Weighting halves at each stage so if nearest neighbour has weight x, the second nearest neighbour has weight x/2
• Might lead to more robust results, but need to run experiments first

Towards a Draft/Submission

• Continued work on this draft, which is somewhat presentable
• The last few parts are still unorganised
• Getting close to having a complete draft which spans roughly 10 pages
• Need to encourage colleagues to glance over it, giving some feedback and suggestion
• The latest version of the document - Draft (PDF)

Submission Length Restrictions

• Still unable to find information on the number of pages permitted
• The instructions are a collection of assorted out-of-date documents and they are not consistent
• Submission deadline is February 1^st
• Corresponding author yet unknown, but S. Warfield was contacted already

References Management

• Migration, assemblage and conversions pending
• Management of references in a Web-based PHP/MySQL tool
• Somewhat of a free CMS that gained popularity
• Open Source, multiple users, categories/groups, collaborative management
• Rich graphical interface, requires authentication, available from any Web browser, anywhere in the world

Talk on NRR Assessment

• With and without ground truth
• Abstract similar to the one embedded in the ISBI submission, so material/content ought to remain similar
• Main advancement is entropic graphs
• Other advancements less significant and possibly beyond scope (~15 minutes only)
• Students Talk on January 27^th, absence is accidental

Summary

• Framework for evaluating graph entropy
• Experiments support the hypothesis
• Paper draft in progress
• Student Talks (or Monday Meeting) coming up
• References management
• Paper draft in progress