Motivation

A fundamental phenomenon of natural language is the variability of semantic expression, where the same meaning can be expressed by or inferred from different texts. Many natural language processing applications, such as Question Answering (QA), Information Retrieval (IR), Information Extraction (IE), and (multi) document summarization need to model this variability in order to recognize that a particular target meaning can be inferred from different text variants. Even though many applications face similar underlying semantic problems, these problems are usually addressed in an application-oriented manner. Consequently it is difficult to compare, under a generic evaluation framework, semantic methods that were developed within different applications. The PASCAL RTE Challenge introduces textual entailment as a common task and evaluation framework, covering a broad range of semantic-oriented inferences needed for practical applications. This task is therefore suitable for evaluating and comparing semantic-oriented models in a generic manner. Eventually, work on textual entailment may promote the development of general semantic “engines”, which will be used across multiple applications.

Textual Entailment

Textual entailment recognition is the task of deciding, given two text fragments, whether the meaning of one text is entailed (can be inferred) from another text (see the Instructions tab for the specific operational definition of textual entailment assumed in the challenge). This task captures generically a broad range of inferences that are relevant for multiple applications. For example, a Question Answering (QA) system has to identify texts that entail the expected answer. Given the question “Who killed Kennedy?”, the text “the assassination of Kennedy by Oswald” entails the expected answer “Oswald killed Kennedy”. Similarly, in Information Retrieval (IR) the concept denoted by a query expression should be entailed from relevant retrieved documents. In multi-document summarization a redundant sentence or expression, to be omitted from the summary, should be entailed from other expressions in the summary. In Information Extraction (IE) entailment holds between different text variants that express the same target relation. And in Machine Translation evaluation a correct translation should be semantically equivalent to the gold standard translation, and thus both translations have to entail each other. Thus, modeling textual entailment may consolidate and promote broad research on applied semantic inference.

Task Definition

Participants in the evaluation exercise are provided with pairs of small text snippets (one or more sentences in English), which we term Text-Hypothesis (T-H) pairs. Examples were manually tagged for entailment (i.e. whether T entails H or not) by human annotators and will be divided into a Development Set, containing 800 pairs, and a Test Set, containing 800 pairs. Participating systems will have to decide for each T-H pair whether T indeed entails H or not, and results will be compared to the manual gold standard.
The goal of the RTE challenges is to provide opportunities for presenting and comparing possible approaches for modeling textual entailment. In this spirit, we aim at an explorative rather than a competitive setting. While participant results will be reported there will not be an official ranking of systems. A development set is released first to provide typical examples of the different types of test examples. The test set will be released three weeks prior to the result submission date. We regard it as acceptable to run automatic knowledge acquisition methods (such as synonym collection from corpora or the Web) specifically for the linguistic constructs that are present in the test data, as long as the methodology is general and fully automated, and the cost of running the learning/acquisition procedure at full scale can be reasonably estimated.

Dataset Collection and Application Settings

The dataset of Text-Hypothesis pairs was collected by human annotators. It consists of four subsets, which correspond to typical success and failure settings in different applications (as listed below). Within each application setting the annotators selected both positive entailment examples (annotated YES), where T does entail H, as well as negative examples (annotated NO), where entailment does not hold (50%-50% split). Some T-H examples appear in the Instructions section. H is a (usually short) single sentence, and T consists of one or more sentences, up to a short paragraph, to simulate an even more realistic scenario.

Information Retrieval (IR):

In this application setting, the hypotheses are propositional IR queries, which specify some statement, e.g. “Alzheimer’s disease is treated using drugs”. The hypotheses were adapted and simplified from standard IR evaluation datasets (TREC and CLEF). Texts (T) that do or do not entail the hypothesis were selected from documents retrieved by different search engines (e.g. Google, Yahoo and Microsoft) for each hypothesis. In this application setting it is assumed that relevant documents (from an IR perspective) must necessarily entail the hypothesis.

Multi-document summarization (SUM):

In this setting T and H are sentences taken from a news document cluster, a collection of news articles that describe the same news item. Annotators were given output of multi-document summarization systems, including the document clusters and the summary generated for each cluster. The annotators picked sentence pairs with high lexical overlap, preferably where at least one of the sentences was taken from the summary (this sentence usually played the role of T). For positive examples, the hypothesis was simplified by removing sentence parts, until it was fully entailed by T. Negative examples were simplified in the same manner. This process simulates the need of a summarization system to identify information redundancy, which should be avoided in the summary.

Question Answering (QA):

Annotators were given questions and the corresponding answers returned by QA systems. Transforming a question-answer pair to text-hypothesis pair consisted of the following stages: First, the annotators picked from the answer passage an answer term of the expected answer type, either a correct or an incorrect one. Then, the annotators turned the question into an affirmative sentence with the answer term “plugged in”. These affirmative sentences serve as the hypotheses (H), and the original answer passage serves as the text (T). For example, given the question, “Who is Ariel Sharon?” and an answer text “Israel‘s prime Minister, Ariel Sharon, visited Prague” (T), the question is turned into the statement “Ariel Sharon is the Israeli Prime Minister” (H), producing a positive entailment pair. This process simulates the need of a QA system to verify that the retrieved passage text indeed entails the provided answer.

Information Extraction (IE):

This task is inspired by the Information Extraction (and Relation Extraction) application, adapting the setting to having pairs of texts rather than a text and a structured template. The pairs were generated using different approaches. In the first approach, ACE-2006 relations (the relations proposed in the ACE-2007 RDR task) were taken as templates for hypotheses. Relevant news articles were then collected as texts (T) and corresponding hypotheses were generated manually based on the ACE templates and slot fillers taken from the text. For example, given the ACE relation ‘X work for Y‘ and the text “An Afghan interpreter, employed by the United States, was also wounded.“(T), a hypothesis “An interpreter worked for Afghanistan.” is created, producing a non-entailing (negative) pair. In the second approach, the MUC-4 annotated dataset was similarly used to create entailing pairs. In the third approach, the outputs of actual IE systems were used to generate entailing and non-entailing pairs. In the forth approach, new types of hypotheses that may correspond to typical IE relations were manually generated for different sentences in the collected news articles. These processes simulate the need of IE systems to recognize that the given text indeed entails the semantic relation that is expected to hold between the candidate template slot fillers.

Task Definition

We consider an applied notion of textual entailment, defined as a directional relation between two text fragments, termed T – the entailing text, and H – the entailed text. We say that T entails H if, typically, a human reading T would infer that H is most likely true. This somewhat informal definition is based on (and assumes) common human understanding of language as well as common background knowledge. Textual entailment recognition is the task of deciding, given T and H, whether T entails H.

Table 1: Example T-H pairs

Some additional judgment criteria and guidelines are listed below (examples are taken from Table 1):

• Entailment is a directional relation. The hypothesis must be entailed from the given text, but the text need not be entailed from the hypothesis.
• The hypothesis must be fully entailed by the text. Judgment would be “NO” if the hypothesis includes parts that cannot be inferred from the text.
• Cases in which inference is very probable (but not completely certain) are judged as “YES“. In example #5 one could claim that although Shapiro’s office is in Century City, he actually never arrives to his office, and works elsewhere. However, this interpretation of T is very unlikely, and so the entailment holds with high probability. On the other hand, annotators were guided to avoid vague examples for which inference has some positive probability which is not clearly very high.
• Our definition of entailment allows presupposition of common knowledge, such as: a company has a CEO, a CEO is an employee of the company, an employee is a person, etc. For instance, in example #6, the entailment depends on knowing that the president of a country is also a citizen of that country.

Data Sets and Format

Both Development and Test sets are formatted as XML files, as follows:

<pair id=”id_num” entailment=”YES|NO” task=”IE|IR|QA|SUM” length=”short|long”>

<t>the text…</t>

<h>the hypothesis…</h>

</pair>

Where:

• each T-H pair appears within a single <pair> element.
• the element <pair> has the following attributes:
  • id, a unique numeral identifier of the T-H pair.
  • task, the acronym of the application setting from which the pair has been generated (see introduction): “IR”,”IE”,”QA” or “SUM”.
  • entailment (in the development set only), the gold standard entailment annotation, being either “YES” or “NO”.
  • length: long indicates when T is a longer snippet.
• the element <t> (text) has no attributes, and it may be made up of one or more sentences.
• the element <h> (hypothesis) has no attributes, and it usually contains one simple sentence.

The data is split to a development set and a test set, to be released separately. The goal of the development set is to guide the development and tuning of participating systems. Notice that since the given task has an unsupervised nature it is not expected that the development set can be used as a main resource for supervised training, given its anecdotal coverage. Rather it is typically assumed that systems will be using generic techniques and resources.

Result Submission

Systems must tag each T-H pair as either “YES”, predicting that entailment does hold for the pair, or as “NO” otherwise. No partial submissions are allowed, i.e. the submission must cover the whole dataset.

Systems should be developed based on the development data set. Analyses of the test set (either manual or automatic) should not impact in any way the design and tuning of systems that publish their results on the RTE-3 test set. We regard it as acceptable to run automatic knowledge acquisition methods (such as synonym collection) specifically for the lexical and syntactic constructs that will be present in the test set, as long as the methodology and procedures are general and not tuned specifically for the test data. In any case, participants are asked to report about any process that was performed specifically for the test set.

Evaluation Measures

The evaluation of all submitted runs will be automatic. The judgments (classifications) returned by the system will be compared to those manually assigned by the human annotators (the Gold Standard). The percentage of matching judgments will provide the accuracy of the run, i.e. the fraction of correct responses.

As a second measure, an Average Precision measure, will be computed. This measure evaluates the ability of systems to rank all the T-H pairs in the test set according to their entailment confidence (in decreasing order from the most certain entailment to the least certain). The more the system is confident that T entails H, the higher the ranking is. A perfect ranking would place all the positive pairs (for which the entailment holds) before all the negative pairs. Average precision is a common evaluation measure for system rankings, and is computed as the average of the system’s precision values at all points in the ranked list in which recall increases, that is at all points in the ranked list for which the gold standard annotation is YES. More formally, it can be written as follows:

1/R * sum for i=1 to n (E(i) * #-entailing-up-to-pair-i/i)

where n is the number of the pairs in the test set, R is the total number of positive pairs in the test set, E(i) is 1 if the i-th pair is positive and 0 otherwise, and i ranges over the pairs, ordered by their ranking. This score will be computed for systems that will provide as output a confidence-ranked list of all test examples (in addition to the YES/NO output for each example).

Result Submission Format

Results will be submitted in a file with one line for each T-H pair in the test set, in the following format:

pair_id<blank space>judgment

where

• pair_id is the unique identifier of each T-H pair, as it appears in the test set.
• judgment is either “YES” or “NO”.

The first line in the file should look as follows:

ranked:<blank space>”YES|NO”

The first line indicates whether the submission includes confidence ranking of the pairs (see evaluation measures above). Average precision will be computed only for systems that specify “ranked: YES” in the first line. If the submission includes confidence ranking, the pairs in the file should be ordered by decreasing entailment confidence order: the first pair should be the one for which the entailment is most certain, and the last pair should be the one for which the entailment is least likely (i.e. the one for which the judgment as “NO” is the most certain). Thus, in a ranked list all the positively classified pairs are expected to appear before all those that are classified as negative.

Each submitted run must be a plain text file, with a filename composed of a unique 5-6 element alpha-numeric string, and the number of run separated by “-“, e.g. CLCT07-run1.txt. Participating teams will be allowed to submit 2 result files per system. The corresponding result files should be named XXXXX-run1.txt, and XXXXX-run2.txt if a second run is submitted for the same system.

The results files should be zipped and submitted via email to infocelct at itc . it, with the subject line “[RTE3 RESULT SUBMISSION]“.

System Reports

Participants are requested to submit a full system report by March 26, 2007. It should be noted that the change of schedule for paper submissions, from April 2 to March 26, 2007, was due to the merging of the RTE workshop with the Paraphrasing Workshop, which will result in unifying the reviewing process of the two types of papers ( system and technical ) and make it more competitive for RTE system reports. As the schedule is quite tight, we suggest preparing a draft of the paper in advance of receiving results of the system evaluation. Report submissions will follow the same procedure for article submissions as for the main workshop (using the START system). Report submissions must be uploaded by filling out the submission form at the following URL: www.softconf.com/acl07/ACL07-WS9/submit.html. Please remember to select the “RTE3 paper” option in the Submission category field.

Reports should include up to 6 double column pages, using the ACL Style files and guidelines. As the reports presented at the workshop are expected to be very informative, in order to further explore entailment phenomena and any alternative methods of approaching it, we suggest an analysis of results and a presentation of any general observations you might have about the entailment recognition problem, in addition to the straightforward system description.

Due to workshop time limitations this year, not all system reports will be presented orally. The reviewing of RTE3 system reports will be blind and the papers should not include the authors’ names and affiliations. The best will presented orally, while the remaining papers which pass a sufficient level of quality will be presented as posters. Reviews with comments for the camera ready version and decisions about presentation in the workshop will be sent back to the authors by April 23, 2007.

The camera ready version of each report, to be included in the workshop proceedings, should be submitted in pdf format (with no page numbers) by May 6, 2007.

Evaluation

The evaluation of all submitted runs will be automatic. The judgments (classifications) returned by the system will be compared to those manually assigned by the human annotators (the Gold Standard). The percentage of matching judgments will provide the accuracy of the run, i.e. the fraction of correct responses.

As a second measure, an Average Precision measure will be computed. This measure evaluates the ability of systems to rank all the T-H pairs in the test set according to their entailment confidence (in decreasing order from the most certain entailment to the least certain). The more the system is confident that T entails H, the higher the ranking is. A perfect ranking would place all the positive pairs (for which the entailment holds) before all the negative pairs. Average precision is a common evaluation measure for system rankings, and is computed as the average of the system’s precision values at all points in the ranked list in which recall increases, that is at all points in the ranked list for which the gold standard annotation is YES (Voorhees and Harman, 1999). More formally, it can be written as follows:
1/R * sum for i=1 to n (E(i) *#-correct-up-to-pair-i/i)

where n is the number of the pairs in the test set, R is the total number of positive pairs in the test set, E(i) is 1 if the i-th pair is positive and 0 otherwise, and i ranges over the pairs, ordered by their ranking.

Note the difference between this measure and the Confidence Weighted Score used in the first challenge.

This score will be computed for systems that will provide as output a confidence-ranked list of all test examples (in addition to the YES/NO output for each example).

Organisers

Danilo Giampiccolo, CELCT, Trento
Ido Dagan, Bar Ilan University
Bill Dolan, Microsoft Research
Bernardo Magnini, ITC-irst, Istituto per la Ricerca Scientifica e Tecnologica, Povo (TN), Italy

is the current motto of machine learning. Therefore, assessing the real added value of prior/domain knowledge is a both deep and practical question. Most commercial data mining programs accept data pre-formatted as a table, each example being encoded as a fixed set of features. Is it worth spending time engineering elaborate features incorporating domain knowledge and/or designing ad hoc algorithms? Or else, can off-the-shelf programs working on simple features encoding the raw data without much domain knowledge put out-of-business skilled data analysts?

In this challenge, the participants are allowed to compete in two tracks:

• The “prior knowledge” track, for which they will have access to the original raw data representation and as much knowledge as possible about the data.
• The “agnostic learning” track for which they will be forced to use a data representation encoding the raw data with dummy features.

zip results.zip *.resu *.conf *.mat

tar cvf results.tar *.resu *.conf *.mat; gzip results.tar

Description

The goal of the challenge is to identify the different Machine Learning (ML) methods proposed so far for structured data, to assess the potential of these methods for dealing with generic ML tasks in the structured domain, to identify the new challenges of this emerging field and to foster research in this domain. Structured data appears in many different domains. We will focus here on Graph document collections and we are organizing this challenge in cooperation with the INEX initiative. This challenge aims at gathering ML, Information Retrieval (IR) and Data Mining researchers in order to:

• Define the new challenges for structured data mining with ML techniques.
• Build Interlinked document collections, define evaluation methodologies and develop software which will be used for the evaluation of classification of documents in a graph.
• Compare existing methods on different datasets.

Results of the track will be presented at the INEX workshop.

 Task : Graph (Semi-)Supervised Classification

Dealing with XML document collections is a particularly challenging task for ML and IR. XML documents are de¯ned by their logical structure and their content (hence the name semi-structured data). Moreover, in a large majority of cases (Web collections for example), XML documents collections are also structured by links between documents (hyperlinks for example). These links can be of different types and correspond to different nformation: for example, one collection can provide hierarchical links, hyperlinks, citations, etc.

Earlier models developed in the field of XML categorization/clustering simultaneously use the content information and the internal structure of XML documents for a list of models) but they rarely use the external structure of the collection i.e the links between documents.

We focus here on the problem of classication of XML documents organized in graph. More precisely, the participants of the task have to classify the document of a partially labelled graph.

Tasks

Collection

The corpus used this year will be a subset of the Wikipedia XML Corpus of INEX 2009. This subset will be different than the one used last year. Mainly:

• Each document will belong to one or more than one categories
• Each document will be and XML document
• The different documents will be organized in a graph of documents where each link correspond to an hyperlink (or wiki link) between two documents

The corpus proposed is a graph of XML documents.

Semi supervised classification

In this track, the goal is to classify each node of a graph (a node corresponds to a document) knowing a set of already labelled nodes (the training documents). In the ML point of view, the track proposed here is a transductive (or semi) supervised classification task.

The following figure gives an example of classification task.

Training set: The training set is composed of XML documents organized in a graph. The red nodes correspond to documents in category 1, the blue nodes corresponds to documents in category 2. The white nodes correspond to documents where the category is hidden. The goal of the categorization task is to find the categories of the white nodes

The goal of the categorization models are to find the color of the unlabelled nodes of the training graph.

The evaluation measure for categorization will be ROC curves and F1 measure

 perl compute.pl all_categories.txt train_categories.txt yourSubmissionFile

Development Kit

The development kit provided for the VOC challenge 2006 is available. You can:

• Download the development kit code and documentation (111KB gzipped tar file)
• Download the PDF documentation (117KB PDF)
• Browse the HTML documentation
• View the guidelines used for annotating the database

Datasets

Following completion of the challenge, the training/validation sets with complete annotation have now been made available on the database page. Some additional example images are online.

The VOC2006 data includes some images provided by Microsoft Research Cambridge and “flickr”. Use of these images must respect the corresponding terms of use:

• MSR Cambridge License (RTF)
• “flickr” terms of use

Details of the contributor of each image can be found in the file “contrib.txt” included in the dataset. Any queries about the use or ownership of the data should be addressed to the organizers.

Second Round

The second round aims to give an opportunity for groups who participated to submit additional results, and for others who might have been unable to meet the first deadline to participate. The deadline for submission of results for the second round is 11pm GMT, Friday 30th June 2006.

The second round will be run as per the first round. In evaluating the second round, submitted results will be evaluated separately from the first round to give consideration for the additional time available. Comparison of the results obtained between the first and second rounds will follow. As in the first round, ground truth annotation will not be provided for the test data.

On completion of the second round, the test set annotation will be released for researchers wishing to use the data for further evaluation.

Post-Challenge Submissions

We invite submission of results obtained after the completion of the challenge, both by participants and those who did not participate in the challenge. Results will be treated as distinct from the results obtained during the challenge due to the availability of the test data annotation. If you would like your results included in the online summary of results, please contact Mark Everingham, me@comp.leeds.ac.uk.

Results

A technical report summarizing the challenge and results is now available. The results may also be browsed online. Slides presented at the challenge workshop can also be downloaded.

• Technical report summarizing challenge and results (641K PDF)
• Browse the results online
• Slides presented at the challenge workshop (3M PDF)

If you make use of the VOC2006 data, please cite the following reference in any publications:

    @misc{pascal-voc-2006,
      author = "Everingham, M. and Zisserman, A. and Williams, C. K. I. and Van~Gool, L.",
      title = "The {PASCAL} {V}isual {O}bject {C}lasses {C}hallenge 2006 {(VOC2006)} {R}esults",
      howpublished = "http://www.pascal-network.org/challenges/VOC/voc2006/results.pdf"
    }

Timetable

• 14 Feb 2006 : Development kit (training and validation data plus evaluation software) made available.
• 31 March 2006: Test set made available.
• 25 April 2006, 11pm GMT: DEADLINE for submission of results. There will be no further extensions.
• 7 May 2006: Challenge workshop to be held in conjunction with ECCV06, Graz, Austria. The programme is now online.
• 30 June 2006, 11pm GMT: DEADLINE for submission of second round results.
• After 1 July 2006: Test set annotation made available.

Introduction

The goal of this challenge is to recognize objects from a number of visual object classes in realistic scenes (i.e. not pre-segmented objects). It is fundamentally a supervised learning learning problem in that a training set of labelled images is provided. The ten object classes that have been selected are:

• bicycle, bus, car, motorbike
• cat, cow, dog, horse, sheep
• person

There will be two main competitions:

1. For each of the ten classes, predicting presence/absence of an example of that class in the test image.
2. Predicting the bounding box and label of each object from the ten target classes in the test image.

Participants may enter either (or both) of these competitions, and can choose to tackle any (or all) of the ten object classes. The challenge allows for two approaches to each of the competitions:

1. Participants may use systems built or trained using any methods or data excluding the provided test sets.
2. Systems are to be built or trained using only the provided training data.

The intention in the first case is to establish just what level of success can currently be achieved on these problems and by what method; in the second case the intention is to establish which method is most successful given a specified training set.

Data

The training data provided consists of a set of images; each image has an annotation file giving a bounding box and object class label for each object in one of the ten classes present in the image. Note that multiple objects from multiple classes may be present in the same image. Some example images can be viewed online.

Annotation was performed according to a set of guidelines distributed to all annotators. These guidelines can be viewed here.

The data will be made available in two stages; in the first stage, a development kit will be released consisting of training and validation data, plus evaluation software (written in MATLAB). One purpose of the validation set is to demonstrate how the evaluation software works ahead of the competition submission.

In the second stage, the test set will be made available for the actual competition. In contrast to the 2005 VOC challenge, no ground truth for the test data will be released until after the challenge is complete.

Following completion of the challenge, the training/validation sets with complete annotation have now been made available on the database page.

The table below summarizes statistics of the training/validation and test data. The data has been split into 50% for training/validation and 50% for testing. The distributions of images and objects by class are approximately equal across the training/validation and test sets. In total there are 5304 images, containing 9507 annotated objects.

Database Rights

The VOC2006 data includes some images provided by Microsoft Research Cambridge and “flickr”. Use of these images must respect the corresponding terms of use:

• MSR Cambridge License (RTF)
• “flickr” terms of use

For the purposes of the challenge, the identity of the images in the database, e.g. source and name of owner, was obscured. Details of the contributor of each image can be found in the file “contrib.txt” in the final release of the data. Any queries about the use or ownership of the data should be addressed to the organizers.

Submission of Results

Participants are expected to submit a single set of results per method employed. Participants who have investigated several algorithms may submit one result per method. Changes in algorithm parameters do not constitute a different method – all parameter tuning must be conducted using the training and validation data alone.

Details of the required file formats for submitted results can be found in the development kit documentation. For each competition you should produce a single text file, named following the examples in the development kit. For example, your results file for competition 1 (classification trained on VOC data) on the “car” class should be named “comp1_cls_test_car.txt”; your results file for competition 4 (detection trained on your own data) on the “car” class should be named “comp4_det_test_car.txt”.

If you intend to submit results for several different methods, please place results for each in a separate, clearly named directory. For each method you submit, please include a brief description of the approach used. If you wish to withhold details of the method, e.g. for commercial reasons, please make this clear. Methods submitted without a description will be judged in a separate category of the challenge.

The results files should be collected in a single archive file (tar/zip) and placed on an FTP/HTTP server accessible from outside your institution. Email the URL and any details needed to access the file to Mark Everingham, me@comp.leeds.ac.uk. Please do not send files directly by email.

Publication Policy

The main mechanism for dissemination of the results will be the challenge webpage. It is also possible that an overview paper of the results will be produced.

If you make use of the VOC2006 data, please cite the technical report of results, see the results section.

Organizers

• Mark Everingham (Oxford/Leeds), me@comp.leeds.ox.ac.uk
• Luc van Gool (Zurich)
• Chris Williams (Edinburgh)
• Andrew Zisserman (Oxford)

Challenge description

There are two main goals for this challenge:

• to test and compare different text visualization methods, ideas and algorithms on a common data-set, and
• to contribute to the Pascal dissemination and promotion activities by using data about scientific publications from Pascal’s EPrints server.

The challenge is split into two parts where the task in the first round is very general. A more precise task, together with precise evaluation criterias, will be decided at the workshop in Venice, based on the submissions and ideas from the first round.

The data is available in the raw ePrints format. Besides it we offer a preprocessed version of the raw data into many standard formats. We also processed the data into a “baseline” visualization which can be viewed with the Document Atlas software. In this way we have split the whole pipeline of visualization process into more steps and allow the participants to work only on the parts of their interests. They can either:

• process the provided data to generate a better input for Document Atlas,
• visualize the provided processed baseline data differently and nicer than Document Atlas, or
• visualize the raw or pre-processed data on a novel way,

with the goal of:

• discovering main areas covered by the papers,
• discovering area developments trough time,
• helping the researchers with recommendation on which papers to read,
• helping at finding the right reviewers for a new papers.

Rules

Anyone is allowed to participate. A participant may be either a single person or a group. A single person can participate in at most two groups. A participant is allowed to submit at most three visualizations made out of any of the provided data. If a participant submits more than three solutions, the last three will be used at evaluation.

Organizers

• Blaz Fortuna, IJS, Slovenia
• Marko Grobelnik, IJS, Slovenia
• Steve R. Gunn, University of Southampton, UK

Description

Given is the data-set with approx. 720 publications uploaded onto Pascal EPrints-Server. The data-set was provided by Steve Gunn.

The following fields are available for each publication from the Eprints-Server:

• Date — date of publication
• Title — short text description
• Authors’ names — normalized and consolidated names (one person has just one surface form)
• Authors’ institutions — normalized and consolidated names (one institutions has just one surface form)
• Abstract — on average, 10-20 lines of text

The documents are sorted by the date of publication with the earlies paper being on the top of the files in all the formats.

We also provide processed data which can be transformed as to an input for Document Atlas visualization tool. You can substitute some parts of the input (document positions, landscape matrix, keywords) with your own results and just use the Document Atlas as a front end (GUI) for your visualization. All the necessary software can be downloaded from this page.

Downloads — data

The data-set is available in the following formats:
(for format descriptions check the bottom of this page!)

Downloads — software

Format descriptions

Here are details about formats in which the upper data is made available:

• XML — easy to read XML format with list of publications.
• MATLAB-bow — sparse term-document matrix (matrix with TFIDF vectors of documents as columns) which can be loaded with Matlab. It includes map of publications/authors/institutions to column numbers and of words to row numbers.
• text — one big text file where each line represent one document with the first word in line being the document’s name.
• MATLAB-graph — sparse adjacency matrix which can be loaded with Matlab. It includes map of words/authors/institutions to vertices number
• Pajek — format appropriate for usage in “Pajek” network analysis package.
• GraphML — standard XML format for storing graphs, to be available soon.

More on Matlab formats:
Files for Matlab are stored in a format, which can be easily transformed to Matlab’s sparse matrix using spconvert command.
Example which loads term-document matrix with papers:
load papers.dat
X = spconvert(papers);
In files with maps, word in i-th line of file corresponds to the lable of i-th element (row or column in matrix or vertex in graph).

Document Atlas:
Visualization in Document Atlas consist of more parts and in order to generate input for it using Txt2VizMap utility they must be provided (each in separate file). Here is the list of parts together with description of their file formats:

• documents — one big text file where each line represent one document with the first word in line being the document’s name.
• documents’ positions — text file containing documents positions. First line of the file holds a list of x coordinates and the second line of the file holds a list of y coordinates. The coordinates should be provided in the same order as the documents in the upper file. All the positions should be normalised so that 0<=x<=1 and 0<=y<=1.
• landscapes — matrix containing the height points of the landscape. One line of the text file holds one row of matrix. The top row of matrix corresponds to the first line in file. The top left corner of the matrix corresponds to the top left corner of the screen and the bottom right corner of the matrix corresponds to the bottom right corner of the screen.
• keywords — list of keywords that appear on the landscape in the Document Atlas. One line of text file corresponds to one word with word being the first element in the line followed by its x and y coordinates.

Note that only file with documents and file with their positions have to be provided. If landscapes are not provided they are calculated using standard Document Atlas method. Same holds for the keywords.

More landscapes can be provided and the user can choose between them inside the Document Atlas. Name of the landscape show in the program corresponds to its file name. A series of landscapes can be used for example to bring a time component into the visualization.

Example: the following command line creates Document Atlas input file from the processed data available from this page:
Txt2VizMap.exe -inlndoc:papers.txt
-ipoint:txt_papers.points.dat
-ils:txt_papers.Landscape1.dat;txt_papers.Landscape2.dat
-ikeywd:txt_papers.keywd.dat -o:txt_papers.VizMap

Evaluation criteria

For evaluation of the submitted results in the second round, we plan to use 4 criteria:

• Usability of visualization — The goal is to assess usability of particular visualization in different practical contexts.
• Innovativeness — The goal is to estimate how innovative are the ideas used for visualization.
• Aesthetics of the image — Here we are aiming to identify the “nicest” images from the challenge.
• General Pascal-researchers’ voting over the web about “who likes what”.

Since all the criteria are subjective, we will hire experts for judging about the quality. Each of the criteria will generate a separate results ranking.

Videos

Videos are here: http://videolectures.net/msht07_paris/

Registration

Registration is free. Still, please email olivier.teytaud@inria.fr to facilitate the organization of the workshop.

Schedule

The preliminary schedule is available at http://www.lri.fr/~teytaud/sched.html.

Accomodation info

(more information coming soon)

(if booking is difficult for language-reasons, please feel free of emailing “teytaud@lri.fr” for taking care of booking)

All hotels below are in the heart of Paris.

• Hotel du marais tel — not very close but not very far and much less expensive — from 25 euros/night to 35 euros/night
• Hotel de Lille, 40 r Lille 75007 Paris, minimum 100 euros/night, tel. (very very close to the workshop)

See also the list of hotels here (from 41 euros/night): http://www.dma.ens.fr/~stoltz/MFLT/Accomodation.html; not very far.

Please feel free of requesting some help (email to olivier.teytaud@inria.fr)

Call For Papers

Multiple Simultaneous Hypothesis Testing is a main issue in many areas of information extraction:

• rule extraction [6],
• validation of genes influence [3],
• validation of spatio-temporal patterns extraction (e.g. in brain imaging [7]),
• other forms of spatial or temporal data (e.g. spatial collocation rule, [8]).
• other multiple hypothesis testing ([4]),

In all above frameworks, the goal is to extract patterns such that some quantity of interest is significantly greater than some given threshold.

• in rule extraction, the goal typically is the extraction of rules with confidence, lift and support significantly higher than a given threshold;
• in multiple hypothesis testing, the goal typically is the extraction of significant comparisons among various averages simultaneously;
• in spatio-temporal patterns extraction, the goal typically is the extraction of smooth (spatio-temporal) subsets of with correlation significantly higher than a given threshold.

Along these lines, a type I error is to extract an entity which does not satisfy the considered constraint while a type II error is to miss an entity which does satisfy the constraint. How to estimate, bound, or (even better !) reduce type I and type II errors are the goals of the proposed challenge.

VC-theory [2], empirical process [5] and various approaches related to simultaneous hypothesis testings [4] are fully relevant, as well as specific approaches, e.g. based on simulations, resamplings or probes [9]. The challenge consists in extending previous results to the field of simultaneous hypothesis testing, or proposing new results specifically related to this topic.

We welcome survey papers related to type I and type II errors, and papers presenting new results, proposing theoretical bounds or smart empirical experiments. In the latter case, the experimental setting as well as the algorithmic principles and explicit criteria must be carefully described and discussed; the use of publicly available software will be much appreciated.

Results combining type I and type II risk are particularly welcome. Asymptotic and non-asymptotic results are equally welcome.

• workshop dates : May, 15-16;
• subject : type I or type II risk in multiple simultaneous hypothesis testing ;
• dead line : February, 25;
• submission : by email to “olivier.teytaud@inria.fr” ;
• format: 18 pages (LaTeX, llncs-style, http://www.springer.com/east/home/computer/lncs?SGWID=5-164-7-72376-0).
• extended versions of selected papers will be published in a special issue of an international machine learning journal.
• info: http://www.lri.fr/~teytaud/risq/risq.html
• info pdf : http://www.lri.fr/~teytaud/risq.pdf

Key words : Empirical process, Learning theory, Multiple hypothesis testing, Rule extraction, Bio-informatics, Statistical Validation of Information Extraction.

Organization

Important dates

• Diffusion of the challenge : January 11, 2006.
• Deadline for submissions : February 10, 2007.
• Notification of acceptance of submitted results : March 2007.
• Challenge Workshop : Paris, France; May, 15-16th, 2007

Submissions

Submissions (in PS or PDF) should be submitted by email to “olivier.teytaud@inria.fr”

Venue

• no fee.
• venue: Université Paris-5, 45 rue des Saints-Pères, Paris (downtown). Close to metro “Saint-Germain-des-Prés”.
  • large map: http://www.justfranceinparis.com/paris677map.html
  • local map: http://kenobi.univ-paris5.fr/cgi-bin/WebObjects/WebSiteAdmin.woa/wa/viewArticle?type=99&typeRoot=99&couNumero=794# (click on “Centre des Saints-Pères”)
  • another local map: http://www.biomedicale.univ-paris5.fr/Plan-d-acces.html
  • how to come:
    • Bus: see map above
    • Metro line 4, station “Saint-Germain-des-Prés”
    • Metro line 10, station “Mabillon”
    • Metro line 12, station “Rue du bac”
    • RER B, to station “Denfert Rochereau” or “Chatelet Les Halles”, and then Metro line 4, to station “Saint-Germain-des-Prés”
    • From Orly airport: metro “Orlyval” to Antony, and then RER B to station “Denfert Rochereau” or “Chatelet Les Halles”, and then Metro line 4, to station “Saint-Germain-des-Prés”.
    • From Roissy-Charles-De-Gaulle, RER B to station “Denfert Rochereau” or “Chatelet Les Halles”, and then Metro line 4, to station “Saint-Germain-des-Prés”.
    • RER A, to station “Chatelet Les Halles”, and then metro line 4 to station “Saint-Germain-des-Prés”
    • from other places: fill in the forms at http://www.ratp.fr.

Email for any information: olivier.teytaud@inria.fr.

Organizing committee

• Gérald Gavin (univ. Lyon 1);
• Sylvain Gelly (univ. Paris-Sud);
• Yann Guermeur (Cnrs, Loria);
• Stéphane Lallich (univ. Lyon 2);
• Jérémie Mary (univ. Lille);
• Michèle Sebag (Cnrs);
• Olivier Teytaud (Inria).

Bibliography

1

M. Antony and P.L. Bartlett, Neural network learning : Theoretical Foundations, Cambridge University Press, 1999.

2

V. N. Vapnik, Statistical Learning Theory, Wiley, 1998.

3

Merrill D. Birkner, Katherine S. Pollard, Mark J. van der Laan, and Sandrine Dudoit, “Multiple Testing Procedures and Applications to Genomics” (January 2005). U.C. Berkeley Division of Biostatistics Working Paper Series. Working Paper 168. http://www.bepress.com/ucbbiostat/paper168

4

J.C. Hsu, Multiple comparisons: theory and methods, Chapman & Hall, 1996.

5

Van Der Vaart A., Wellner J.A. Weak Convergence and Empirical Processes. Springer series in statistics, 1996.

6

R. Agrawal, T. Imielinski, and A. Swami. Mining Association Rules between Sets of Items in Large Databases. In Proceedings of SIGMOD-93, pages 207-216, 1993.

7

D.Pantazis, T.-E. Nichols, S. Baillet, R.-M. Leahy, “A Comparison of Random Field Theory and Permutation Methods for the Statistical Analysis of MEG data”, Neuroimage, 25, 355-368, April, 2005.

8

M. Salmenkivi. Efficient Mining of Correlation Patterns in Spatial Point Data. In Proceedings of PKDD 2006, pages 359-370.

9

H. Stoppiglia, G. Dreyfus, R. Dubois, Y. Oussar. Ranking a random feature for Variable and Feature selection. JMLR 2003.

This project is dedicated to stimulate research and reveal the state-of-the art in “model selection” by organizing a competition followed by a workshop. Model selection is a problem in statistics, machine learning, and data mining. Given training data consisting of input-output pairs, a model is built to predict the output from the input, usually by fitting adjustable parameters. Many predictive models have been proposed to perform such tasks, including linear models, neural networks, trees, and kernel methods. Finding methods to optimally select models, which will perform best on new test data, is the object of this project. The competition will help identifying accurate methods of model assessment, which may include variants of the well-known cross-validation methods and novel techniques based on learning theoretic performance bounds. Such methods are of great practical importance in pilot studies, for which it is essential to know precisely how well desired specifications are met.

zip results.zip *.resu *.conf *.guess *.mat

tar cvf results.tar *.resu *.conf *.guess *.mat; gzip results.tar

 Motivation

A fundamental phenomenon of natural language is the variability of semantic expression, where the same meaning can be expressed by or inferred from different texts. Many natural language processing applications, such as Question Answering (QA), Information Retrieval (IR), Information Extraction (IE), and (multi) document summarization need to model this variability in order to recognize that a particular target meaning can be inferred from different text variants. Even though many applications face similar underlying semantic problems, these problems are usually addressed in an application-oriented manner. Consequently it is difficult to compare, under a generic evaluation framework, semantic methods that were developed within different applications. The PASCAL RTE Challenge introduces textual entailment as a common task and evaluation framework, covering a broad range of semantic-oriented inferences needed for practical applications. This task is therefore suitable for evaluating and comparing semantic-oriented models in a generic manner. Eventually, work on textual entailment may promote the development of general semantic “engines”, which will be used across multiple applications.

Textual Entailment

Textual entailment recognition is the task of deciding, given two text fragments, whether the meaning of one text is entailed (can be inferred) from another text (see the Instructions tab for the specific operational definition of textual entailment assumed in the challenge). This task captures generically a broad range of inferences that are relevant for multiple applications. For example, a Question Answering (QA) system has to identify texts that entail the expected answer. Given the question “Who killed Kennedy?”, the text “the assassination of Kennedy by Oswald” entails the expected answer “Oswald killed Kennedy”. Similarly, in Information Retrieval (IR) the concept denoted by a query expression should be entailed from relevant retrieved documents. In multi-document summarization a redundant sentence or expression, to be omitted from the summary, should be entailed from other expressions in the summary. In Information Extraction (IE) entailment holds between different text variants that express the same target relation. And in Machine Translation evaluation a correct translation should be semantically equivalent to the gold standard translation, and thus both translations have to entail each other. Thus, modeling textual entailment may consolidate and promote broad research on applied semantic inference.

Task Definition

Participants in the evaluation exercise are provided with pairs of small text snippets (one or more sentences in English), which we term Text-Hypothesis (T-H) pairs. Examples were manually tagged for entailment (i.e. whether T entails H or not) by human annotators and will be divided into a Development Set, containing 800 pairs, and a Test Set, containing 800 pairs. Participating systems will have to decide for each T-H pair whether T indeed entails H or not, and results will be compared to the manual gold standard.

The goal of the RTE challenges is to provide opportunities for presenting and comparing possible approaches for modeling textual entailment. In this spirit, we aim at an explorative rather than a competitive setting. While participant results will be reported there will not be an official ranking of systems. A development set is released first to provide typical examples of the different types of test examples. The test set will be released three weeks prior to the result submission date. We regard it as acceptable to run automatic knowledge acquisition methods (such as synonym collection from corpora or the Web) specifically for the linguistic constructs that are present in the test data, as long as the methodology is general and fully automated, and the cost of running the learning/acquisition procedure at full scale can be reasonably estimated.

Dataset Collection and Application Settings

The dataset of Text-Hypothesis pairs was collected by human annotators. It consists of four subsets, which correspond to typical success and failure settings in different applications (as listed below). Within each application setting the annotators selected both positive entailment examples (annotated YES), where T does entail H, as well as negative examples (annotated NO), where entailment does not hold (50%-50% split). Some T-H examples appear in the Instructions section. H is a (usually short) single sentence, and T consists of one or two sentences.

One of the main goals for the RTE-2 dataset is to provide more realistic text-hypothesis examples, originating from actual applications. Therefore, the examples are based mostly on outputs of existing web-based systems (see Acknowledgments below). We allowed only minimal correction of texts extracted from the web, e.g. fixing spelling and punctuation but not style, therefore the English of some of the pairs is less-than-perfect.

Information Retrieval (IR):

In this application setting, the hypotheses are propositional IR queries, which specify some statement, e.g. “Alzheimer’s disease is treated using drugs”. The hypotheses were adapted and simplified from standard IR evaluation datasets (TREC and CLEF). Texts (T) that do or do not entail the hypothesis were selected from documents retrieved by different search engines (e.g. Google, Yahoo and Microsoft) for each hypothesis. In this application setting it is assumed that relevant documents (from an IR perspective) must necessarily entail the hypothesis.

Multi-document summarization (SUM):

In this setting T and H are sentences taken from a news document cluster, a collection of news articles that describe the same news item. Annotators were given output of multi-document summarization systems, including the document clusters and the summary generated for each cluster. The annotators picked sentence pairs with high lexical overlap, preferably where at least one of the sentences was taken from the summary (this sentence usually played the role of T). For positive examples, the hypothesis was simplified by removing sentence parts, until it was fully entailed by T. Negative examples were simplified in the same manner. This process simulates the need of a summarization system to identify information redundancy, which should be avoided in the summary.

Question Answering (QA):

Annotators were given questions and the corresponding answers returned by QA systems. Transforming a question-answer pair to text-hypothesis pair consisted of the following stages: First, the annotators picked from the answer passage an answer term of the expected answer type, either a correct or an incorrect one. Then, the annotators turned the question into an affirmative sentence with the answer term “plugged in”. These affirmative sentences serve as the hypotheses (H), and the original answer passage serves as the text (T). For example, given the question, “Who is Ariel Sharon?” and an answer text “Israel’s prime Minister, Ariel Sharon, visited Prague” (T), the question is turned into the statement “Ariel Sharon is the Israeli Prime Minister” (H), producing a positive entailment pair. This process simulates the need of a QA system to verify that the retrieved passage text indeed entails the provided answer.

Information Extraction (IE):

This task is inspired by the Information Extraction (and Relation Extraction) application, adapting the setting to having pairs of texts rather than a text and a structured template. The pairs were generated using 4 different approaches. In the first approach, ACE-2004 relations (the relations tested in the ACE-2004 RDR task) were taken as templates for hypotheses. Relevant news articles were then collected as texts (T) and corresponding hypotheses were generated manually based on the ACE templates and slot fillers taken from the text. For example, given the ACE relation ‘X work for Y‘ and the text “An Afghan interpreter, employed by the United States, was also wounded.” (T), a hypothesis “An interpreter worked for Afghanistan.” is created, producing a non-entailing (negative) pair. In the second approach, the MUC-4 annotated dataset was similarly used to create entailing pairs. In the third approach, the outputs of actual IE systems, for both the MUC-4 dataset and the news articles collected for the ACE relations, were used to generate entailing and non-entailing pairs. In the forth approach, new types of hypotheses that may correspond to typical IE relations were manually generated for different sentences in the collected news articles. These processes simulate the need of IE systems to recognize that the given text indeed entails the semantic relation that is expected to hold between the candidate template slot fillers.

 Challenge Organizers

Acknowledgments

The following sources were used in the preparation of the data:

• AnswerBus question answering system, provided by Zhiping Zheng, Computational Linguistics Department, Saarland University.
• PowerAnswer question answering system, from Language Computer Corporation, provided by Dan Moldovan, Abraham Fowler, Christine Clark, Arthur Dexter and Justin Larrabee.
• Columbia NewsBlaster multi-document summarization system, from the Natural Language Processing group at Columbia University’s Department of Computer Science.
• NewsInEssence multi-document summarization system, provided by Dragomir R. Radev and Jahna Otterbacher from the Computational Linguistics And Information Retrieval research group, University of Michigan.
• IBM’s information extraction system, provided by Salim Roukos and Nanda Kambhatla, I.B.M. T.J. Watson Research Center.
• New York University’s information extraction system, provided by Ralph Grishman, Department of Computer Science, Courant Institute of Mathematical Sciences, New York University.
• ITC-irst’s information extraction system, provided by Lorenza Romano, Cognitive and Communication Technologies (TCC) division, ITC-irst, Trento, Italy.
• MUC-4 information extraction dataset, from the National Institute of Standards and Technology (NIST).
• TREC-QA question collections, from the National Institute of Standards and Technology (NIST).
• CLEF-QA question collections, from DELOS Network of Excellence for Digital Libraries.

We would like to thank the people and organizations that made these sources available for the challenge. In addition, we thank Oren Glickman and Dan Roth for their assistance and advice.

We would also like to acknowledge the people and organizations involved in creating and annotating the data: Malky Rabinowitz, Dana Mills, Ruthie Mandel, Errol Hayman, Vanessa Sandrini, Allesandro Valin, Elizabeth Lima, Jeff Stevenson, Amy Muia and the Butler Hill Group.

Datasets

development set
Note: if you downloaded the RTE-2 development set between 26/3/2007 (after RTE-3 results submission deadline), and 27/4/2008, please download it again from the above link, as it was found that this link temporarily pointed to an incorrect version of the development set (there was no problem with the test set)

test set
annotated test set
Preprocessed versions of the development and test set, including sentence splitting and dependency parsing, can be found here.

Errata: a slightly modified version of the development set which fixes minor typological errors in two pairs: #378 and #480, is available from the Stanford NLP group:

development set – Stanford

RTE-1 datasets

Task Definition

We consider an applied notion of textual entailment, defined as a directional relation between two text fragments, termed T – the entailing text, and H – the entailed text. We say that T entails H if, typically, a human reading T would infer that H is most likely true. This somewhat informal definition is based on (and assumes) common human understanding of language as well as common background knowledge. Textual entailment recognition is the task of deciding, given T and H, whether T entails H.

Table 1: Example T-H pairs

Some additional judgment criteria and guidelines are listed below (examples are taken from Table 1):

• Entailment is a directional relation. The hypothesis must be entailed from the given text, but the text need not be entailed from the hypothesis.
• The hypothesis must be fully entailed by the text. Judgment would be NO if the hypothesis includes parts that cannot be inferred from the text.
• Cases in which inference is very probable (but not completely certain) are judged as YES. In example #5 one could claim that although Shapiro’s office is in Century City, he actually never arrives to his office, and works elsewhere. However, this interpretation of T is very unlikely, and so the entailment holds with high probability. On the other hand, annotators were guided to avoid vague examples for which inference has some positive probability which is not clearly very high.
• Our definition of entailment allows presupposition of common knowledge, such as: a company has a CEO, a CEO is an employee of the company, an employee is a person, etc. For instance, in example #6, the entailment depends on knowing that the president of a country is also a citizen of that country.

Data Sets and Format

Both Development and Test sets are formatted as XML files, as follows:

<pair id=”id_num” entailment=”YES|NO” task=”task_acronym”>

<t> the text… </t>

<h> the hypothesis… </h>

</pair>

Where:

• each T-H pair appears within a single <pair> element.
• the element <pair> has the following attributes:
• id, a unique numeral identifier of the T-H pair.
• task, the acronym of the application setting from which the pair has been generated (see introduction): ‘IR’,’IE’,’QA’ or ‘SUM’.
• entailment (in the development set only), the gold standard entailment annotation, being either ‘YES’ or ‘NO’.
• the element <t> (text) has no attributes, and it may be made up of one or more sentences.
• the element <h> (hypothesis) has no attributes, and it usually contains one simple sentence.

The data is split to a development set and a test set, to be released separately. The goal of the development set is to guide the development and tuning of participating systems. Notice that since the given task has an unsupervised nature it is not expected that the development set can be used as a main resource for supervised training, given its anecdotal coverage. Rather it is typically assumed that systems will be using generic techniques and resources.

Data preprocessing

Following requests made in the first RTE challenge, this year we preprocessed the text and hypothesis of each pair. The preprocessing includes sentence splitting, using MXTERMINATOR (Reynar and Ratnaparkhi, 1997) and dependency parsing using MINIPAR (Lin, 1998).  See the “Datasets” tab for further details. Using the pre-processed data is optional, and it is allowed, of course, to use alternative tools for preprocessing.  Note that since the preprocessing is done automatically it does contain some errors. We provide this data as-is, and give no warranty on the quality of the pre-processed data.

Submission

Systems should tag each T-H pair as either YES, predicting that entailment does hold for the pair, or as NO otherwise. As indicated originally, this year partial submissions are not allowed – the submission should cover the whole dataset.

Systems should be developed based on the development data set. Analyses of the test set (either manual or automatic) should not impact in any way the design and tuning of systems that publish their results on the RTE-2 test set. We regard it as acceptable to run automatic knowledge acquisition methods (such as synonym collection) specifically for the lexical and syntactic constructs that will be present in the test set, as long as the methodology and procedures are general and not tuned specifically for the test data. In any case, participants are asked to report about any process that was performed specifically for the test set.

Evaluation Measures

The evaluation of all submitted runs will be automatic. The judgments (classifications) returned by the system will be compared to those manually assigned by the human annotators (the Gold Standard). The percentage of matching judgments will provide the accuracy of the run, i.e. the fraction of correct responses.

As a second measure, an Average Precision measure will be computed. This measure evaluates the ability of systems to rank all the T-H pairs in the test set according to their entailment confidence (in decreasing order from the most certain entailment to the least certain). The more the system is confident that T entails H, the higher the ranking is. A perfect ranking would place all the positive pairs (for which the entailment holds) before all the negative pairs. Average precision is a common evaluation measure for system rankings, and is computed as the average of the system’s precision values at all points in the ranked list in which recall increases, that is at all points in the ranked list for which the gold standard annotation is YES (Voorhees and Harman, 1999). More formally, it can be written as follows:

1/R * sum for i=1 to n (E(i) *#-correct-up-to-pair-i/i)

where n is the number of the pairs in the test set, R is the total number of positive pairs in the test set, E(i) is 1 if the i-th pair is positive and 0 otherwise, and i ranges over the pairs, ordered by their ranking.

Note the difference between this measure and the Confidence Weighted Score used in the first challenge.

This score will be computed for systems that will provide as output a confidence-ranked list of all test examples (in addition to the YES/NO output for each example).

Results Submission Format

Results will be submitted in a file with one line for each T-H pair in the test set, in the following format:

pair_id<blank space>judgment

where

• pair_id is the unique identifier of each T-H pair, as it appears in the test set.
• judgment is either YES or NO.

The first line in the file should look as follows:

ranked:<blank space>yes/no

The first line indicates whether the submission includes confidence ranking of the pairs (see evaluation measures above). Average precision will be computed only for systems that specify “ranked: yes” in the first line. If the submission includes confidence ranking, the pairs in the file should be ordered by decreasing entailment confidence order: the first pair should be the one for which the entailment is most certain, and the last pair should be the one for which the entailment is least likey (i.e. the one for which the judgment as “NO” is the most certain). Thus, in a ranked list all the positively classified pairs are expected to appear before all those that were classified as negative.

For example, suppose that the pair identifiers in the test set are 1…6. A valid submission file that includes ranking may look as follows:

ranked: yes

4 YES

3 YES

6 YES

1 NO

5 NO

2 NO

Participating teams will be allowed to submit results of up to 2 systems. The corresponding result files should be named run1.txt, and run2.txt if a second run is submitted.

The results files should be zipped and submitted via the submit form.

System Reports

Participants are requested to submit a full system report by February 21 (we have a tight schedule this year due to the EACL 2006 conference in Trento, scheduled right before the RTE 2 Workshop).  Reports should include up to 6 double column pages, using the ACL style files and guidelines. The reports should include system description, quantitative results on the test set, and qualitative and quantitative analysis of the results and system behavior. Reviews with comments for the camera ready version and decisions about presentation in the workshop will be sent back to the authors on early March.

Reports are to be sent by email to Benardo Magnini (magnini@itc.it) with the following subject line: “RTE2 report submission”.

We aim to have an interactive and informative workshop setting, which will enable exploring the space of the entailment phenomenon and alternative methods to approach it. Therefore we encourage, apart from the straightforward system description, some analysis of results and general observations you might have about the entailment recognition problem. We strongly believe that to understand and interpret the results of a system the plain score is not sufficiently informative. In particular, we advocate including:

• General observations and analysis of the entailment phenomena, data types, problems, etc.
• Analysis of system performance – analysis and characterization of success and failure cases, identifying inherent difficulties and limitations of the system vs. its strengths.
• Description of the types of features used by the system; getting a feeling (with examples) for the concrete features that actually made an impact.
• Noting if there is a difference in performance between the development and test set. Identifying the reasons; was there over fitting?
• If your system (as described in the report) is complex – identifying which parts of the system were eventually effective vs. parts that were not crucial or even introduced noise, illustrating the different behaviors through examples.
• In case the development set was used for learning detailed information (like entailment rules) – discussing whether the method is scalable.
• Providing illustrative examples for system behavior.
• Providing results also for the development set.

Camera ready report:

Camera ready version of the report, to be included in the workshop proceedings, should be submitted in pdf format (with no page numbers) by March 14.

References

1998. Lin. 1998. Dependency-based evaluation of MINIPAR. In Proceedings of the Workshop on Evaluation of Parsing Systems at LREC 1998. Granada, Spain.
1999. C. Reynar and A. Ratnaparkhi. 1997. A Maximum Entropy Approach to Identifying Sentence Boundaries. In Proceedings of the Fifth Conference on Applied Natural Language Processing,March 31-April 3. Washington, D.C.
2000. Voorhees and D. Harman. 1999. Overview of the seventh text retrieval conference. In Proceedings of the Seventh Text Retrieval Conference (TREC-7). NIST Special Publication.

Results

Download RTE-2 evaluation script: RTE2_evaluate.pl (requires perl)

Usage: perl RTE2_evaluate.pl annotated_xml_filename <run_filename

(RTE-2 annotated test set is available here)

RTE-2 submitted runs and results

download all runs

* Resubmitted after publication of the official results. Resubmission was allowed only in case of a bug fix, so that the updated results are the correct output of the system described in the RTE-2 proceedings.

The objective of the Challenge is to design a statistical machine learning algorithm that segments words into the smallest meaning-bearing units of language, morphemes. Ideally, these are basic vocabulary units suitable for different tasks, such as text understanding, machine translation, information retrieval, and statistical language modeling.

The scientific goals are:

• To learn of the phenomena underlying word construction in natural languages
• To discover approaches suitable for a wide range of languages
• To advance machine learning methodology

The results will be presented in a workshop arranged in connection with other PASCAL challenges on machine learning. Please read the rules and see the schedule. The datasets are available for download. Instructions on how to submit your camera-ready documents are given on the Workshop page. We are looking forward to an interesting competition!

Organizers

• Mikko Kurimo, Mathias Creutz and Krista Lagus
• Neural Networks Research Centre, Helsinki University of Technology

Program comittee

• Levent Arslan, Boğaziçi University
• Samy Bengio, IDIAP
• Tolga Cilogu, Middle-East Technical University
• John Goldsmith, University of Chicago
• Kadri Hacioglu, Colorado University
• Chun Yu Kit, City University of Hong Kong
• Dietrich Klakow, Saarland University
• Jan Nouza,Technical University of Liberec
• Erkki Oja, Helsinki University of Technology
• Richard Wicentowski, Swarthmore College
• Murat Saraclar, Boğaziçi University

References

Mathias Creutz and Krista Lagus (2005). Unsupervised Morpheme Segmentation and Morphology Induction from Text Corpora Using Morfessor 1.0. Publications in Computer and Information Science, Report A81, Helsinki University of Technology, March.

Teemu Hirsimäki, Mathias Creutz, Vesa Siivola, Mikko Kurimo, Janne Pylkkönen, and Sami Virpioja (2005). Unlimited vocabulary speech recognition with morph language models applied to Finnish. Preprint accepted for publication in Computer Speech and Language.