Task Guideline

From Ritewiki

Jump to: navigation, search

Last updated: Sep 30, 2011.

Contents

Introduction

NTCIR-9 RITE Tasks

RITE has four subtasks BC, MC, Entrance Exam & RITE4QA.

BC subtask result is mandatory to submit, whereas other subtasks are optional (although we strongly recommend you to submit a result to the MC subtask). Evaluation will be conducted automatically using a tool included in RITE SDK

BC Subtask

Given a text pair (t1,t2) identify whether t1 entails (infers) a hypothesis t2 or not.

MC Subtask

A 5-way labeling subtask to detect (forward / reverse / bidirection) entailment or no entailment (contradiction / independence) in a text pair.

Entrance Exam Subtask

Same as the BC subtask in terms of input and output. All the data is created from actual college-level entrance exams.

Please submit an additional agreement form if you haven't done so yet. Please download the form, make two copies in double-sided print, fill them in, and send them to the following address.

NTCIR Project Office (Rm.1309)
National Institute of Informatics
2-1-2 Hitotsubashi Chiyoda-ku, 
Tokyo 101-8430, JAPAN

RITE4QA Subtask

Same as the BC subtask in terms of input and output, but as an embedded answer validation component in Question Answering system. This way, an impact of a RITE BC component to an overall end-to-end application can be measured.


How RITE4QA data is generated

Source: QA question/answer dataset (JA & CT) from past NTCIR + a run result from a good QA system (up to 5 answer candidates per question, each with source document ID).

CS data is transliterated from CT.


Note:

Data Format

Development (training) and test (formal run) data will be provided at this website in the following xml format in all four subtasks.

Note: the label attribute is not contained for the test data.

<dataset>
  <pair id="1" label="Y">
    <t1>アテネの市域の中心にアクロポリスの丘、北東部にリュカベットス山がそびえ、パルテノン神殿、聖イヨルイヨス礼拝堂などがある。</t1>
    <t2>パルテノン神殿の建つ丘は,アクロポリスと呼ばれている。</t2>
  </pair>
  <pair id="2" label="N">
    <t1>パルテノン神殿は、ドーリア式神殿の最高傑作と言える作品である。</t1>
    <t2>パルテノン神殿は,ヘレニズム文化の影響下で建設された。</t2>
  </pair>
: : : 
</dataset>


Submission Format

Each line represents an output for a text pair

(Text Pair ID)[SPACE](Label)[SPACE](Confidence)[CR]

Example:

1 Y 0.852
2 Y 0.943
3 Y 0.993
4 Y 1.000

Confidence score in the third column takes a real number between 0 and 1. In BC and MC subtask, the confidence column is optional (but recommended). In Entrance Exam and RITE4QA subtask, confidence column is going to be required for tie-breaking multiple Y labels on series of pairs on a certain topic.

Formal run

Please freeze the system development before you open the formal run data. Once the system is frozen, only trivial bugs such as output formatting bug are allowed to be fixed.

Number of runs

For each subtask, you can submit up to 3 runs.

Submission method

Store each run result in the format described above. After verifying the format is correct, use the following Run ID for the file name for each run.

RITE1-(Team Name)-(Lang)-(Subtask Name)-(Run Number)

Example (.txt extension allowed):

RITE1-LTI-JA-BC-01.txt

Please also create a text file with has a paragraph-length brief system description, with information on tools (NLP tools etc), and resources (any language resource, web etc) used.

Example:

RITE1-LTI-JA-BC-01
Tools: MeCab morphological analyzer, in-house statistical dependency parser.
Resources: Japanese wordnet, Goitaikei
Method: Extracted a set of morphemes from t1 and t2, and decide on the label 
based on the percentage of overlap between the sets of morphemes.

RITE1-LTI-JA-BC-02
Tools: Same as RITE1-LTI-JA-BC-01
Resources: Same as RITE1-LTI-JA-BC-01
Method: Same as RITE1-LTI-JA-BC-01 except that the threshold parameter is 
automatically tuned from parameter sweep on training data.

Archive all files in zip, and send the result to ntc9-rite-organizers [at] cs.cmu.edu via email attachment by the end of the formal run period (9/25 23:59 for BC and MC subtask; TBD for the rest of subtasks).

We'll email you a notification of acceptance as a reply within 24 hours of submission. Contact us if you did not receive the notification.

Writing a Paper

Start preparing for your participant paper, using the ACM conference style files (unchanged since NTCIR-8). Participant paper length limit is 8 pages.

You will receive a DRAFT task overview paper from your task organizers in Sep around formal run period (so that participant paper do not need to duplicate task description etc). Cite this overview, and submit your DRAFT participant paper to EasyChair by Oct 1, 2011. In a draft version, please include at least algorithms, tools, and resources you used (so we can summarize trends in organizer's paper). We're NOT reviewing papers; we are only providing suggestions for improving the paper. It is OK even if some parts (e.g. evaluation, Entrance Exam and RITE4QA subtask part etc) are missing at this point due to time constraint. Let us know if you may not be able to make it by this draft due.

Submit your camera ready paper by Nov 4, 2011. There will be no deadline extension for this camera ready due. All papers are accepted in the NTCIR proceedings. (Since NTCIR-9 is a nonrefereed workshop, we encourage you to extend the paper and submit it to a good conference.)

For more details (e.g. style template, paper submission instructions), please visit http://research.nii.ac.jp/ntcir/ntcir-9/papers.html

Items expected in your paper

If you’re not sure how to write a participant paper, please have a look at papers from previous NTCIRs: http://research.nii.ac.jp/ntcir/publication1-en.html

Attending the NTCIR-9 meeting

Personal tools
Namespaces
Variants
Actions
Menu (English)
メニュー (日本語)
目次 (繁體中文)
Navigation
Toolbox