Main Page

From Ritewiki

Overview

RITE is a generic benchmark task that addresses major text understanding needs in various NLP/Information Access research areas. We will evaluate systems which automatically detect entailment, paraphrase, and contradiction in texts written in Japanese, Simplified Chinese, or Traditional Chinese. In BC subtask, an input text pair looks as follows:

t₁: Yasunari Kawabata won the Nobel Prize in Literature for his novel "Snow Country"

t₂: Yasunari Kawabata is the writer of "Snow Country"

A system is given a credit for this pair if it returns the same label as human assessors'; in this specific case, "Y" (meaning "Yes", a human reading the premise t₁ would infer that the hypothesis t₂ is most likely true).

NTCIR-9 RITE Subtasks

• Binary-class (BC) subtask - Given a text pair (t₁,t₂) identify whether t₁ entails (infers) a hypothesis t₂ or not.
• Multi-class (MC) subtask - 5-way labeling task to detect (forward / reverse / bidirection) entailment or no entailment (contradiction / independence) in a text pair.
• Entrance Exam subtask (Japanese Only) - Same as the BC subtask in terms of input and output. All the data is created from actual college-level entrance exams.
• RITE4QA subtask - Same as the BC subtask in terms of input and output, but as an embedded answer validation component in Question Answering system. This way, the impact of RITE to an overall end-to-end application can be measured.

Who should participate?

RITE is a generic task that addresses major Text Understanding needs in various research areas e.g. Information Retrieval, Question Answering, Text Summarization, Opinion Analysis. Researchers in Natural Language Processing, Machine Learning community are also very welcome. Since RITE is a friendly evaluation effort, do not hesitate to try ambitious/exploratory approaches that do not necessarily work. A uniqueness in your method would also be a big contribution to the community. Don't worry about the lack of language resource/tool to participate. We will build the Resource Pool on this wiki, so that useful resources can be shared among participants, and they can be used to build a system for the formal evaluation.

Menu

• Task Guideline defines the task in more detail, such as input/output format.
• Data - sample data, development data (and formal data during the formal run period) are downloadable.
• Resource Pool contains useful material for participating in RITE.
• Detailed Schedule - contains detailed schedule.
• RITE-SDK - a Java framework which enables one to build a RITE system easily.

Current Status

• Formal run for all 4 subtasks are done. We received nearly 200 runs from 24 teams, which is one of the largest numbers in a single NTCIR task ever! The NTCIR-9 workshop has been held in Dec 2011. You can access an online proceedings here.

Schedule Date Event 2011-01~03 Training data development 2011-05- Dry run 2011-04-06 Test data development 2011-09 19-25 Formal run for BC & MC subtask 2011-09-27 Evaluation results return for BC & MC 2011-10-01 Participant paper draft due 2011-10-04 Training data release for EXAM 2011-10 12-19 Formal run for EXAM & RITE4QA 2011-11-04 Participant paper due (firm deadline) 2011-12 NTCIR-9 Workshop Meeting Detailed schedule is available here. Mailing list If you are interested in the task, join our mailing list by sending an empty email to ntc9-rite-list-subscribe [at] cs.cmu.edu

Organizers Language Organizers Japanese Hideki Shima (Carnegie Mellon University) Teruko Mitamura (Carnegie Mellon University) Hiroshi Kanayama (IBM Research - Tokyo) Koichi Takeda (IBM Research - Tokyo) Yusuke Miyao (National Institute of Informatics) Simplified Chinese Shuming Shi (Microsoft Research Asia) Traditional Chinese Chuan-Jie Lin (National Taiwan Ocean University) Cheng-Wei Lee (Academia Sinica) Email ntc9-rite-organizers [at] cs.cmu.edu if you have any questions. Links NTCIR-9 website NTCIR-9 RITE Online Proceedings ACL Portal RTE which has links to PASCAL/TAC RTE EVALITA iRTE CLEF Answer Validation Excercize