RunID Definition for Question Analysis, IR4QA, CCLQA
Each run has to be associated with a RunID which is an identity for each run. Using NTCIR CLIR format as a base, the RunID is defined as follows:
[Group's ID]-[Topic Language]-[Document Language]-[pp]-[Run Type]
Two character language codes are as follows:
- EN (English)
- CS (Simplified Chinese)
- CT (Traditional Chinese)
- JA (Japanese)
RunID Definition for CCLQA using IR4QA
Based on the definition in the previous section, there are Group ID for both IR4QA and CCLQA. See definition below:
[IR4QA Group's ID]-[CCLQA Group's ID]-[Topic Language]-[Document Language]-[pp]-[Run Type]
For example, suppose TEAM1 submitted an IR4QA result with a RunID:
If a CCLQA participant TEAM2 uses this result as an input to the system, TEAM2 will submit the "CCLQA using IR4QA" result with a following RunID:
The result of this run will only be evaluated in automatic way.
The 'pp' is two digits used to represent the priority of the run, taking 01 as the highest and 03 as the lowest value. It will be used as a parameter for pooling. The participants have to decide the priority for each submitted run in the basis of each language pair.
For CCLQA and Monolingual QA participants, please submit at least one T-run (only QUESTION field is used). You can submit up to 3 runs, but the first run (01) should be the T-run as mandatory.
- T-run: only QUESTION field is used
- D-run: only NARRATIVE field is used
- DN run: both QUESTION and NARRATIVE field is used
For example, a participating group, CMUJAV, submits 3 runs for English-Japanese. The first is a T run, the second is a D run and the third is a DN run. Therefore, the Run ID for each run is
CMUJAV-EN-JA-01-T CMUJAV-EN-JA-02-D CMUJAV-EN-JA-03-DN
Or, if the group uses different ranking techniques in T run for English-Japanese, the RunID for each run has to be
CMUJAV-EN-JA-01-T CMUJAV-EN-JA-02-T CMUJAV-EN-JA-03-D
Note: Top X documents in each of the submitted runs will be collected and put into the document pool. Only documnets in the pool will be judged by human assessors. If the number of the submitted runs are too large, the runs to be put in the pool may be selected based on the priority that you assign to each of the runs.