<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<p class="elementToProof" style="margin-top: 0px; margin-bottom: 0px;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Call for Participation</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b> </b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Introduction</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">The National Institute of Standards and Technology (NIST) is organizing a small-scale evaluation as part
of an open track in TREC’2024 for researchers outside of the Defense Advanced Research Projects Agency (DARPA) Computational Cultural Understanding (CCU) program. The DARPA CCU program focuses on finding and identifying sociocultural norms from social media.
This track is targeted at researchers who want to participate in technology developments in CCU and aims to build critical mass to solve challenges posed in the CCU program.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">The first evaluation track focuses on the detection of sociocultural norms in video recordings of a variety
of naturally occurring interactions between two or more people in Mandarin Chinese. Successful communication entails not only knowing the local language but also understanding local cultures and customs. Violation of cultural norms may derail a conversation
and lead to negative and sometimes even disastrous consequences. Therefore, detecting social norms and determining if a speaker is adhering to or violating them are foundational components in dialogue assistance applications. Such applications could facilitate
successful communication between individuals who do not speak a common language or are not familiar with each other’s culture. The DARPA CCU program seeks to develop such capabilities to provide effective dialogue assistance to monolingual operators in cross-cultural
interactions.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">For more information about the evaluation task, data, metrics, protocol, and schedule, refer to the evaluation
plan posted on the evaluation page: </span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(17, 85, 204);"><a href="https://www.nist.gov/itl/iad/mig/computational-cultural-understanding-open-evaluation-openccu" id="OWAa3a15df2-8748-b22c-326e-36b428fc274d" class="OWAAutoLink" style="color: rgb(17, 85, 204); margin-top: 0px; margin-bottom: 0px;">https://www.nist.gov/itl/iad/mig/computational-cultural-understanding-open-evaluation-openccu</a></span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b> </b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Task</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">First, detect if a set of predefined socio-cultural norms exists in video recordings of naturally occurring
interactions of two or more individuals speaking in Mandarin Chinese.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Next, determine whether the individuals adhere to or violate the detected norms.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b> </b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Data</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Development ~ 1100 videos</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Pilot evaluation ~1100 videos</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Formal evaluation ~2400 videos</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b> </b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Metrics</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Precision, Recall, AP, mAP. These are defined in the evaluation plan linked above.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(33, 33, 33);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Schedule</b></span></p>
<table style="width: 720px; border-collapse: collapse; border-spacing: 0px; box-sizing: border-box;">
<tbody>
<tr>
<td style="border-width: 1pt; border-style: solid; border-color: black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Registration period;</span></p>
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Development data available for release</span></p>
</td>
<td style="border-top: 1pt solid black; border-right: 1pt solid black; border-bottom: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Feb 13 - Jun 25, 2024</span></p>
</td>
</tr>
<tr>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; border-left: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Pilot evaluation period</span></p>
</td>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">May 7 - 14, 2024</span></p>
</td>
</tr>
<tr>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; border-left: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Evaluation period</span></p>
</td>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Sep 3 - 10, 2024</span></p>
</td>
</tr>
<tr>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; border-left: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">System description paper due</span></p>
</td>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Oct 15, 2024</span></p>
</td>
</tr>
<tr>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; border-left: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">TREC workshop</span></p>
</td>
<td style="border-right: 1pt solid black; border-bottom: 1pt solid black; padding: 5pt; vertical-align: top; width: 3.75in;">
<p style="line-height: normal; margin: 0in; font-family: Arial, sans-serif; font-size: 11pt;">
<span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Nov 18 - 22, 2024</span></p>
</td>
</tr>
</tbody>
</table>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Participation Registration</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">If you wish to participate in this evaluation, register at
</span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(17, 85, 204);"><a href="https://sat.nist.gov/openccu" id="OWAa20aa2a9-ada3-3c70-a465-f09f4db5e8e4" class="OWAAutoLink" style="color: rgb(17, 85, 204); margin-top: 0px; margin-bottom: 0px;">https://sat.nist.gov/openccu</a></span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">.</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Track Coordinators</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Audrey Tong, NIST</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Jonathan Fiscus, NIST</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Leora Morgenstern, SRI</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">Stephanie Strassel, LDC</span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"> </span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);"><b>Contact</b></span></p>
<p style="text-align: justify; line-height: normal; margin: 0in;"><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">For more information/questions, email
</span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(17, 85, 204);"><a href="mailto:nist_ccu@nist.gov" id="OWA75dc2c74-9c45-eba1-9f8d-44215c588236" class="OWAAutoLink" style="color: rgb(17, 85, 204); margin-top: 0px; margin-bottom: 0px;">nist_ccu@nist.gov</a></span><span style="font-family: Calibri, sans-serif; font-size: 10pt; color: rgb(0, 0, 0);">.</span></p>
</body>
</html>