[PlanetKR] CFP: NIST/TREC Challenge on Identifying Sociocultural Norms

[Leora Morgenstern]

Call for Participation



Introduction

The National Institute of Standards and Technology (NIST) is organizing a small-scale evaluation as part of an open track in TREC’2024 for researchers outside of the Defense Advanced Research Projects Agency (DARPA) Computational Cultural Understanding (CCU) program. The DARPA CCU program focuses on finding and identifying sociocultural norms from social media. This track is targeted at researchers who want to participate in technology developments in CCU and aims to build critical mass to solve challenges posed in the CCU program.



The first evaluation track focuses on the detection of sociocultural norms in video recordings of a variety of naturally occurring interactions between two or more people in Mandarin Chinese. Successful communication entails not only knowing the local language but also understanding local cultures and customs. Violation of cultural norms may derail a conversation and lead to negative and sometimes even disastrous consequences. Therefore, detecting social norms and determining if a speaker is adhering to or violating them are foundational components in dialogue assistance applications. Such applications could facilitate successful communication between individuals who do not speak a common language or are not familiar with each other’s culture. The DARPA CCU program seeks to develop such capabilities to provide effective dialogue assistance to monolingual operators in cross-cultural interactions.



For more information about the evaluation task, data, metrics, protocol, and schedule, refer to the evaluation plan posted on the evaluation page:  https://www.nist.gov/itl/iad/mig/computational-cultural-understanding-open-evaluation-openccu.



Task

First, detect if a set of predefined socio-cultural norms exists in video recordings of naturally occurring interactions of two or more individuals speaking in Mandarin Chinese.

Next, determine whether the individuals adhere to or violate the detected norms.



Data

Development ~ 1100 videos

Pilot evaluation ~1100 videos

Formal evaluation ~2400 videos



Metrics

Precision, Recall, AP, mAP. These are defined in the evaluation plan linked above.



Schedule

Registration period;

Development data available for release

Feb 13 - Jun 25, 2024

Pilot evaluation period

May 7 - 14, 2024

Evaluation period

Sep 3 - 10, 2024

System description paper due

Oct 15, 2024

TREC workshop

Nov 18 - 22, 2024



Participation Registration

If you wish to participate in this evaluation, register at https://sat.nist.gov/openccu.



Track Coordinators

Audrey Tong, NIST

Jonathan Fiscus, NIST

Leora Morgenstern, SRI

Stephanie Strassel, LDC



Contact

For more information/questions, email nist_ccu at nist.gov<mailto:nist_ccu at nist.gov>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://kr.org/pipermail/planetkr/attachments/20240403/d5ca2b44/attachment-0001.htm>