Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

ImageCLEF 2018 Lifelog - LMRT

Lifelog moment retrieval


Completed
49
Submissions
47
Participants
2102
Views

Overview


Important note:

The ImageCLEF Lifelog - Lifelog Moment Retrieval (LMRT) challenge has officially ended and we would like to thank everybody for their participation. You can find the official results at http://imageclef.org/2018/lifelog.

Post-challenge submissions and the leaderboard will remain enabled for a few weeks so you will still be able to submit result files and have them continuously evaluated during a limited period. Please consider that in order to see the version of the leaderboard with the post-challenge submissions integrated, you have to turn on the switch Show post-challenge submission right below the leaderboard.

At the same time we’d like to encourage you to submit a CLEF Working notes paper until the end of May.

Please also note that participants registering from now on will not be automatically registered with CLEF anymore.


Note: ImageCLEF Lifelog 2018 is divided into 2 subtasks (challenges). This challenge is about Lifelog moment retrieval (LMRT). For information on the Activities of Daily Living understanding (ADLT) challenge click here . Both challenges share the same dataset, so registering for one of these challenges will automatically give you access to the other one.

Note: Do not forget to read the Rules section on this page

Motivation

An increasingly wide range of personal devices, such as smartphones, video cameras as well as wearable devices that allow capturing pictures, videos, and audio clips in every moment of our life are becoming available. Considering the huge volume of data created, commonly referred to as lifelogs, there is a need for systems that can automatically analyse the data in order to categorize, summarize and also query to retrieve the information the user may need.

Despite the increasing number of successful related workshops and panels ( JCDL 2015 , iConf 2016 , ACM MM 2016 , ACM MM 2017 ) lifelogging has seldom been the subject of a rigorous comparative benchmarking exercise as, for example, the new lifelog evaluation task at NTCIR-13 or the last year edition of the ImageCLEFlifelog task. In this edition of this task we aim to bring the attention of lifelogging to an as wide as possible audience and to promote research into some of the key challenges of the coming years.

Challenge description

The participants have to retrieve a number of specific moments in a lifelogger’s life. We define moments as semantic events, or activities that happened throughout the day. For example, they should return the relevant moments for the query “Find the moment(s) when I was shopping for wine in the supermarket.” Particular attention should be paid to the diversification of the selected moments with respect to the target scenario.

The ground truth for this subtask was created using manual annotation.

Data

The task will be split into two related subtasks using a completely new multimodal dataset which consists of 50 days of data from a lifelogger, namely: images (1,500-2,500 per day from wearable cameras), visual concepts (automatically extracted visual concepts with varying rates of accuracy), semantic content (semantic locations, semantic activities) based on sensor readings (via the Moves App) on mobile devices, biometrics information (heart rate, galvanic skin response, calorie burn, steps, etc.), music listening history. The dataset is built based on the data available for the NTCIR-13 - Lifelog 2 task .

Format of the metadata

The metadata is stored in an .xml file, which is a simple aggregation of all users data. It is structured as follows:

The root node of the data is the USERS tag. Each user element contains all the data of that user (u1 or u2). Each user has a tag USER that contains the user ID as an attribute, example: [user id=”u1”]. For this year, only user u1 is considered. Inside the USER element, is his/her data:

Following that there is a tag DAYS, this tag contains the lifelogging information of that user organised per day, each day is included in a tag DAY that has the data (a tag DATA), the relative path to the directory that contains the images captured in that particular day (the tag IMAGES-DIRECTORY), then the minutes of of that day under a root tag called MINUTES.

At the start of each day there is a set of daily metatdata for that user. This data is of three forms; BIOMETRICS, ACTIVITIES & PERSONAL LOGS. The biometrics contains WEIGHT, FAT MASS, HEART RATE, SYSTOLIC blood pressure & DIASTOLIC blood pressure, which were readings taken after waking up each day. The activities contains summary activities: STEPS taken that day, DISTANCE walked in metres that day & ELEVATION climbed in metres that day. The personal logs contain HEALTH LOGS, including the TIME of reading, GLU Glucose levels in the blood, BP Blood Pressure, HR Heart Rate, MOOD manually logged every morning and sometimes a COMMENT, as well as DRINK LOGS and FOOD LOGS which were manually logged throughout the dat.

Following that, the day’s data is organised into minutes. The MINUTES element, contains exactly 1440 child elements (called MINUTE), each child has an ID (example: [minute id=“0”], [minute id=“1”], [minute id=“2”]… etc), and it represent one minute in the day ordered from 0 = 12:00 AM, to 1439 = 23:59PM.

Each minute contains: 0 or 1 location information (LOCATION tag), 0 or one activity information (ACTIVITY tag), biometrics, 0 or more captured images (IMAGES tag with IMAGE child element (each element has has a relative path to the image and a unique image ID), and 0 or 1 MUSIC tag giving details of the music listened to at that point in time.

-The location information is captured by Moves app (https://www.moves-app.com/), and they represent to semantic locations (Home, Work, DCU Computing building, GYM, Name of a Store, etc…), or to landmark locations registered by Moves. This tag can contain information in several languages. For locations that are not (HOME) or (WORK), the GPS locations are provided.

Submission instructions


As soon as the submission is open, you will find a “Create Submission” button on this page (just next to the tabs)


A submitted run for the LST sub-task must be in the form of a CSV file in the following format:

[topic id, image id, confidence score]

Where: - topic id: Number of the queried topic, e.g., from 1 to 5 for the development set. - image id: ID of a relevant image. - confidence score: from 0 to 1.

The CSV file should contain a diversified summarization in 50 images for each query.

Sample:

1, u1_2016-08-26_095916_1, 1.00

1, u1_2016-08-26_095950_2, 0.95

1, u1_2016-08-26_100028_1, 0.92

10, u1_2016-09-01_145314_1, 1.00

10, u1_2016-09-01_145345_2, 0.89

10, u1_2016-09-01_145531_1, 0.86

Evaluation

Metrics

For assessing performance, classic metrics will be deployed. These metrics are:

  • Cluster Recall at X (CR@X) - a metric that assesses how many different clusters from the ground truth are represented among the top X results;

  • Precision at X (P@X) - measures the number of relevant photos among the top X results;

  • F1-measure at X (F1@X) - the harmonic mean of the previous two.

Various cut off points are to be considered, e.g., X=5, 10, 20, 30, 40, 50. Official ranking metrics this year will be the F1-measure@10, which gives equal importance to diversity (via CR@10) and relevance (via P@10).

Participants are allowed to undertake the sub-tasks in an interactive or automatic manner. For interactive submissions, a maximum of five minutes of search time is allowed per topic. In particular, the organizers would like to emphasize methods that allow interaction with real users (via Relevance Feedback (RF), for example), i.e., beside of the best performance, the way of interaction (like number of iterations using RF), or innovation level of the method (for example, new way to interact with real users) are encouraged.

Ground truth format

Ground truth is provided in two individual txt files: one file for the cluster ground truth and one file for the relevant image ground truth.

In the cluster ground-truth file each line corresponds to a cluster where the first value is the topic id, followed by cluster id number, followed by the cluster user tag separated by comma. Lines are separated by an end-of-line character (carriage return). An example is presented below:

1, 1, Badger & Dodo Cafe

1, 2, Costa coffee

2, 1, Airport Restaurant

2, 2, Arnotts Department Store

In the relevant ground-truth file the first value on each line is the topic id, followed by a unique photo id, and then followed by the cluster id number (that corresponds to the values in the cluster ground-truth file) separated by comma. Each line corresponds to the ground truth of one image and lines are separated by an end-of-line character (carriage return). An example is presented below:

1, u1_2016-09-17_124915_1, 1

1, u1_2016-09-17_125300_1, 1

1, u1_2016-09-17_125332_2, 1

1, u1_2016-08-27_070424_1, 2

1, u1_2016-08-27_070456_2, 2

1, u1_2016-08-27_070528_1, 2

2, u1_2016-08-27_133126_1, 1

2, u1_2016-08-27_133158_2, 1

2, u1_2016-08-27_133230_1, 1

2, u1_2016-08-17_121617_1, 2

2, u1_2016-08-17_121704_1, 2

Rules

Note: In order to participate in this challenge you have to sign an End User Agreement (EUA). You will find more information on the ‘Dataset’ tab.

ImageCLEF lab is part of the Conference and Labs of the Evaluation Forum: CLEF 2018. CLEF 2018 consists of independent peer-reviewed workshops on a broad range of challenges in the fields of multilingual and multimodal information access evaluation, and a set of benchmarking activities carried in various labs designed to test different aspects of mono and cross-language Information retrieval systems. More details about the conference can be found here .

Submitting a working note with the full description of the methods used in each run is mandatory. Any run that could not be reproduced thanks to its description in the working notes might be removed from the official publication of the results. Working notes are published within CEUR-WS proceedings, resulting in an assignment of an individual DOI (URN) and an indexing by many bibliography systems including DBLP. According to the CEUR-WS policies, a light review of the working notes will be conducted by ImageCLEF organizing committee to ensure quality. As an illustration, ImageCLEF 2017 working notes (task overviews and participant working notes) can be found within CLEF 2017 CEUR-WS proceedings.

Important

Participants of this challenge will automatically be registered at CLEF 2018. In order to be compliant with the CLEF registration requirements, please edit your profile by providing the following additional information:

  • First name

  • Last name

  • Affiliation

  • Address

  • City

  • Country

  • Regarding the username, please choose a name that represents your team.

This information will not be publicly visible and will be exclusively used to contact you and to send the registration data to CLEF, which is the main organizer of all CLEF labs

Participating as an individual (non affiliated) researcher

We welcome individual researchers, i.e. not affiliated to any institution, to participate. We kindly ask you to provide us with a motivation letter containing the following information:

  • the presentation of your most relevant research activities related to the task/tasks

  • your motivation for participating in the task/tasks and how you want to exploit the results

  • a list of the most relevant 5 publications (if applicable)

  • the link to your personal webpage

The motivation letter should be directly concatenated to the End User Agreement document or sent as a PDF file to bionescu at imag dot pub dot ro. The request will be analyzed by the ImageCLEF organizing committee. We reserve the right to refuse any applicants whose experience in the field is too narrow, and would therefore most likely prevent them from being able to finish the task/tasks.

Prizes

ImageCLEF 2018 is an evaluation campaign that is being organized as part of the CLEF initiative labs. The campaign offers several research tasks that welcome participation from teams around the world. The results of the campaign appear in the working notes proceedings, published by CEUR Workshop Proceedings (CEUR-WS.org). Selected contributions among the participants, will be invited for publication in the following year in the Springer Lecture Notes in Computer Science (LNCS) together with the annual lab overviews.

Resources

Recommanded Reading

[1] Duc-Tien Dang-Nguyen, Luca Piras, Michael Riegler, Giulia Boato, Liting Zhou, Cathal Gurrin, “Overview of ImageCLEFlifelog 2017: Lifelog Retrieval and Summarization” , CLEF2017 Working Notes, Dublin, Ireland, 2017, vol 1866.

[2] Cathal Gurrin, Xavier Giro-i-Nieto, Petia Radeva, Mariella Dimiccoli, Håvard Johansen, Hideo Joho, Vivek K Singh, “LTA 2016: The First Workshop on Lifelogging Tools and Applications” , ACM Multimedia, Amsterdam, The Netherlands, 2016.

[3] Cathal Gurrin, Xavier Giro-i-Nieto, Petia Radeva, Mariella Dimiccoli, Duc Tien Dang Nguyen, Hideo Joho, “LTA 2017: The Second Workshop on Lifelogging Tools and Applications” , ACM Multimedia, Mountain View, CA USA, 2017.

[4] Cathal Gurrin, Hideo Joho, Frank Hopfgartner, Liting Zhou, Rami Albatal, “Overview of NTCIR-12 Lifelog Task” , Proceedings of the 12th NTCIR Conference on Evaluation of Information Access Technologies, Tokyo, Japan, 2016.

[5] Duc-Tien Dang-Nguyen, Luca Piras, Giorgio Giacinto, Giulia Boato, Francesco GB De Natale, “Multimodal Retrieval with Diversification and Relevance Feedback for Tourist Attraction Images” , ACM Transactions on Multimedia Computing, Communications, and Applications, vol 13, n° 4, 2017.

[6] Duc-Tien Dang-Nguyen, Luca Piras, Giorgio Giacinto, Giulia Boato, Francesco GB De Natale, “A hybrid approach for retrieving diverse social images of landmarks” , IEEE International Conference on Multimedia and Expo (ICME), Turin, Italy, 2015.

[7] Working notes of the 2015 MediaEval Retrieving Diverse Social Images task , CEUR-WS.org, Vol. 1436, ISSN: 1613-0073.

[8] B. Ionescu, A.L. Gînscă, B. Boteanu, M. Lupu, A. Popescu,H. Müller, “Div150Multi: A Social Image Retrieval Result Diversification Dataset with Multi-topic Queries” , ACM MMSys, Klagenfurt, Austria, 2016.

Helpful tools and resources

Contact us

We strongly encourage you to use the public channels mentioned above for communications between the participants and the organizers. In extreme cases, if there are any queries or comments that you would like to make using a private communication channel, then you can send us an email at :

  • Sharada Prasanna Mohanty: sharada.mohanty@epfl.ch
  • Duc-Tien Dang-Nguyen: duc-tien[DOT]dang-nguyen[AT]dcu[DOT]ie
  • Luca Piras: luca[DOT]piras[AT]diee[DOT]unica[DOT]it
  • Ivan Eggel: ivan[DOT]eggel[AT]hevs[DOT]ch

More information

You can find additional information on the challenge here: http://imageclef.org/2018/lifelog