Loading
Cookies help us deliver our services. By using our services, you agree to our use of cookies. Learn more

crowdAI is shutting down - please read our blog post for more information

LifeCLEF 2019 Bird Recognition

Bird species recognition in soundscapes


Completed
25
Submissions
114
Participants
6685
Views

Validation files and metadata

Posted by MfN over 2 years ago

Hi, thank you for organizing this new bird identification challenge. I’m looking forward to see the final results! A few questions came up, looking at the provided data: In the validation set there are 13 files without annotations. Does that mean there are no birds present in those files or does it mean those files were not annotated? Also can somebody maybe say a few words about the eBird data in the additional_metadata folder? I didn’t find any explanation what this data represents and how it might be useful.

Thanks, Mario

1

Posted by TUC_CLO  over 2 years ago |  Quote

Hi Mario, those files do actually not contain any vocalization. There will be similar cases in the test data - mostly nighttime recordings. The eBird metadata contains the occurrence probabilities for each of the 48 weeks of the year according to eBird checklists for the two main regions in North and South America. You could use the files to determine plausible species compositions for date and location of the recorded soundscapes. Unfortunately, eBird discontinued the Api v1.1 and the data can no longer be accessed online. If you have any further questions, please let me know. Best, Stefan

Posted by Matt Harvey  over 2 years ago |  Quote

Running the evaluator on input with a MediaID from a no-vocalization val set file causes an exception. Will this cause trouble when we submit test set runs?

Reproduction:

Edit EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py so that submission_file_path is to the single-line file: SSW0025;00:00:00-00:00:05;norcar;1.0

$ python EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py Traceback (most recent call last): File “EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py”, line 347, in result = evaluator._evaluate(submission_file_path) File "EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py", line 40, in _evaluate predictions = self.load_predictions(submission_file_path) File "EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py", line 184, in load_predictions .format(query_id, self.line_nbr_string(lineCnt))) Exception: MediaID 'SSW0025' in submission file does not exist in testset (Line nbr 1)

Posted by TUC_CLO  over 2 years ago |  Quote

We updated the evaluation script and everything should be working fine now. Could you please try to submit again? Thanks.

Posted by TUC_CLO  over 2 years ago |  Quote

Running the evaluator on input with a MediaID from a no-vocalization val set file causes an exception. Will this cause trouble when we submit test set runs?

No, that should not happen for the test data, we included those IDs and marked them as empty. If you have trouble with any of your submissions, please let me know.

Reproduction:

Edit EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py so that submission_file_path is to the single-line file: SSW0025;00:00:00-00:00:05;norcar;1.0

$ python EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py Traceback (most recent call last): File “EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py”, line 347, in result = evaluator._evaluate(submission_file_path) File "EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py", line 40, in _evaluate predictions = self.load_predictions(submission_file_path) File "EvaluatorBirdCLEF2019SoundscapesCrowdAIOnValidationSet.py", line 184, in load_predictions .format(query_id, self.line_nbr_string(lineCnt))) Exception: MediaID 'SSW0025' in submission file does not exist in testset (Line nbr 1)