Want to organize your own competition?
Contact us with your project, we can help you finding funding and sponsors and setting up your challenge. Full ChaLearn competition organizer guidelines may be downloaded here. Briefly:
Ten tips to organize a winning competition:
- IMPACT: Choose a problem with high impact (economical, humanitarian, societal, etc.)
- DATA: Make sure that
- you have access to large enough datasets to make the challenge interesting and draw conclusive results [how much is enough?];
- the data can be made freely available;
- the ground truth has been kept confidential.
- Download of data
- Submission of entries
- Verification procedures
- Uses of entries
- Criteria of evaluation
- Conditions of participation
- Restrictions of participation
- Result dissemination
- Intellectual property
- Errors and frauds
Task 1: Finding data. Identify a good problem and a good dataset.
Task 2: Formatting data. Preprocess and format the data to simplify the task of participants and reduce the need for domain knowledge.
Task 3: Assessment. Define a task and evaluation metrics. Define and implement methods of scoring the results and comparing them.
Task 4: Baseline. Try to solve your own problem to see whether it is feasible and provide baseline results.
Task 5: Result formats. Define the formats in which the results should be returned by the competitors.
Task 6: Benchmark protocol. Define the rules of the competition and determine the sequence of events.
Task 7: Web portal. Create a web portal allowing on-line submissions and displaying results on a leaderboard.
Task 8: Guidelines to participants. Write the competition rules, document the formats and the scoring methods, write FAQs.
Task 9: Seek private sponsors. Find additional sponsor money to give travel awards so the participants can attend the workshop.
Task 10: Prepare the workshop. Look for tutorial speakers. Select speakers. Create a schedule. Advertise.
Task 11: Competition result analysis. Compile the results. Produce graphs. Derive conclusions.
Task 12: Release the results on-line. Make available on-line the competition result analyses, fact sheets of the competitors's methods, the workshop slides.
Task 13: Post competition tests. Reproduce the results of the best competitors. Identify candidate essential ingredients of success. Perform a systematic study such ingredients.
Task 14: Write technical reports. Write reports on the benchmark design, the results of the competition, and the results of the post-competition tests.
Task 15: Prepare workshop proceedings. Solicit papers, organize the review process, and edit the papers.