Want to organize your own competition?
We are now accepting proposals for review. Reviewed and accepted proposals are eligible to display the ChaLearn logo and may
Contact us with your project, we can help you finding funding and sponsors and setting up your challenge. Full ChaLearn competition organizer guidelines may be downloaded here. Briefly:
Ten tips to organize a winning competition:
Task 1: Finding data. Identify a good problem and a good dataset.
Task 2: Formatting data. Preprocess and format the data to simplify the task of participants and reduce the need for domain knowledge.
Task 3: Assessment. Define a task and evaluation metrics. Define and implement methods of scoring the results and comparing them.
Task 4: Baseline. Try to solve your own problem to see whether it is feasible and provide baseline results.
Task 5: Result formats. Define the formats in which the results should be returned by the competitors.
Task 6: Benchmark protocol. Define the rules of the competition and determine the sequence of events.
Task 7: Web portal. Create a web portal allowing on-line submissions and displaying results on a leaderboard.
Task 8: Guidelines to participants. Write the competition rules, document the formats and the scoring methods, write FAQs.
Task 9: Seek private sponsors. Find additional sponsor money to give travel awards so the participants can attend the workshop.
Task 10: Prepare the workshop. Look for tutorial speakers. Select speakers. Create a schedule. Advertise.
Task 11: Competition result analysis. Compile the results. Produce graphs. Derive conclusions.
Task 12: Release the results on-line. Make available on-line the competition result analyses, fact sheets of the competitors's methods, the workshop slides.
Task 13: Post competition tests. Reproduce the results of the best competitors. Identify candidate essential ingredients of success. Perform a systematic study such ingredients.
Task 14: Write technical reports. Write reports on the benchmark design, the results of the competition, and the results of the post-competition tests.
Task 15: Prepare workshop proceedings. Solicit papers, organize the review process, and edit the papers.
Contact us with your project, we can help you finding funding and sponsors and setting up your challenge. Full ChaLearn competition organizer guidelines may be downloaded here. Briefly:
Ten tips to organize a winning competition:
- IMPACT: Choose a problem with high impact (economical, humanitarian, societal, etc.)
- DATA: Make sure that
- you have access to large enough datasets to make the challenge interesting and draw conclusive results [how much is enough?];
- the data can be made freely available;
- the ground truth has been kept confidential.
- RELEVANCE: The problem should be relevant to the computational intelligence community and should be solvable without extensive domain knowledge.
- CHALLENGE: The problem posed should be scientifically or technically challenging. However, it should not be impossible to solve; the organizers should provide baseline results. Think of illustrating the same scientific problem using several datasets from various application domains.
- APPEAL: The competition setup should be attractive with preferably on-line submission and feed-back and/or an on-site contest at the conference.
- EVALUATION: It should be possible to evaluate the results objectively (provide a metric of significance in performance differences).
- RESOURCES: The organizers should make sure to have enough resources (team member availability, computers, support staff, other equipment, sponsors).
- SCHEDULE: Propose a reasonable schedule leaving enough time for the organizers to prepare the Event (a few months), enough time for the participants to develop their methods (e.g. 90 days), enough time for the organizers to review the entries, analyze and publish the results.
- ADVERTISEMENT: Figure out how the event will be advertised to ensure that potentially interested participants have a fair chance of getting informed.
- RULES: [General CHALEARN contest rules] [Modify our template] Draft rules including:
- Description
- Registration
- Download of data
- Submission of entries
- Verification procedures
- Uses of entries
- Criteria of evaluation
- Judges
- Prizes
- Deadlines
- Conditions of participation
- Restrictions of participation
- Result dissemination
- Notifications
- Sponsors
- Privacy
- Intellectual property
- Errors and frauds
Task 1: Finding data. Identify a good problem and a good dataset.
Task 2: Formatting data. Preprocess and format the data to simplify the task of participants and reduce the need for domain knowledge.
Task 3: Assessment. Define a task and evaluation metrics. Define and implement methods of scoring the results and comparing them.
Task 4: Baseline. Try to solve your own problem to see whether it is feasible and provide baseline results.
Task 5: Result formats. Define the formats in which the results should be returned by the competitors.
Task 6: Benchmark protocol. Define the rules of the competition and determine the sequence of events.
Task 7: Web portal. Create a web portal allowing on-line submissions and displaying results on a leaderboard.
Task 8: Guidelines to participants. Write the competition rules, document the formats and the scoring methods, write FAQs.
Task 9: Seek private sponsors. Find additional sponsor money to give travel awards so the participants can attend the workshop.
Task 10: Prepare the workshop. Look for tutorial speakers. Select speakers. Create a schedule. Advertise.
Task 11: Competition result analysis. Compile the results. Produce graphs. Derive conclusions.
Task 12: Release the results on-line. Make available on-line the competition result analyses, fact sheets of the competitors's methods, the workshop slides.
Task 13: Post competition tests. Reproduce the results of the best competitors. Identify candidate essential ingredients of success. Perform a systematic study such ingredients.
Task 14: Write technical reports. Write reports on the benchmark design, the results of the competition, and the results of the post-competition tests.
Task 15: Prepare workshop proceedings. Solicit papers, organize the review process, and edit the papers.
