Submission and Evaluation

Submission Deadline: 23:59 (11:59 pm) GMT on Sunday, October 1, 2023.

Submission File

Submissions will contain both the code used to generate your final GeoPackage file, and the GeoPackage itself, named “lake_poygons_test.gpkg”, containing polygon outlines of your lakes, uploaded either directly into your GitHub repository or another accessible URL (such as a Google Drive, etc).

Code should be written in Python 3 and uploaded to a GitHub repository accessible to the contest judges. (If code in any other language is provided, clear instructions must be made so that the judges can execute your code. No exceptions will be made to the “open-access” requirement.) The GeoPackage file may be included in the GitHub repository in a sub-directory called “GPKG”, or may be available separately to download at another accessible URL (from a public link to a Google Drive, for instance). The GeoPackage should contain polygons outlining all the lakes specified in the “TEST” regions.

We will not accept multiple file submissions: all lakes should be combined into a single GeoPackage file. The filename of the image from which each lake was derived should be contained in the “image” column of the GeoPackage. The GitHub repository should have a clearly-readable README file that explains clearly how to run the code contained in the repository. All libraries used in the solution should be publicly available and open-access, so that judges can reproduce your results. The link to your GitHub repository should be provided in the Submission form.

A GeoPackage with your "answers" to the "test" regions should

  • be named “lake_polygons_test.gpkg”
  • have an identical column format to the “lake_polygons_training.gpkg” file, with each lake polygon specifying the image from which it was derived specified in the “image” column
  • be in the EPSG:3857 (“WGS 84 / Pseudo-Mercator”) coordinate reference system, the same as the input rasters and all provided GeoPackage files.

Submission Link

When your final solution is ready, SUBMIT your answer using the SIGSPATIAL GISCup 2023 Submission Form. Please make sure your answers are final and code is complete before filling out the submission form.

Evaluation Procedure

Submission Deadline: 23:59 (11:59 pm) GMT on Sunday, October 1, 2023.

For each image, the reviewers have a third geojson, which includes the hand-tagged lakes identified in the “test” regions, that we do not release to the participants. The submitted geojson files provided for the “test” regions will be scored using an F-test for lake correctness against the “test” GeoJson of polygons. Precision and recall (false positives and false negatives) will be equally weighted. Each polygon will be assessed as “true positive” (TP), a “false positive” (FP – identifying a lake that does not appear in the test dataset), or a “false negative” (FN – not identifying a lake that is contained in the test dataset). An F1-score will be produced as such: F1=2TP/(2TP + FP + FN).

It is understood that no algorithm is expected to achieve a perfect F1 score of 100% (1.0); you will soon find that these classifications are tricky to nail down precisely, and that sometimes “judgment calls” have been made in the training dataset as to what (and what does not) constitute a “lake” in the imagery. Use the “training” polygons as a guide.

A contest winner will be chosen based on the following criteria:

  1. The lakes have to be auto-generated and filtered using open-access code available to the judges. No hand-drawn lakes may be submitted.
  2. Results can be reproduced by the judges using the code in the GitHub repository.
  3. The submission with the highest F1 score that meets the above two criteria will be declared the winner.

It is important to note that identified lake polygons are unlikely to exactly match the outlines of polygons drawn in the “test” dataset. Hence, an individual lake polygon will be deemed a “true positive” if it meets the following criteria:

  1. The polygon does not overlap any other polygon corresponding to the same image and region, in the same submitted GeoJson file. Duplicate or overlapping polygons will be classified as “false positives”. This avoids contestants outlining the same lake in the same image multiple times and having each considered a “true positive” to generate inflated F1 scores. Competitors should take care that each lake is associated with the correct image.
  2. The polygon partially or fully overlaps at least one polygon identified in the “test” dataset. (If yes, go to Step 3. If no, it is considered “False Positive.”). If it overlaps more than one test polygon, it will be associated with the test polygon for which it has the greatest overlap (by area). Lakes will not be penalized for partially overlapping or “clipping” the corner of a nearby lake polygon in the test dataset.
  3. The submitted polygon area is no less than 50% of the area of the overlapping test polygon identified in Step 2.
  4. The submitted polygon area is no more than 160% of the area of the overlapping test polygon identified in Step 2.
  5. After assessing each submitted lake, each remaining test polygon that does not have an associated submitted lake, will be considered a “False Negative” in computing the F1 score.

Awards & Prizes

Through a generous donation from W. Randolph Franklin, the top three (3) submissions to GISCup 2023 will receive prizes of $1000 USD, $600, and $400, for entries ranked #1-3, respectively. Additionally, the top eight (8) submissions will receive a $1000 travel grant to assist with travel expenses to the SigSpatial Conference in November. To receive the travel grants through GISCup, individuals should submit a travel grant application through the travel grant application and highlight they are GISCup participants in appropriate questions.

Amazon has indicated it will supply Amazon Web Services cloud-computing credits to the top three (3) submissions as well. The exact amounts of these credits are being finalized and will be posted shortly.

Travel grant awardees will be contacted by the SigSpatial travel grant committee after winners are determined. Monetary awards and AWS credits will be distributed during the SigSpatial conference in November.