🐹 Go Reached 1.96s In One Billion Row Challenge

C++ and Rust get out of way. Golang yet prooves it’s not only safe but also super fast

On January, 1, 2024 Gunnar Morling announced an algorithic contest initially for Java developers.

The contest was held until the end of January. The goal of it was to write a Java app that will process a file with one billion temperature readings looking like this:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
...

Each row contains a reading of a temperature. The location name and reading is separated by a semicolon. Temperature reading has one fraction number.

Subscribe!

As a result the app should return statistics about the data:

{Abha=5.0/18.0/27.4, Abidjan=15.7/26.0/34.1, Abéché=12.1/29.4/35.6, Accra=14.7/26.4/33.1, Addis Ababa=2.1/16.0/24.3, Adelaide=4.1/17.3/29.7, ...}

So it has to output min, mean and max value for each location.

Quite easy. Except that there is a billion of readings in a file of the size of 13 GB.

These are Go flashcards I've designed. You can order them on https://summonthejson.com/products/golang-flashcards

The challenge became quite popular in the Java community attracting a lot of people trying to bend the time to process the file as fast as possible.

Clap!

The fastest implemention turn out to be the one made by Thomas Wuerthinger, Quan Anh Mai, Alfonso² Peterssen reaching whooping 1.535s in Java.

Since that time the 1BRC challenge attracted programmers in other languages, also in Golang.

First implementation I’ve seen was the one made by Shraddha Agrawal that was able to lower the time from over 6 minutes to 14 seconds. The journey to that number is quite fascinating.

The final result isn’t tested independently, but this doesn’t matter. More impressive is the scale of the improvement of 25 times faster processing. Also the article walks nicely through all the optimisations made. Some are mins blowing.