Member-only story

šŸ¹ Go Reached 1.96s In One Billion Row Challenge

Tom Smykowski
3 min readMar 23, 2024

--

C++ and Rust get out of way. Golang yet prooves itā€™s not only safe but also super fast

On January, 1, 2024 Gunnar Morling announced an algorithic contest initially for Java developers.

The contest was held until the end of January. The goal of it was to write a Java app that will process a file with one billion temperature readings looking like this:

Hamburg;12.0
Bulawayo;8.9
Palembang;38.8
St. John's;15.2
Cracow;12.6
...

Each row contains a reading of a temperature. The location name and reading is separated by a semicolon. Temperature reading has one fraction number.

Subscribe!

As a result the app should return statistics about the data:

{Abha=5.0/18.0/27.4, Abidjan=15.7/26.0/34.1, AbƩchƩ=12.1/29.4/35.6, Accra=14.7/26.4/33.1, Addis Ababa=2.1/16.0/24.3, Adelaide=4.1/17.3/29.7, ...}

So it has to output min, mean and max value for each location.

Quite easy. Except that there is a billion of readings in a file of the size of 13 GB.

These are Go flashcards I've designed. You can order them on https://summonthejson.com/products/golang-flashcards

The challenge became quite popular in the Java community attracting a lot of people trying to bend the time to process the file as fast as possible.

Clap!

The fastest implemention turn out to be the one made by Thomas Wuerthinger, Quan Anh Mai, AlfonsoĀ² Peterssen reaching whooping 1.535s in Java.

Since that time the 1BRC challenge attracted programmers in other languages, also in Golang.

First implementation Iā€™ve seen was the one made by Shraddha Agrawal that was able to lower the time from over 6 minutes to 14 seconds. The journey to that number is quite fascinating.

The final result isnā€™t tested independently, but this doesnā€™t matter. More impressive is the scale of the improvement of 25 times faster processing. Also the article walks nicely through all the optimisations made. Some are mins blowing.

--

--

Tom Smykowski
Tom Smykowski

Written by Tom Smykowski

Hi! My name is Tom Smykowski, and Iā€™m a Staff Frontend Engineer. Grab a free scalable Angular app checklist: https://tomasz-smykowski.com/scalable-angular

Responses (5)