Battle of the Serverless — Part 1: Rust vs Go vs Kotlin vs F# vs C#

This weekend was supposed to be about some deep exploration on Go and dusting off my “archineer” and “engitect” hats. Some late night frustration with Go, however, on Friday prompted me to do some experimentation around serverless technologies in general; some benchmarking and studying. Read on if you want to see some observations and data points based on ~750,000 requests over 3 days on AWS Lambda, API Gateway, and some compiled serverless “hello world” type services; some are with AWS provided runtimes; some are with BYOR, or Bring-Your-Own-Runtime like Rust.

Battle of the Serverless “Hello Worlds”

For this test, I built a pretend suite of microservices exposed via API Gateway to form an API with a code name of Slipspace in a mock company called STG. Slipspace drives are how the ships in the Halo universe travel so quickly to different sectors of the galaxy through something called Slipstream Space, so thought it was cool for a name requiring awesome warp API speeds.

Quick observations

In part 2, I’m going to add in interpreted languages and runtimes like TypeScript/Node.js and Python. I’ll also share the GitHub repository with all the code, examples, and README’s so others can experiment and extend. I’ll also share with the general public my process of selecting technologies to archineer and engitect with, including a newly included “Love-joy” factor.

For now though, here are some quick observations:

• Go (or @golang) is 95% of the way there for me; the 5% remaining is so weird it’s hard to describe. It’s that “good enough” language for microservices that is fun to learn, but it’s not for business logic. I was productive in it after only a day, and I’ve spent days afterwards wondering what else was there. Feels like the creators built it solely so that compilation was fast; it’s very simple, like 25-ish keywords, and embraces very simple concepts. It’s not elegant, but is pragmatic. Go had the best overall performance for my initial tests in cold and warm functions.
• Rust is currently fascinating me. Feels like a modern C/C++ that brings a level of joy to me that I’ve not felt in a long time. Will be focusing on that tons in the future, feels like my new friend that I want to start traveling with. Rust was also very stable in the benchmark numbers. It turned in solid numbers in cold and warm functions, was not the fastest, but nearly a flat line with respect to performance (which is a good thing.) I’m hopeful that it’s the #1 loved language on Stack Overflow for good reason.
• F# also intrigues me. I’m just burnt out on C# so it wasn’t that fun working through that test setup for me, but F# introduced functional programming in a way that was both familiar (.NET Core) and new. F# and C# on the .NET Core 2.1 runtime in AWS Lambda were quick when functions were warm, but was very inconsistent on cold starts and showed periodic spikes. Need to digest more F# to see if it has a place in my future.
• Kotlin should have been introduced to me earlier in my career. Had I known Kotlin earlier, perhaps I would’ve gone through more miles with JVM technologies. Much better than Java, more concise, more joy, but still very heavy and overly verbose. More fun that Java, but still not as much fun as Rust or Go. Kotlin deployments were much larger than the others, so much was packaged in with them, but had very respectable performance.

So what was the test?

The test this round was put together quickly and will likely need some fine-tuning. Each different stack (Rust vs Go vs Kotlin vs F# vs C#) was created quickly with the Serverless framework, modified only slightly in the code and serverless.yml files of each project, and then deployed, each with it’s own API Gateway and CloudFormation stack.

Once deployed, a quick bash script was created called api-cannon.sh that fired off three concurrent requests to each API Gateway endpoint, then slept for 5 seconds, and repeated for 50,000 iterations. With 5 endpoints, 50,000 iterations, and 3 concurrent requests per endpoint, we ended up with ~750,000 requests.

5 endpoints * 50,000 iterations * 3 concurrent requests per endpoint = ~750,000 requests

50,000 iterations * 5 seconds = ~3 days of continuous testing

All resources were deployed to the Oregon (us-west-2) region in AWS. The code executed in each project’s AWS Lambda handler was a very basic execution which resulted in simple JSON payloads being returned.

And the numbers are in

The values for this round of testing in table format are as follows. Go took the overall cake for being in the top 3 in each duration captured. It was also the only language that took home < 1 ms average response times. Kotlin and F# both put up very solid numbers, tying for 2nd (according to me) both having spots in 2nd and 3rd in 2 categories each.

For average duration, ALL of them are freaking fast (< 10 ms) for this simple type of test, being similar to “hello world” response of a simple JSON payload. I’m also shocked that minimum duration's were possible of < .20 ms response times.

Also, if I’m being honest, I wanted Rust to do better. It didn’t do as well as I’d thought, but might be related to the BYOR scenario. The one thing it did do was be very steady: it had the least variation in performance, which could be a win for solutions requiring extremely predictable speeds.

Table format for duration’s captured during 3 days of testing

For average duration of AWS Lambda function execution, these are the data points. Both .NET Core languages were quite spiky in spots, but still turned in good times for averages. Go and Rust were as stead as Dr. Strange’s hands before his accident, and Kotlin only showed minor fluctuations.

Number call outs for average duration in benchmark tests

Sampling of line charts for average duration in benchmark tests

For maximum duration of AWS Lambda function execution, these are the data points. Both .NET Core languages showed big spikes in cold starts. Only Kotlin and .NET Core languages broke the > 1 s threshold for maximum response time. Go and Rust win here.

Number call outs for maximum duration in benchmark tests

Sampling of line charts for maximum duration in benchmark tests

For minimum duration of AWS Lambda function execution, these are the data points. I love how level these numbers are. All but Kotlin broke the < 1 ms floor threshold. Go and .NET Core languages took the cake here, but honestly these are all awesome to me and are imperceptible to the any users or things on the other side of a service.

Number call outs for minimum duration in benchmark tests

Sampling of line charts for minimum duration in benchmark tests

FIN/ACK

In summary, consider this all a preview of the real next part: Part 2 (coming soon). I’m coming out of this weekend loving Rust, Go (most of the time) and intrigued by F# for functional programming. Go is still that trusted friend that I can use reliably to get most things done with type safety, speed, and simplicity, even if it does piss me off once in a while. Rust is the new travel buddy that I think will be friends with for a long time and even longer projects. And F#…need to go to the shooting range with that one to see if we can be friends. Kotlin wants to be my friend, but I’m not sure I can be yet.

I’m also very happy with the lessons learned through my own testing, benchmarking. This round wasn’t perfect and there are some better tools out there to provide more meaningful data points; will be sharing those next time. I like knowing that I was in control of the code, that I knew exactly what code was executed. Next time, some CRUD operations against a DynamoDB backend will be introduced; some inter-connectivity with other AWS resources, too.