Using ExAws to setup SQS, SNS, and consume messages

ExAws is a collection of Elixir libraries to interact with Amazon Web Services.

The ExAws documentation is in general very good, but I figured it would be nice to show example code of a concrete solution using a small subset of the functionality. Hopefully you can expand/modify the code to your own needs. I assume the reader is familiar with Elixir and has some basic AWS knowledge.

In this post I will describe how to accomplish the following with Elixir and ExAws:

  1. Create a SQS queue
  2. Create a subscription to an existing SNS topic
  3. Set SNS subscription attributes
  4. Consume messages from the SQS queue

The presented code will be functioning but of prototype nature. That means tests, error handling and refactoring opportunities are left to the reader as an exercise ;-)

If you want to jump straight to the code it is up on github.com/vorce/ex_aws_example.


First we need to add the ex_aws dependencies to our project:

In mix.exs:

defp deps do
    {:ex_aws, "~> 2.1"},
    {:ex_aws_sqs, "~> 3.0"},
    {:ex_aws_sns, "~> 2.0"},
    {:sweet_xml, "~> 0.6"},
    {:hackney, "~> 1.15"},
    {:poison, "~> 4.0"}

and fetch them (mix deps.get).

Create a SQS queue

This is fairly straight forward with ExAws.SQS.create_queue/2. Let’s start with a fresh module to hold our functions that we will expand on as we go along.

defmodule ExAwsExample do
  @moduledoc "Collection of functions demonstrating how to use ExAws.SQS and ExAws.SNS"

  alias ExAws.SQS

  def create_queue(queue_name, opts \\ []) do
    |> SQS.create_queue(opts)
    |> ExAws.request()

queue_name is just a string, opts is an optional keyword list containing any extra options you might need (outlined here) - leave it out completely or set it to the empty list ([]) if you don’t need any attributes.

Create a subscription to an existing SNS topic

To create a subscription we need to use ExAws.SNS.subscribe/3. Since you can subscribe different things to a SNS topic, it’s not as self explanatory how to use it. You can get the topic_arn from the AWS console (looks something like “arn:aws:sns:region:12345678:topicname”), this identifies the SNS topic you want to get messages from. It’s not immediately clear what values we can use for the protocol parameter, looking at the AWS console we can see the following options:

AWS SNS Subscription options

So far I know these values work: "email", "http", "https", and "sqs" - in this example we will use "sqs". The third parameter to the subscribe function is a string called endpoint, this is where the message from the topic will go. In our SQS case this will be the ARN of the SQS queue we want the messages from the SNS topic to end up in. How do we get the ARN of the queue then? Ideally it would be returned in the response from the create_queue call, but it’s not. We do get the queue URL back from that call though and we can transform it to the ARN.

With our new knowledge let’s add a new alias and two functions to our module.

alias ExAws.SNS

def create_sqs_subscription(topic_arn, queue_url) do
  queue_arn = queue_url_to_arn(queue_url)

  |> SNS.subscribe("sqs", queue_arn)
  |> ExAws.request()

defp queue_url_to_arn(queue_url) do
  [_protocol, "", host, account_id, queue_name] = String.split(queue_url, "/")
  [service, region, _, _] = String.split(host, ".")


Let’s run our code. To do that we need to specify our AWS credentials. The easiest way is to set the two environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (more info in ex_aws docs) for our iex session. We also want to explicitly set the region in config.exs: AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... iex -S mix

Here’s how to use our two functions to create a queue, and then set up a subscription:

{:ok, %{body: %{queue_url: queue_url}}} = ExAwsExample.create_queue("my-great-queue")
{:ok, _} = ExAwsExample.create_sqs_subscription(my_topic_arn, queue_url) # assuming you know the arn of the SNS topic

After running this code you should see a queue in the SQS console called “my-great-queue”, and a subscription for it in the SNS console. At this point you might expect that you can publish messages to the topic and they end up in the queue - but not so fast.


We need to make sure that we give permission to the topic to publish to our queue. You can do this in the AWS SQS console - in fact it will automatically add the correct permissions if you manually add a subscription to a SNS topic. You can of course also accomplish the same thing with code which is handy. We have to add a policy option in the create_queue call. A policy is just a json document, more details in the AWS docs. The policy below will grant the topic permission to send messages to our particular queue.

def queue_policy(region, account_id, queue_name, topic_arn) do
    "Version": "2012-10-17",
    "Statement": [
        "Effect": "Allow",
        "Principal": {
          "AWS": "*"
        "Action": [
        "Resource": "arn:aws:sqs:#{region}:#{account_id}:#{queue_name}",
        "Condition": {
          "ArnEquals": {
            "aws:SourceArn": "#{topic_arn}"

Let’s create a new queue with a policy:

my_policy = ExAwsExample.queue_policy("eu-west-1", "123456778900", "my-great-queue2", my_topic_arn)
{:ok, %{body: %{queue_url: queue_url}}} = ExAwsExample.create_queue("my-great-queue2", policy: my_policy)
{:ok, _} = ExAwsExample.create_sqs_subscription(my_topic_arn, queue_url)

You can now verify that messages end up in the queue by manually publishing something to your SNS topic in the AWS web console.

Set SNS subscription attributes

A very useful attribute we can set when the subscription protocol is “sqs” is “RawMessageDelivery”, this attribute will skip the metadata/envelope around the actual contents of the message. We definitely want this on our subscription. To make it happen we need to use SNS.set_subscription_attributes/3.

In some cases you only want to forward a subset of the messages that flows to the SNS topic to the queue. To accomplish that we need to apply a filter policy to the subscription.

Ok, so we have two propertiers we need to set on the SNS subscription — time for another function:

def set_subscription_attributes(subscription_arn, opts \\ []) do
  filter = Keyword.get(opts, :filter, "{}") # default to an empy filter json document
  raw_message_delivery = Keyword.get(opts, :raw_message_delivery, "true")

  |> SNS.set_subscription_attributes(filter, subscription_arn)
  |> ExAws.request()

  |> SNS.set_subscription_attributes(raw_message_delivery, subscription_arn)
  |> ExAws.request()

The subscription_arn is conveniently returned in the response from the subscribe function. How the filter option look depend on the messages you expect, but here’s an example:

@filter_policy ~s"""
  "Source": ["MySystemX"],
  "PayloadType": ["user_event"],
  "PayloadVersion": [{"numeric": ["=", 1]}]

So after successfully creating the SNS subscription, we can set the subscription attributes:

{:ok, %{body: %{subscription_arn: sub_arn}}} = ExAwsExample.create_sqs_subscription(my_topic_arn, queue_url)
ExAwsExample.set_subscription_attributes(sub_arn, opts)

Consume messages from the SQS queue

Now that we have building blocks for setting up our infrastructure we can build the consumer for the queue that will get and process the messages that comes in. We want the message consumer to continuously poll the SQS queue, so let’s create a worker and add it to our supervision tree.

Consumer GenServer

We need a new module (sqs_consumer.ex) that will contain our queue consumer functionality.

defmodule ExAwsExample.SQSConsumer do
  @moduledoc """
  Consumes messages from a SQS queue
  alias ExAws.SQS

  require Logger

  use GenServer

  @account_id "041669274849"
  @queue_name "my-great-queue2"
  @queue_url "https://sqs.eu-west-1.amazonaws.com/#{@account_id}/#{@queue_name}"
  @topic_arn "arn:aws:sns:eu-west-1:#{@account_id}:test"
  @subscription_filter_policy "{}"

  def start_link(_) do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)

  @impl true
  def init(:ok) do
    Logger.debug("Setting up queue #{@queue_name} and subscription to topic #{@topic_arn}")

    policy = ExAwsExample.queue_policy("eu-west-1", @account_id, @queue_name, @topic_arn)

    {:ok, %{body: %{queue_url: queue_url}}} =
      ExAwsExample.create_queue(@queue_name, policy: policy)

    {:ok, %{body: %{subscription_arn: sub_arn}}} =
      ExAwsExample.create_sqs_subscription(@topic_arn, queue_url)

    subscription_opts = [filter: @subscription_filter_policy, raw_message_delivery: "true"]
    ExAwsExample.set_subscription_attributes(sub_arn, subscription_opts)


    {:ok, %{queue_name: @queue_name, last_message_time: nil}}

  def schedule_check(check_interval \\ 1_000) do
    Process.send_after(self(), :get_messages, check_interval)

  def handle_messages() do
    case get_messages(@queue_url, wait_time_seconds: 5, max_number_of_messages: 10) do
      {:ok, []} ->

      {:ok, messages} ->
          "Received #{length(messages)} messages from queue #{@queue_name}, processing them..."


      {:error, _} = unexpected ->
          "Could not get messages from queue #{@queue_name}, reason: #{inspect(unexpected)}"

  defp get_messages(queue_url, opts) do
    result =
      |> SQS.receive_message(opts)
      |> ExAws.request()

    with {:ok, %{body: %{messages: messages}}} <- result, do: {:ok, messages}

  def process_messages(messages) do
    Enum.each(messages, fn message ->
      Logger.info("Handling message: #{inspect(message)}")
      # do interesting stuff here


  @impl GenServer
  def handle_info(:get_messages, state) do

    {:noreply, state}

This is a lot at once, but what it is doing is connecting to the queue and asking for messages. To start the consumer we need to add it to the application supervisor.

We are missing one thing though. After we are done with the message we should remove it so that no other consumer will also process it. We need another function in the ExAwsExample module:

def delete_message(queue, receipt) do
  |> SQS.delete_message(receipt)
  |> ExAws.request()

The receipt is an identifier for the message, contained in the message itself. Let’s call delete_message/2 after process_messages/1 in SQSConsumer:

|> Enum.each(fn %{receipt_handle: receipt_handle} ->
  ExAwsExample.delete_message(@queue_url, receipt_handle)

Note: You really want to only delete the message if you know the processing of it went well. If something goes wrong consider using deadletter queues.

Putting it together

We now have most of the pieces we need to have a working application.

Full code and instructions on how to run it here.

Happy Elixir AWS hacking!

Jerry - a silly macos automatic mouse move & click toy

Recently I wanted to look into how to move the mouse around in macos. The end result of my research and experimenting is “Jerry” — a little Swift playground snippet I’ve put up as a gist. When you run Jerry it makes your mouse move around and (left) click at each point. The movement uses easing to make it a little more human-like.

While reading up on the API calls, and refreshing my shallow Swift knowledge I stumbled upon the amazing cliclick tool. That’s where I got the easing code from. To move the mouse around and click I could most likely just have used cliclick instead of writing something myself. But it was fun to brush up on Swift and explore the macos mouse API.

Lasso - a Phoenix LiveView app running on Gigalixir

In this post I’ll outline how I built Lasso, my first Phoenix Liveview app and how I deployed it to Gigalixir. The Lasso code is available on github.

I’ve been meaning to play with Phoenix LiveView since the moment it came out. A year or so ago I wanted to build something similar to webhookinbox - which I find quite handy. But I never got started on it. However I had three hours to kill on the train to Code BEAM STO and figured this could be something suitable for trying LiveView on.

The idea is to have a web page where you can see HTTP requests to a certain URL. The spicy part would be to be able to see those request as they come in — live.

Generate project and add dependencies

Nothing exciting here. Standard phoenix project, without ecto since I don’t want a database for this: mix phx.new --no-ecto lasso

After that I followed the instructions for adding LiveView from its README.

Main pieces

I quickly realize I needed a couple of different components:

  1. A way to create a new, unique URL destination (that expires) which can accept HTTP requests
  2. A page that you can share to view those requests as they come in
  3. Some sort of state of historic requests per destination (sliding window of some size)

Creating the destinations

I knew I didn’t want the destination URLs to be guessable, and they needed to be unique. So I made each url contain a generated uuid. To keep track of the existing destinations the uuid is added to a an in memory cache with a TTL of 24 hours.

When a request is made to the destination we first check that it actually exists in the cache, and if so proceed to handle it, if not the client will get a 404.

An example destination URL would be http://localhost:4000/hooks/3c72c523-9f8c-4ae9-98c9-faa878d12f58

Visualizing the requests

Here’s where LiveView gets into the picture. To view the requests we need a page connected to the destination, the uuid is perfect for that. The visualization URL for the example destination would be http://localhost:4000/lasso/3c72c523-9f8c-4ae9-98c9-faa878d12f58. Here once again we need to check that the uuid is in the cache before serving the page. To serve the page we will finally need to create a LiveView.

Most of the examples I’ve seen of LiveView has the template inlined in code with a sigil. I wanted to use a separate file for it so the first thing I did was to create a .leex file and put some text with a link to the destination URL in it. This works just like a normal .eex template so far, nothing live about it yet.

Live updates

I had no idea how to actually “trigger” the update. Most examples I found used some element on the page that was served (using phx-click) to trigger updates. I needed something entirely decoupled from the page to be the trigger (specifically the request to the destination URL).

Luckily I found something interesting in the phoenix_live_view_example repo. In the CRUD demo whenever a user is created/updated/deleted a liveview will know about it. To accomplish that the demo uses Phoenix.PubSub.

I had never used Phoenix PubSub before so got pretty excited in trying that out as well. At the same time it seems a bit overkill since I will only run Lasso on one machine (to start with). I decided to try it out anyway.

When mounting the LiveView for the destination we start subscribing to the pubsub topic, again using the uuid for the destination as key. Then whenever we handle a request for a destination we push request details to the pubsub topic. Since the LiveView view is a process itself it can then handle_info and pattern match on the type of messages we expect from the topic, and finally push the update to the socket.


Now we can see updates on the page! That’s very cool. To make the project more useful you might want to be able to share the view URL and let others see the same thing, including any requests that came in before they opened the page. To accomplish that we have to store some state of previous requests.

Luckily we have the cache where the uuid is stored, with a very small change we can associate it with a list of requests. And every time a destination URL is hit we append to that list. I decided to limit the history to the last 100 requests.

After this feature was added I spent quite some time on trying to make the presentation of each request make sense and look at least half decent. Not qutie sure I succeeded :)


I’ve tried various services for deploying and running apps. Bare metal, Mesos/Marathon, Heroku, AWS Elasticbeanstalk, Kubernetes. Each comes with its pros and cons. For personal projects my requirements are usually: Low cost and low effort.

Heroku free tier tends to wins on both of those fronts. This time I needed something that would not lose its in-memory contents too frequently though (heroku free tier apps are restarted now and then).

Gigalixir’s free tier promises fewer constraints for my app, so I decided to give that service a spin.

Following the Gigalixir guide on phoenix apps made the setup and deploy completely pain free - it just worked on the first attempt. Adding SSL was also smooth. If you are building an elixir app and want somewhere simple and cheap to run it definetly give Gigalixir a try!


  • Phoenix LiveView is very cool and super useful for adding some interactive elements to your elixir phoenix app.
  • Gigalixir is impressively easy to use and its free tier quite capable
  • Train rides are the best (I wrote this blog post on the way back from Code BEAM STO)

Good stuff in 2018

Happy New Year. It’s 2019. Here’s a short retrospective on some of the things I enjoyed in 2018.


I’ve listened to a lot of different stuff this year. Here are some of my favorites: Top 16 releases 2018


  • Isle of dogs
  • Three Billboards Outside Ebbing, Missouri
  • Death of Stalin
  • Mandy


  • Kazuo Ishiguro - Nocturnes
  • Margaret Atwood - The handmaid’s tale
  • Martin Kleppmann - Designing Data-Intensive Applications (my notes)
  • Francesco Cesarini - Designing for Scalability with Erlang/OTP
  • Stanisław Lem - The invincible


  • Aphex Twin at Funkhaus in Berlin
  • Rival consoles at Prince Charles in Berlin
  • Nils Frahm at Malmö Live konserthus in Malmö
  • Sleaford mods at Columbiahalle in Berlin
  • Fever Ray at Columbiahalle in Berlin


Skateboard films

Tv series

  • Maniac
  • Dilan & Moa
  • Black Mirror s4
  • The handmaid’s tale
  • Stranger things s2
  • Dark


  • Cinque e cinque in Livorno (sandwich)
  • Chili pepper plants 🌶
  • Masthugget parking garage (skate spot)
  • Negroni (cocktail)

On Designing Data-Intensive Applications

Designing Data-Intensive Applications — The big ideas behind reliable, scalable, and maintainable systems — is a book written by Martin Kleppmann, published in 2017 by O’reilly.

I read this book cover-to-cover, which is probably not necessary or even recommended. But it worked quite well for me, although it took a long time to finish. Towards the end of the book I started thinking about writing a sort of review of it. Partly to make some of the ideas and concepts stick better, and partly because I think it’s a great book that more people should read.

While reading technical books I usually make markings on sections I like to get back to. Either to re-read, or maybe as a starting point for a small experiment of my own, or just because the page/section is really good. In this post I’ll write a short snippet about each such marking I made for Designing Data-Intensive Applications.

My expectations on the book was different than the actual content. I had the impression that DD-IA was going to be more like cookbook style read, recipes on how to architect scalable backend apps. That is not the case at all, instead the book focuses on the underlying theory for vital pieces used in such designs. Things like how database features work under the hood, theories for distributed systems, concurrency truths, and tradeoffs around consistency, fault tolerance etc. You might think “why should I care about stuff like that?”. My answer is that this sort of stuff give you guidance on what technology to chose for certain use-cases, and what characteristics you can expect from it. Plus it’s great material if you ever want to build some database like system yourself.

My markings

Constructing and maintaining SSTables

Where: Page 78, Chapter 3: Storage and retrieval

I think my main motivation for marking this section is that it clearly outlines how some storage engines work. It describes an algorithm that is “essentially what is used in LevelDB and RocksDB”! Also cool that red-black trees are mentioned as a good way to store a sorted structure in memory for a storage engine. I quite recently implemented a persistent red-black tree in Elixir myself.

Partitioning and Secondary Indexes

Where: Page 206, Chapter 6: Partitioning

My interest in search engines triggered this marking. Here we get a deep dive into why secondary indexes are so useful, how they can be implemented, and some tradeoffs.

I really like the way Kleppmann makes it easy to understand how partitioning affect secondary indexes. And how the scatter/gather approach suffers from some issues with tail latency amplification (all too familiar after having worked on Elasticsearch backed projects).

Preventing Lost Updates

Where: Page 242, Chapter 7: Transactions

Description of the very common issue of data loss due to concurrent updates; where you read a value, (possibly) modify it and then write it back. Then follows a list of solutions to the problem with plenty of details:

  • Atomic write operations
  • Explicit locking
  • Automatically detecting lost updates
  • Compare-and-set
  • Conflict resolution and replication

Very practical information that helps every developer.

Safety and liveness

Where: Page 308, Chapter 8: The Trouble with Distributed-Systems

I learned that there are two important kinds of properties for correctness of distributed algorithms; safety and liveness. These two kinds can help us to deal with difficult system models. Typically in distributed systems you want the safety properties to always hold—even if all nodes crash for example. Liveness properties can be more flexible.


This property can be informally defined as nothing bad happens, the more precise definitions is:

If a safety property is violated, we can point at a particular point in time at which it was broken (…). After a safety property has been violated, the violation cannot be undone — the damage is already done.


A liveness property works the other way round: it may not hold at some point in time (for example, a node may have sent a request but not yet received a response), but there is always hope that it may be satisfied in the future (namely by receiving a response).

Linearizability is stronger than causal consistency

Where: Page 342, Chapter 9: Consistency and Consensus

This is a section I want to re-read to make sure I understand it completely. I’ve tried to immerse myself more into the distributed systems world and all the interesting and sometimes mind bending problems. So much research go through and so much head scratching.

What caught my eye in this section is this passage:

In fact, causal consistency is the strongest possible consistency model that does not slow down due to network delays, and remains available in the face of network failures.

In many cases, systems that appear to require linearizability in fact only require causal consistency, which can be implemented more efficiently.

Apparently this is kind of recent research, and we may still see cool things be implemented based on this!

Chapter 9 Summary

Where: Page 373, Chapter 9: Consistency and Consensus

Not going to include everything, but I liked this summary as a sort of reference on some common terms and their meanings in distributed systems.

  • Linearizability: Popular consistency model. Goal is to make replicated data appear as though there is only a single copy. All operations act on the single copy atomically. Appealing because it’s easy to understand. Downside of being slow. Especially in environments with large network delays (cloud).
  • Causality: Imposes an ordering on events in a system. Weaker consistency model than linearizability. Some things can be concurrent. Does not incur coordination overhead like linearizability, also less sensitive to network delays.
  • Consensus: All nodes agree on the decision. Many problems can be reduced to the consensus problem; Linearizable compare-and-set registers, Atomic transaction commit, Total order broadcast, Locks and leases, Membership/coordination service, Uniqueness constraint. Straightforward on single node, or single node decision maker.

Using logs for message storage

Where: Page 447, Chapter 11: Stream Processing

A very good breakdown how using a log can be used to power a message broker (like Kafka or Kinesis). Both in the simplest way with a single node, and how it can be partitioned across machines.

I think it would be a really fun experiment to try and build a simple message broker like this myself.

Designing for auditability

Where: Page 531, Chapter 12: The Future of Data Systems

I enjoyed this whole chapter a lot, and some of the concerns Kleppmann bring up really resonate with me.

One thing that may become more important in the close future is how auditable our systems are. If you mutate a table in a relational database, the previous state is lost and there might not be a trace of why something was changed. This can be a huge problem.

Event-based systems have an advantage here if the event is kept around (event sourcing). We need to design systems that are deterministic, repeatable and explicit around data flow. Immutable events can allow us to do time travel debugging.

Feedback loops

Where: Page 536, Chapter 12: The Future of Data Systems

This section is about (bad) feedback loops in recommendation systems and other such predictive applications.

We’ve all heard about social media echo chambers or filter bubbles. Recommendation systems can be a huge contributing factor to this phenomenon.

Another example is if an employer use credit-scores to evaluate potential hires. If you miss a payments on bills (due to some catastrophic event beyond your control), your credit-score suffer which in turn makes it harder for you to get a job and get back on track.

Difficult problems that must be tackled head on; more and more ML is powering important parts of all of our lives with increasing adoption rates. Scary stuff even if built with good intentions.

Data as assets and power

Where: Page 540, Chapter 12: The Future of Data Systems

I can’t even count how many scandals about the misuse of user data that has been reported on in 2018. It seems to be prevalent and have left me very skeptical and paranoid of all online services.

In this section Kleppmann argues that we should perhaps view data not as an asset like any other, but more like a hazardous one. We need to safe guard it against current abuse and also make sure it does not fall into the wrong hands.

When collecting data, we need to consider not just today’s political environment, but all possible future governments. There is no guarantee that every government elected in the future will respect human rights and civil liberties, so “it is poor civic hygiene to install technologies that could someday facilitate a police state”

In conclusion

Read the book. Learn about how databases work. Learn the underlying ideas of the technology you might already be using and what you can expect from it. Get better at deciding on what technology to use and when. The choices you make will have very real impact on the development and operation of your systems.

Think about your system holistically: correctness, behaviour in face of network partitions, observability, traceability, auditability. Think about how the system affects users and non-users. Think about data as a potentially hazardous asset.

This is a book I recommend all developers read.

Finding bugs with property based testing

Property based testing has been on my radar ever since being forced to write QuickCheck tests for the introduction to functional programming course at Chalmers university some 10+ years ago.

However examples based tests has always felt more intuitive, and maybe more importantly has been the go to approach to development in the places I’ve worked.

Recently I’ve tried to force myself into the invariant based reasoning mode, and apply property based testing. This was prompted by some interesting issues around testing inputs consisting of date times and time zones. Edge cases lead to head scratching bugs despite good unit test coverage. (If you want a good introduction I recommend Jessitron’s post Property-based testing: what is it?)

I still haven’t gotten to the stage where I can get property based tests to drive design of the implementation. But I’ve gotten much more comfortable at wielding this amazing tool.


Edge cases

This probably doesn’t come as a surprise. Generating inputs instead of selecting them yourself yields impressive coverage and can catch the most subtle cases.

Algorithms and data-structures

Property based tests seems like the perfect fit for these type of functions. Maybe it’s because there is usually quite clear desired properties in this space.

I was happy and impressed by how fast property based tests found bugs in my own (persistent) data structure implementations in Elixir.

Large combination of inputs

More generally, it’s getting apparent to me that anything where there the combined input set is very large property bases tests might be a good fit.


I think there are way more opportunities to improve confidence in any code base with property based tests than what we usually think.

I’m also convinced that the — “what needs to go right?” — reasoning mode in your arsenal can make some hairy issues a bit easier to work with.

Experiment with gzipped and chunked HTTP responses with Plug

Recently I wanted to add gzip support to an API endpoint which could give back very large amounts of JSON. Internally the API gets data from a source where the size of the payload is not known upfront. To deliver the data it uses a HTTP (1.1) feature called chunked transfer encoding with Plug.Conn.chunk(). After finishing the implementation I found some, to me, surprising results.


  • We do not want to keep the data in memory because it can be very large, and there may be many similar requests coming in.
  • The data itself comes from some source we can stream from, like a database or AWS s3.
  • Elixir implementation


Adding compression of the data using erlang’s zlib will speed up the transfer iff the data size exceeds some threshold.


The easiest thing would be to simply gather the data in-memory and then pop it out in one go with :zlib.gzip(mydata) and the correct http headers. But we implemented chunking so that we don’t have to keep the whole data in memory. So what to do? We can compress the chunks. This was not straigh forward to do and I’m still not sure that what I cooked up is the optimal way to do it.


Before getting to the individual chunks we first have to indicate to the client that we are going to send gzipped things.

Plug.Conn.put_resp_header(conn, "content-encoding", "gzip")

Of course we should only do this if the client actually asked for compression — ie has set the request header Accept-Encoding to application/gzip or gzip, deflate.

Test first

I knew exactly what outcome I wanted, but not quite how the implementation would look so I started with a test.

test "should gzip json data if requested" do
  conn =
    |> put_req_header("accept-encoding", "application/gzip")
    |> get("/data")

  assert conn.status == 200
  assert Enum.member?(conn.resp_headers, {"content-type", "application/json; charset=utf-8"})
  assert Enum.member?(conn.resp_headers, {"content-encoding", "gzip"})
  assert {:ok, _} = conn.resp_body |> :zlib.gunzip() |> Poison.decode()

Attempt one

This boiled down to gzipping every chunk payload like so: Plug.Conn.chunk(conn, :zlib.gzip(chunk_payload)). This worked in my ExUnit tests, and using curl but it did not work in any of the browsers I tried (Firefox, Safari, Chrome). In the browsers I either got the first chunk only, or a JSON parse error. Very interesting! This was surely a smell that my implementation was not correct.

It was not clear why the data was not understood by the browsers but this thread on Stack Overflow made me suspect that it had to do with including the full gzip headers in every chunk. The first attempt’s implementation meant: It output one full gzipped “file” per chunk that had to be unpacked individually instead of one large “file” that was fully assembled once the last chunk was sent.

Attempt two

I did not know exactly how to proceed, but figured I should read up on the erlang zlib documentation to see what’s possible.

Getting to know erlang’s zlib

The SO thread mentioned details on window bits and init functions. Is there something like that in the erlang implementation I can control? Yes zlib.deflateInit/6 has a WindowBits option, promising. deflateInit needs a zstream though, what’s that? “A zlib stream, see open/0.” Hmmm okay time to play around in iex.

z = :zlib.open()
:ok = :zlib.deflateInit(z, :default, :deflated, 31, 8, :default)

Fancy, and lots of magic parameters. All of them except 31 for the WindowBits are defaults for deflateInit/6. This is a good start, we have a zstream and have configured it. At this point all I want to do is to compress some text, and be able to uncompress it again.

first = :zlib.deflate(z, "my first string")
second = :zlib.deflate(z, "my second string", :finish)
:zlib.gunzip(first ++ second) # "my first stringmy second string"

With a bit of imagination we can see how this could fit together with the chunks; call deflate on the chunk_payload and send that. Will this approach pass the browser test?

Plugging it in

Awful title aside, here’s how a module which wraps the zlib functionality could look like

defmodule MyApp.ChunkCompressor do
  def init(%Plug.Conn{} = conn) do
    |> Plug.Conn.assign(gzip, gzip_requested?(conn)) # gzip_requested? return boolean based on accept-encoding header
    |> content_encoding()
    |> init_zstream()

  def chunk(%Plug.Conn{assigns: %{gzip: false}} = conn, payload, _deflate_option) do
    Plug.Conn.chunk(conn, payload)

  def chunk(%Plug.Conn{assigns: %{gzip: true, zstream: zstream}} = conn, payload, :sync) do
    compressed_payload = :zlib.deflate(zstream, payload, :sync)
    Plug.Conn.chunk(conn, compressed_payload)

  def chunk(%Plug.Conn{assigns: %{gzip: true, zstream: zstream}} = conn, payload, :finish) do
    compressed_payload = :zlib.deflate(zstream, payload, :finish)
    Plug.Conn.chunk(conn, compressed_payload)

  defp init_zstream(%Plug.Conn{assigns: %{gzip: false}} = conn), do: conn
  defp init_zstream(%Plug.Conn{assigns: %{gzip: true}} = conn) do
    with zstream <- :zlib.open(),
         :ok <- :zlib.deflateInit(zstream, :default, :deflated, 31, 8, :default) do
      Plug.Conn.assign(conn, :zstream, zstream)

And here’s a phoenix controller using our ChunkCompressor:

# ...

def data(conn, _params) do
  conn =
    |> put_resp_content_type("application/json")
    |> MyApp.ChunkCompressor.init()
    |> send_chunked(200)

  with {:ok, conn} <- stream_data(conn),
       {:ok, conn} <- MyApp.ChunkCompressor.chunk(conn, "", :finish) do

def stream_data(conn) do
# Stream from source
|> MyApp.ChunkCompressor.chunk(conn)

Using this we can finally verify with a browser that we do get all the expected json. Celebration!

Comparison against uncompressed transfer

Let’s take a look at how our compressed data transfer compares against our uncompressed baseline. Gotta be honest I was quite confident that it would be way faster.

CompressionData sizeTime takenAvg. d/l speedNotes
No1125M0:00:2055.1M/sBaseline w/ 2 CPUs
Yes295M0:00:555427k/sDefault compression

Wha? How can it be!? Average download speed is a tenth of the baseline — WTF!

George Costanza is also confused

Very disappointing. But I slowly realized that the overhead of compressing a chunk simply didn’t outweigh the size benefit. I started experiment with the zlib parameters to see if I could improve things:

CompressionData sizeTime takenAvg. d/l speedNotes
Yes346M0:00:359.9M/s:best_speed compression level
Yes348M0:00:389337k/sHighest mem_level
Yes755M0:00:3521.5M/s:huffman_only strategy

Turned out it was tough to beat the baseline’s 20s time taken. The last option was getting close in download speed but then the compression was garbage. Not worth. I implemented parallel compression, but that wouldn’t fly:

** (EXIT) an exception was raised:
    ** (ErlangError) Erlang error: :not_on_controlling_process

Because zlib streams should only be changed from one thread.

There’s an alternative erlang zlib library called ezlib “optimized for streaming” that sounded very promising — unfortunately I couldn’t get it to work.


It seems like the original idea of compressing chunks is not giving enough benefits, the transfer speed is worse than the uncompressed version. After the fact I guess it simply makes sense, doing work per chunk — especially computationally intensive work — inflicts a heavy penalty on the throughput.

There’s one more thing I can think of that might help the compression and that is to increase the size of the chunks that we get from the data source (right now they are around 150 - 1000KB). This way deflate/2 is called fewer times. I could also try smaller values for window bits, but I am not sure that would do much.

All in all this was a “failed” experiment, although fun and I learned some things.


HTTP/2 does not support chunked transfer encoding (it even forbids it). I don’t know how a solution would look like for HTTP/2, but maybe I’ll write a follow-up if I get to experiment a bit with that.

Game of Life in Elixir and Scenic

After seeing Boyd Multerer’s 2017 talk on Scenic – an OpenGL backed UI framework for Elixir/Erlang – I have been following its progress. This year at ElixirConf we got a new talk by Mr. Multerer with even more impressive and polished demos, and most importantly the Scenic repo went public.

I knew I wanted to build something simple to get familiar with the framework. Yesterday I came up with the brilliant idea to add a graphical view of my very old Elixir game of life golex. Golex was one of my first Elixir projects, conceived back in 2013 when Elixir was still in beta (hipster cred?). I cloned the project, built it, and ran the text based simulation. Worked like a charm and the code looked modular enough to build a graphical view on top of.

Getting started with Scenic

This was pretty straightforward with clear instructions for macOS in the Getting started guide. It didn’t take long at all before I was looking at the sample app and clicking around.

I created a new scene, and looked at the other ones for clues on how to proceed.

The grid

I decided to start by just drawing a 2D grid in my GameOfLife scene to get my feet wet.

First attempt:

alias Scenic.Graph
alias Scenic.Primitives

@width 800
@height 600
@cell_size 10

@grid Graph.build()
  |> Primitives.line({{0, 0}, {@width, 0}}, stroke: {1, :white})
  |> Primitives.line({{0, @cell_size}, {@width, @cell_size}}, stroke: {1, :white})
  |> Primitives.line({{0, 2 * @cell_size}, {@width, 2 * @cell_size}}, stroke: {1, :white})


This is already old after three lines. I will write a function since I’ve heard that computers are good at executing repetitive tasks. Second attempt:

def build_grid(graph, {width, height}, spacing) do
  horizontal =
    Enum.reduce(0..height, graph, fn y, acc ->
      Scenic.Primitives.line(acc, {{0, spacing * y}, {width, spacing * y}}, stroke: {1, :white})

  Enum.reduce(0..width, horizontal, fn x, acc ->
    Scenic.Primitives.line(acc, {{spacing * x, 0}, {spacing * x, height}}, stroke: {1, :white})

# in init/2
|> build_grid({@width, @height}, @cell_size)
|> push_graph()

– that’s better! I actually found a bug in this function as I was writing this post (proof). Here’s the beautiful output:

golux grid

The cells

Since the grid drawing was so painless I was eager to get the cells out on the board. This turned out to also be quite easy. I stared at the different primitives for a bit expecting something like rect(x, y, width, height) (like in processing or quil). I found quad/3 instead. It wasn’t really clear to me how to translate a rect at first, so I thought let’s just go with quad now to Get Shit Done.

def cell_graph(graph, %Golex.Cell{alive: false}), do: graph
def cell_graph(graph, %Golex.Cell{position: {x, y}, alive: true}) do
  xp = x * @cell_size
  yp = y * @cell_size

    {{xp, yp}, {xp, yp + @cell_size}, {xp + @cell_size, yp + @cell_size}, {xp + @cell_size, yp}},
    fill: :white

So there’s a couple of things going on here. First off the pattern matching makes it really clear that we simply ignore dead cells. For alive cells we need to calculate where on the grid they should show up. That’s what xp and yp is for (naming is hard etc). Now obviously we need to call this function for every single cell in our world.

def world_graph(graph, %Golex.World{cells: cells}) do
  Enum.reduce(cells, graph, fn {_, cell}, acc ->
    cell_graph(acc, cell)

# init/2 can now do
world = Golex.random_world(...) # Get a new world from golex

|> world_graph(world)
|> build_grid({@width, @height}, @cell_size)
|> push_graph()

We have the living cells on the grid, great. Except the game still sucks. We need to make it move.


I’m sure there are a bunch of ways to do this but I really liked the idea of making the scene send a message to itself on a fixed interval, and that would trigger the world update + re-render. To achieve that we reach into our Erlang toolbox and find :timer.send_interval/2 (also used in other Scenic demos). I figured that updating the scene once a second to start with should be conservative enough. I had no clue or expectations on how slow/fast scenic and golex would be.

To handle the message we have to implement handle_info/2 – standard OTP stuff.

# In init/2
:timer.send_interval(1_000, :world_tick)

def handle_info(:world_tick, state) do
  new_world = Golex.world_tick(state.world)

  |> world_graph(new_world)
  |> build_grid({@width, @height}, @cell_size)
  |> push_graph()

  {:noreply, %{state | world: new_world}}

It’s slow :(

The one second update interval turned out to not be quite conservative enough. The updates looked a bit dodgy / not good. I measured how long it took to do the stuff in handle_info/2 above and it took a bit more than a second (~1.2). Was Scenic really this slow? Of course not. It was golex’ world_tick/1 function that was very very naive – but I fixed that! Didn’t worry about performance after that and could lower the timer interval a lot (to 100ms).

Animated game of life


Part of the fun of game of life is seeing different starting conditions play out. So adding a way to restart the game with a fresh world would be cool. A simple left click on the board to do that maybe? I was a bit confused at first on how to implement this, because I kept trying to use filter_event/3. That was wrong because I don’t have any components in my Scene. Components can generate events. In our case we need to deal with lower level inputs with handle_input/3. A nice way to understand what events are available and how to handle them is to add:

def handle_input(msg, _, state) do
  IO.inspect(msg, label: "handle_input")
  {:noreply, state}

to your scene. This way every time some input event happen you will see how it looks. To handle mouse left click release I just pattern matched like this:

def handle_input({:cursor_button, {:left, :release, _, _}}, _input_context, state) do
  IO.puts("Generating a new world")
  new_world = Golex.random_world(div(@width, @cell_size), div(@height, @cell_size))

  {:noreply, %{state | world: new_world}}
def handle_input(_msg, _, state), do: {:noreply, state} # Need to handle all other events

Voilà we have new worlds on left mouse click.


This section came about after getting some feedback from Mr. Multerer himself on twitter. What happens with the performance if we switch the cell drawing from quads to rects?

time difference for quads and rects

PercentileQuads (µs)Rects (µs)

Rects with translation is a bit faster than quads, and probably better memory wise (although I haven’t actually verified that). A note on the numbers here, they were gathered from a small sample size of 27 ticks for each type with timer.tc/1. What I measured was the time it took to update the game of life world and render with Scenic.

Wrapping up

This turned out to be a fun way to get to know Scenic at least a little bit. I look forward to doing more with this promising framework.

Here’s the code which is using rect instead of quads, and has some additional controls: golux

Creating a statically generated site - the hard way

Forget about installing Jekyll, Next, Hugo, or any of the other mainstream static site generators out there. To make Förvillelser I opted for the Carl Sagan method:

If you wish to make apple pie from scratch, you must first create the universe

I knew that I wanted a site generator that I could mess around with properly. Right now I am very fond of Elixir so naturally I looked at the elixir options out there. Obelisk immediately caught my attention because of its nice mix task UX. I followed the README to create my site, and I hit the first road block right away. Obelisk didn’t build out of the box. No problem, I updated one of its dependencies and continued. Next I tried to build my site, this time obelisk crashed and it wasn’t obvious why. That prompted me to look closer on its github page – maybe someone else had the same problem? Uh oh, lots of PRs with no attention, and the project seemed generally abandoned by its creator. One guy even wanted to take over the project but hadn’t gotten a response :(

Okay. That messed up my plan, since if I hack on this thing I’d like to give the result back to the community. Or at least have the option to do so. Forking seemed unfun, as did the prospect of debugging and fixing. Looked at Serum as well. Nice but not quite what I was after. Oh well time to roll up my sleeves and get cranking on My Own Static Site Generator! How hard can it be!? Haha!

I, as a software developer by trade, of course had some requirements. First I wanted my new shiny sites pages and posts to be written in Markdown. I wanted these documents to be versioned controlled. And I wanted to be able to deploy changes to my site with git commit+push.

Nothing out of the ordinary.

I got a first good enough version of Gonz together in an evening or so. I started working on Förvillelser early so that I could feed the Gonz development with fixes and features I needed for my particular site.

After a few more days of sporadic hacking I realized that I needed to publish Förvillelser somewhere to actually deliver value to my customer (me). Around this time other requirements popped up in my head:

  • I want SSL on the thing. http without s is so lame!
  • I don’t want to actually run a server myself. All the dev without the ops for this please.

With these in mind it seemed my choices were either github pages or netlify. I really liked the prospect of not having the baked HTML files in my repo, only the raw markdown files. Not sure why, I guess something something tidy, ridiculous nerd bullshit. That meant using netlify, because they can build the site for you. Cool beans 😎

I signed up and set up my site. Boom – “Site deploy failed”: make: mix: Command not found. Not sure why I expected them to support Elixir out of the box, but hey a guy can dream. Started looking around for some neat workaround or whatever. No real luck, but I did find netlify’s build-image repo – all open source and beautiful. I forked that, added erlang and elixir and sent a PR. Surprisingly it was approved (eventually) and even merged to master!

In conclusion

  1. Build your own static site generator in a non-mainstream programming language X
  2. Build the actual site
  3. Deploy the site somewhere that doesn’t support X
  4. Contribute changes to that place’s stack to add support for X
  5. P R O F I T 🎉

Goodbye Tumblr

After using Tumblr as my blogging platform of choice for the last 10 years I have decided to move to something else. The move is mostly triggered because of philosophical reasons. I think the internet is getting increasingly walled off into corporate silos and I don’t like it. Tumblr is not a particularly bad, or big player in terms of this issue but it’s definitely part of the problem.

This time I decided to go back to self-hosting despite the amount of maintenance work that means. I like the idea of owning all my content and having full control. While I’m at it I can consolidate my personal site and blog. Maybe later I will go even further by hosting my git repos myself. We’ll see about that though.

I’ll have to come up with some plan on how to add all my old Tumblr posts here. I’ve exported the tumblr blog and got a 309MB size zip file 😅

Content from joel.vorce.se will also be added here. I will probably also point that DNS to here.