ElixirConf.EU 2018

I attended ElixirConf.EU 2018, it took place in Warsaw this time. The food was fantastic, the weather was very favourable and the presentations a blast.

The Food

Announcement: This blog is from now on about food ..Not.
That zapiecek place was sooo good though. We ate there almost twice a day. They had those ravioli-like pasta called pierogi, absolutely mouth-watering. I might visit Poland again just for the food!
</food>

Yummy pierogis

We also enjoyed strolling around Łazienki park.

Lazienki park

Conference - Day 1

Keynote - Introducing HDD: Hughes Driven Development - José Valim

{slides, video}

The title of the talk “Hughes Driven Development” is a reference to John Hughes who pioneered in the area of property based testing as one of the developers of QuickCheck.

The Elixir core team is developing a library, stream_data for property-based testing.

A test for a String concatenation function could look like:

check all left <- string(),
  right <- string() do
  string = left <> right
  assert String.contains?(string, left)
  assert String.contains?(string, right)
end

This library is likely to be part of ExUnit as of Elixir@1.7.

José also talked about some of the progress in the development of code formatter (mix format) and its algorithmic foundations:

Pretty Printing - Wadler
Strictly Pretty - Linding

At the end of his talk he was awarded a 3D-printed Elixir logo by the people of kloeckner.i. What a nice surprise!

🏅🏅🏅

Robust Data Processing Pipeline with Elixir and Flow - László Bácsi

{slides, video}

His talk revolved around the topic of using Flow to express complex computations.

If you’re thinking re-writting everything using flow, don’t. It’s an overkill.

defmodule ImportFeeds do
  @spec flow(Enumerable.t()) :: Flow.t()
  def flow(%Flow{} = flow) do
    flow
    |> Flow.map(&ensure_loaded/1)
    # |> ...
    |> Flow.emit(:state)
  end

  def flow(input) do
    input |> Flow.from_enumerable() |> flow()
  end
end

and now such functions can be composed like:

feeds
|> PrepareFeeds.flow()
|> ImportFeeds.flow()
|> Flow.run()

References

Going low level with TCP sockets and :gen_tcp - Orestis Markou

{slides, video}

He did a live demo of a client-server pair with code from this repo using just :gen_tcp.

Code to make a simple HTTP request:

defmodule Active do
  def hello do
    {:ok, socket} = :gen_tcp.connect('www.google.com', 80, [:binary, active: true])
    :ok = :gen_tcp.send(socket, "GET / HTTP/1.0 \r\n\r\n")
    recv()
  end

  def recv do
    receive do
      {:tcp, socket, msg} ->
        IO.puts msg
        recv()
      {:tcp_closed, socket} -> IO.puts "=== Closed ==="
      {:tcp_error, socket, reason} -> IO.puts "=== Error #{inspect reason} ==="
    end
  end

end

Active.hello()

In this case the active: true option converts packets to erlang messages (see: doc).

An interesting highlight is that sockets are mutable and can change on-the-fly (for example between :binary and :active modes). This is demonstrated in this example which is intended to implement a small fragment of the memcached protocol. It establishes a connection in packet: :line mode which reads from the socket until a newline is encountered, parses the bytesize of the next message, changes the socket mode to packet: 0 (raw mode) and finally reads from the socket using the parsed bytesize.

SSH Server/Client in Erlang

{slides, video}

Milad started his presentation with a brief history and feature overview of SSH. Erlang has had a builtin SSH library since 2005 and it’s quite easy to start an SSH server from your Elixir code.

Example:

defmodule SshTalk.Server do
  @sys_dir String.to_charlist("#{Path.expand(".")}/sys_dir")

  def basic_server do
    :ssh.daemon(
      2222,
      system_dir: @sys_dir,
      user_passwords: [{'foo', 'bar'}]
    )
  end
end

You can then start your server using:

iex -S mix

and connect to it using any SSH client using:

ssh foo@localhost -p 2222

Neat, isn’t it? It gets even cooler as you can have your server provide an Elixir shell using:

def server_with_elixir_cli do
  :ssh.daemon(
    2222,
    system_dir: @sys_dir,
    user_dir: @usr_dir,
    auth_methods: 'publickey',
    shell: &shell/2
  )
end

def shell(username, peer) do
  IEx.start([])
end

Milad has also used the :ssh module to build GixirServer which is a Git server working over SSH.

SPAs Without the SPA - Ian Duggan

The following quote catch my attention at the start of this presentation:

Has Rails ruined a generation of programmers?

Keep it simple
Be suspicious of frameworks
Take only what you need

Ian is the creator of a framework to build SPAs using Elixir. The framework is called Presto.

Some of the design goals were:

The feel and Reactivity of React
The simplicity of Elm’s model/update/view
Completely in Elixir (minimal JavaScript)

Code samples:

# Updating a component
@impl Presto.Page
def update(message, model) do
  case message do
    %{"event" => "click", "id" => "inc"} -> model + 1
    %{"event" => "click", "id" => "dec"} -> model - 1
  end
end

# Rendering
def render(model) do
  div do
    "Counter is: #{inspect(model)}"

    button(id: "inc", class: "presto-click") do
      "More"
    end

    button(id: "dec", class: "presto-click") do
      "Less"
    end
  end
end

HTML markup is generated in Elixir-space using taggart.

Conference - Day 2

Keynote - Building a Task Queue System with GenStage, Ecto, and PostgreSQL - Evadne Wu

{slides, video}

This presentation was an eye-candy, well researched and delivered beautifully.

She begins by exploring some basic scheduling concepts, like for example task deadlines:

Type	If Missed	Applicable?
Soft	People Unhappy	Yes
Firm	Results Useless	Yes
Hard	Very Bad Things	No

Don’t treat your RDBMS as a dumb data store

People will try to integrate your databases directly, regardless of design intent
Try to encode essential constraints directly into the database

Using PostgreSQL for background jobs can be as simple as:

Function

CREATE FUNCTION jobs.enqueue_process_charges() RETURNS trigger AS $$
  BEGIN
  INSERT INTO jobs.process_charges (id)
  VALUES (OLD.id);
  RETURN OLD;
  END;
$$ LANGUAGE plpgsql;

Trigger

CREATE TRIGGER enqueue_process_charges
  AFTER UPDATE ON purchases
FOR EACH ROW
  WHEN NEW.status = 'processing'
EXECUTE PROCEDURE
  jobs.enqueue_process_charges();

Notification

CREATE FUNCTION new_job_notify() RETURNS trigger AS $$
 BEGIN
 PERFORM pg_notify('new_job', row_to_json(NEW)::text);
 RETURN NEW;
 END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER notify
  AFTER INSERT ON jobs
FOR EACH ROW
  WHEN NEW.status = 'pending'
EXECUTE PROCEDURE
  new_job_notify();

Listening for Notifications

config = Repo.config |> Keyword.merge(pool_size: 1)
channel = "new_jobs"

{:ok, pid} = Postgrex.Notifications.start_link(config)
{:ok, ref} = Postgrex.Notifications.listen(pid, channel)

receive do
  {:notification, connection_pid, ref, channel, payload} -> # ?
end

Caveats

PostgreSQL uses a single global queue for all async notifications
All listening backends get all notifications and filter out the ones they don’t want (performance degrages as listeners are added)
There’s a maximum payload size

Using Elixir to Build a High-Availability Fleet Management Solution for Autonomous Vehicles - Serge Boucher

Serge begins with a description of the product of Easy Mile, a company specialising in autonomous vehicle technology.

Highlights

Their stack is:

Custom Protocol — Protobuf over TCP/IP / MQTT
Elixir
Phoenix
ES6/React/Redux/Brunch/Flow/Jest/Storybooks
Kubernetes
AWS

10 things Serge wish he knew before starting the project:

GenServers aren’t classes
Put all Business Logic in Pure Functions
ExUnit is for Unit Testing
You Don’t Need “Medium” Tests
Monolith is not a Bad Word (relative-tweet)
Heroku isn’t the Real World
There Are Missing Libraries
The Community is Great!
Elixir is Fast Enough
remote_console is awesome!
Compilation Time Rises Quickly
Erlang Programmers are Awesome
High Availability is Builtin

About “Monolith is not a bad word”, they started with a microservices based architecture which included 3 Elixir services. Their fleet manager application used distributed Erlang PubSub and at some point someone asked “Why do we have those 3 different things separated?”. In the end they merged it to a monolith.

Coupling Data and Behaviour - Guilherme de Maio

{slides, video}

Finally, another downside to object-oriented
programming is the tight coupling between function
and data. In fact, the Java programming language forces
you to build programs entirely from class hierarchies,
restricting all functionality to containing methods in a
highly restrictive “Kingdom of Nouns” (Yegge 2006).

– Fogus/Houses | The joy of Clojure

The problem I have with Erlang is that the language is
somehow too simple, making it very hard to eliminate
boilerplate and structural duplication. Conversely, the
resulting code gets a bit messy, being harder to write,
analyze, and modify. After coding in Erlang for some
time, I thought that functional programming is inferior
to OO, when it comes to efficient code organization.

– Sasa Juric | Why Elixir

Elixir protocols can be of help.

A custom implementation of protocols using function dispatching could look like:

defmodule Protocolz do
 def dispatch(function_name, data) do
   struct_module = data.!_struct!_
   :erlang.apply(struct_module, function_name, [data])
 end
end

But dispatching is not so efficient, that’s why elixir does protocol consolidation at compile-time.

Protocol Use-Cases

poison

defmodule Person do
  @derive [Poison.Encoder]
  defstruct [:name, :age]
end

defimpl Poison.Encoder, for: Person do
  def encode(%{name: name, age: age}, opts) do
    Poison.Encoder.BitString.encode("!#name} (!#age})", opts)
  end
end

elasticsearch-elixir

defimpl Elasticsearch.Document, for: MyApp.Post do
  def id(post), do: post.id
  def type(_post), do: "post"
  def parent(_post), do: false
  def encode(post) do
    %{
      title: post.title,
      author: post.author
    }
  end
end

Property-Based Testing is a Mindset - Andrea Leopardi

{slides, video}

Most common way of testing is by writing unit-tests. Example-based testing which can also be considered table-based.

test "sorting" do
  assert sort([]) == []
  assert sort([1, 2, 3]) == [1, 2, 3]
  assert sort([2, 1, 3]) == [1, 2, 3]
end

input	Output
[]	[]
[1, 2, 3]	[1, 2, 3]
[2, 1, 3]	[1, 2, 3]

Unit-tests are easy to write, good for regressions and non-corner cases, but it’s hard to express properties.

Andrea is working on stream_data, which is a property-based framework which generates valid inputs and tests for the properties of the output.

check all list <- list_of(int()) do
  sorted = sort(list)
  assert is_list(sorted)
  assert same_elements?(list, sorted)
  assert ordered?(sorted)
end

The framework exposes generators to produce values to test their output. The framework tries to prove you wrong.

iex> Enum.take(integer(), 4)
[1, 0, -3, 1]

Generators are infinite streams growing in complexity as they produce values.

Property-based testing can be stateful. Andrea mentioned Redis and LevelDB and provided examples. There’s a GSOC project for stream_data to work in tandem with dialyzer. There’s ongoing research on automatically inferring generators from typespec types.

There was a very interesting question about the performance degradation of property-based testing and the answer was “It’s terrible, but you can keep some data locally to make it faster”.

Closing Keynote - Chris McCord

Highlights

In Phoenix@1.4 channels will be more flexible
The big feature is HTTP/2 (via cowboy ~> 2.0). You can already opt-in by changing the version of cowboy in your mix.exs
There’s no JS API for HTTP/2
Faster project recompilation
Moving to WebPack. 3 years ago it was a sane choice as it was easier to configure and had better docs

H2 Push

defmodule AppWeb.PageController do
  use AppWeb, :controller

  plug :push, "/js/app.js"
  plug :push, "/images/logo.png"

  def index(conn, _params) do
    render(conn, "index.html")
  end
end

<img src="<%= static_push(@conn, "/img/logo.png") %>" />

<script src="<%= static_push(@conn, "/js/app.js)") %>">
</script>

You can push from the template!

Explicit Router Functions

In Phoenix version >= 2.0 you’ll have to explicitly alias route helpers:

conn
|> put_flash(:info, "Job updated successfully")
|> redirect(to: job_path(conn, :show, job))

# Becomes
alias AppWeb.Router.Router

conn
|> put_flash(:info, "Job updated successfully")
|> redirect(to: Routes.job_path(conn, :show, job))

New Presence JS API

let channel = socket.channel("room:lobby")
let presence = new Presence(channel)

presence.onSync(() => {
  renderUsers(presence.list())
})

presence.onJoin(...)
presence.onLeave(...)

channel.join()

He concluded by making a remark about trusting the database. Elixir developers tend to get overexcited with the capabilities of the Erlang platform and replace parts of their applications which traditionally belonged to the database with GenServers and Tasks.

The database is your friend

It offers:

Transactions
Consistency
Backups
Replication

To replace the database we need better tooling.

Outro

I hope you enjoyed this post. It was meant to be published only a couple of days after the conf, but someone forgot to push the draft to git, changed computer and then had to write it from scratch 😑. Please contact me or submit a PR for corrections.

Team Quiqup {R.Soares, L.Varela and the author} with José

Dimitris Zorbas

Code spells, smells and sourcery

The Food

Conference - Day 1

Keynote - Introducing HDD: Hughes Driven Development - José Valim

Robust Data Processing Pipeline with Elixir and Flow - László Bácsi

Going low level with TCP sockets and :gen_tcp - Orestis Markou

SSH Server/Client in Erlang

SPAs Without the SPA - Ian Duggan

Conference - Day 2

Keynote - Building a Task Queue System with GenStage, Ecto, and PostgreSQL - Evadne Wu

Using Elixir to Build a High-Availability Fleet Management Solution for Autonomous Vehicles - Serge Boucher

Coupling Data and Behaviour - Guilherme de Maio

Property-Based Testing is a Mindset - Andrea Leopardi

Closing Keynote - Chris McCord

Outro

Upcoming Elixir Conferences