Scala on Adam J King

Pull up a seat with FS2 Streams

Thu, 26 Feb 2026 00:00:00 +0000

I usually put FS2 amongst my top favourite libraries in my career. This streaming library powers some of the most popular Scala Typelevel libraries. I often encounter streaming design problems that make me think to myself, “you can do this with a couple lines of FS2”. Enough gushing. I typically find that folks who come across FS2 struggle with creating a correct mental model for how a stream in FS2 works.

This article assumes you know about Cats-Effect and FS2 already and would like to dive deeper. Understanding how FS2 works requires a mental model for how suspended side effects work too.

On the surface, an FS2 Stream looks and feels a lot like a collection (like List). It supports common collections operations like map, flatMap, or filter. In fact, when we go to complete a stream, we can convert our stream to a host of different collection types.

1
2
3
4


val stream: Stream[F, Int] = ???

stream.compile.toList // F[List[Int]]
stream.compile.toVector // F[Vector[Int]]

This property of Streams makes it easy to understand the API at first, but can make it difficult to understand how it works internally. A Stream differs from a List in that it incorporates an effect type (F[_]) as part of its definition (Stream[F, A]). So why do we need an effect type for our stream definition? A Stream does not necesserily hold all elements in memory at once. For example, if we stream an HTTP response, not all body bytes arrive from the network at the same time.

When using an effectful type (like IO), a Stream[IO, A] actually represents a single IO[Option[A]] that we call repeatedly for new results. We refer to this as a “pull-based” stream because we need to perform an action to retrieve the next elements of the stream. This works the same as Scala’s Iterator class when lazy generating results. As this diagram shows, we don’t materialise the elements of the Stream until we reach them.

Another important thing to note about Streams; they don’t (always) contain single elements. Instead, they contain multiple groups of elements (Chunks), which allows individual pulls to return more than one element. Useful when working with things like response payloads where consuming bodies byte by byte could harm performance. At one time a Stream only holds a single chunk. It pulls a new chunk as necessary when we consume the stream.

Pull API

Armed with this understanding of how a Stream works we can actually access the underlying representation type, Pull. Given an arbitrary stream we can call .pull to convert a Stream into a Pull. This type covers some internal implementation details but, in a nutshell, a Pull monad represents a single “step” of the underlying representation. For example, a single Pull might represent polling a connection for new response bytes.

Technically .pull returns a ToPull type. Again, this type covers some internal implementation details but acts as a springboard for calls like uncons or peek, which can optionally consume the current chunk. Operations on this type always yield a Pull so for our purposes I will focus on that.

A Pull type can cause confusion because it has three type parameters.

1
2
3
4
5
6
7


 Pull[F, O, R]
 ▲ ▲ ▲
 │ │ └─────── Return Type
 │ │
 │ └────────── Output Type
 │
 └───────────── Effect Type

The effect type simply represents the monad in use (e.g. IO). The output type represents the type of the parent Stream. The return type represents a “working” type to allow use to compose multiple Pulls into a single stream step. We can only collapse Pulls back to a normal Stream if they return a Unit type (signalling they are “done”), using the Pull.stream API. This distinction can make Pull a confusing type to work with at first.

1
2
3
4
5
6
7
8



val pull: Pull[IO, String, Int] =
 for {
 i <- Pull.pure(1)
 _ <- Pull.output1(i.toString)
 } yield i + 1

val stream: Stream[IO, String] = pull >> Pull.done

Pull.output1 emits the given value as an element in our stream. Pull.done just represents a Pull[F, Nothing, Unit] and acts as a convenient way of signalling the current pull has finished outputting any elements.

You can define custom steps on an existing Stream by using .repeatPull. This gives you a high degree of control over when and how we pull elements from the upstream. This function provides a base ToPull and asks you to return an Option[Stream[F, A]] to indicate if and how the stream should continue. An interesting property of this stage is that we don’t have to return elements from our upstream definition. We can insert new elements or switch to a different stream entirely.

Motivation and Summary

FS2’s Stream already provides a lot of powerful abstractions for working on a stream without dipping into the pull API, so why might you want to?

Performant consumption of chunks. Individually stream operations can provide either full chunks (via .chunks) or single elements. If you ever want to combine elements across chunks or dynamically merge chunks then you will need to pull and merge chunks yourself.
Single step logic. Sometimes you might want to partially consume a stream before making a decision on how to process the full stream. The pull API allows you to arbitrarily consume the Stream without committing to a single strategy.
Full control over execution. You might have a sensitive upstream API where you want to carefully choose when and how effects get executed.

Understanding how a Stream works underneath can help you intuit what a stream can (and can’t) do, as well as diagnosing any performance issues. The pull API unlocks performance improvements around chunk handling that the typical Stream interface might not provide. This might seem like a dense topic at first, but sometimes the best way to learn something means using it. Then you should find the concepts click into place (and if they don’t, I apologise unreservedly).

Enjoy The View

Tue, 20 Aug 2024 00:00:00 +0000

Aimed at developers unfamiliar with lazy Scala collections.

Most Scala code uses strict collections. That means that for any given collection we store every element in memory at the same time. This also means that when creating a list, we create every element ahead of time. This can impact the performance of our collections when we don’t need every element.

Examples include:

Collections with many elements. E.g. an infinite list of exponentially increasing numbers.
Collections with large elements. E.g. a list of HD images.
Expensive elements. E.g. a list of pending Futures making network calls.

In these cases some methods, such as find or head, don’t need every element of the collection. They would benefit from lazy data structures.

We can make some data structures lazy fairly easily. Scala’s defines a linked list (List type) an element and a pointer to the next element. Instead of pointing to the next element, we could suspend the construction and store a function instead. See the example below for how this works.

1
2


strictList = 1 -> 2 -> 3 -> 4 -> 5
lazyList = 1 -> (n => n + 1)

Lists in Scala are not lazy by default, and so if we want this behaviour we’ll actually need to use List’s sister class LazyList. This type works a lot like a list, but instead the list is stored as an element and a “thunk”, a function stored on the heap, which can be used to calculate the next element. Now when we use our find or head operations we only calculate as many elements as we need rather than creating elements unnecessarily. It also opens up some interesting algorithm designs that make use of infinite lists. For example, implementing a zipWithIndex function would look like this.

1
2


scala > LazyList.range(0, Int.MaxValue).zip("a cool example").toList
val res0: List[(Int, Char)] = List((0, a), (1, ), (2, c), (3, o), (4, o), (5, l), (6, ), (7, e), (8, x), (9, a), (10, m), (11, p), (12, l), (13, e))

Notice how the initial LazyList.range(0, Int.MaxValue) goes all the way to the maximum integer. If you replaced the LazyList with a regular strict List you’ll find the program takes a lot longer to complete (if it even does at all).

Lazy lists actually retain their elements after they’re calculated, so an infinitely sized LazyList can still risk overrunning your heap memory! If you want to discard elements, you’ll need to work recursively (with tail call optimization)

What if we’re not working with a lazy collection? We can actually work with any collection lazily by using a View. You can summon a view for a collection by using .view and continue to use the familiar collection traversal methods. The upside is that every operation we do will be evaluated lazily just like the lazy list!

This example shows how the order of operations differs between a strict list and a lazy view on the same collection.

1
2
3
4
5
6
7
8
9


val normalList = List.range(0, 2)
val viewedList = normalList.view

val tappedList = normalList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))
val tappedView = viewedList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))

println("Not done yet?")
// realise the view
tappedView.toList

Run this snippet, and you’ll see something interesting.

1
2
3
4
5
6
7
8
9


Operation A
Operation A
Operation B
Operation B
Not done yet?
Operation A
Operation B
Operation A
Operation B

The strict list evaluates the entire list first for the first tapEach, and then evaluates it again for the second tapEach (meaning we’ve actually crossed the whole list twice). In the view, however, the operations are actually combined, making our update more efficient. Similarly, we actually don’t see the output of the view until we force it back into a list later on (sometimes referred to as “realising” the list). A view isn’t always the right choice for the job. Views are great for collection traversal but don’t support functions that might access the collection in a random order (for example, sorting). We should avoid storing unevaluated thunks in situations featuring many, small chunks of code. In those cases it costs less to evaluate them upfront than to keep them and negatively impact GC pressure. It’s also worth noting, the View design also means that evaluated View chunks remain in memory while the view itself is still referenced.

Turning Docker Compositions into Test Suites

Wed, 06 Sep 2023 00:00:00 +0000

Use Docker compositions in tests with Test-Containers and ScalaTest.

Sample code repository.

We use docker-compose as a tool that can build, deploy, and manage multiple containers on the same network. These configurations, known as compositions, remove a lot of the manual setup we often associate with running infrastructure locally. It provides local versions of dependencies for a faster, more realistic feedback loop when developing applications.

Our application level tests could look as simple as this;

Test Containers

For our tests we will use test-containers. It offers us;

Access to a Docker context from within tests.
Integration with test harnesses provided by popular testing frameworks.
The automatic creation and clean-up of Docker containers.

Test-Containers has a Docker-Compose plugin that allows us to run compositions from our tests. First, let’s define a simple composition file.

I’ve kept it simple by only including our application, but from this point on I can extend it with whatever setup we need without having to change our tests at all. To start using this in our tests, we’ll need to do three things.

Build and publish our application image locally.
Integrate our composition setup into our test-suite.
Create a client that can call our application.

1. Build and publish our application image locally.

Before we can use it in a test suite, we need to build our application into a Docker image. Unfortunately Docker-Compose does not have a concept of running external build tasks to get an image. That said, most build tools will allow us to build an image before running our test suite. With SBT and native-packager plugin makes it easy to add our Docker build stage as a prerequisite for our integration tests.

An SBT quirk means we need to specify our Docker build step for every test task.

1
2
3


 Test / test := (Test / test).dependsOn(server / Docker / publishLocal).value,
 Test / testOnly := (Test / testOnly).dependsOn(server / Docker / publishLocal).evaluated,
 Test / testQuick := (Test / testQuick).dependsOn(server / Docker / publishLocal).evaluated

2. Integrate our composition setup into our test-suite.

Next, we add the image details to our tests.

The Test-Containers library has integrations for many test frameworks, which makes this easier, but we’ll still need to do some wiring. The Docker-Compose module needs to know the location of our compose-file. You might want to hard-code this, but that affects the portability of your tests. Instead, I recommend passing the location in as a property and using SBT to locate the file. This has the benefits for us by enabling testing against multiple compose-files; useful for testing against different environments or setups.

If you use an IDE to run test-suites, you may need to update the runner config before this will work. In IntelliJ, you can save a run configuration and share it by committing it to the repo.

We need to wait for our containers to start before using them. We can get around this by making our access details lazy or use the Test-Containers helpers (which run after the containers finish starting).

Lastly, we need to instruct Test-Containers on how to use our containers.

Some additional considerations you might make;

We added a wait condition for the service’s health check so that Test-Containers knows to start testing. For more complex start conditions check out the test-containers documentation.
We only listen to logs of the main container as logging everything at once creates too much noise. To help with that, we can either:
- Set noisy application’s log-levels through the Docker environment.
- Use Test-Containers logging classes to build something custom.

3. Create a client to call our application.

Finally, let’s create our HTTP client to call our service. We have full access to the Docker containers and, if needed, we can connect to our dependencies directly. Alternatively, if you provide clients for your service, you can use them here instead.

Summary

With our test fixture code in place, we can now start writing code without too much concern about the state of our containers.

Now if we hit a test failure, we can launch our docker-composition to debug our test case. This helps us switch to an iterative, live feedback loop for manual testing and experimentation.

Custom Request DSLs with Http4s

Thu, 30 Mar 2023 00:00:00 +0000

Recently I implemented some custom routing mechanics for an Http4s app. Sometimes we want to route requests based on more than just HTTP details. Http4s’ AuthedRoutes serves as a great example for this.

1
2
3
4
5
6
7


val normal: HttpRoutes[IO] = HttpRoutes.of {
 case GET -> / "user" / UserId (user) / "resource" => ???
}

val authed: AuthedRoutes[UserId, IO] = AuthedRoutes.of {
 case GET -> / "user" / "resource" as user => ???
}

The second route shows an example of using some new DSL syntax. That small as user at the end of the second route definition does a lot of heavy lifting around user authentication. Even better, as it modifies the route type itself, we can’t forget to add authentication to a route (which we might if added auth route by route).

For my change I will focus on a relatively neutral change that works with the normal Http4s request and response types. Let’s work with a simple requirement.

Expose routes to a specific user-agent only.

How might that look? Let’s sketch out a possible DSL. We have something like case [service] using [route] => where;

service — identifies the requesting client.
using — our custom DSL, which activates our routing logic.
route — the standard Http4s route definition.

In code this ends up looking like this:

1
2
3
4


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
 case Service.Aggregator using GET -> Root / "resource" / IntVar(id) =>
}

To understand how we might add this, we need to explore how Http4s cleverly uses Scala’s pattern match system. When we pattern match on a case class, Scala invokes a method called unapply to decide if the given value matches. Typically, that method might look something like this.

1
2
3


object Foo {
 def unapply(foo: Foo): Option[Int]
}

In this case the implementation knows how to decompose a Foo into a Int, by using the class’ foo field. Sometimes we can’t access a field like that (for example, an ADT), so the optional return type allows us to indicate the match failed. We can define this method explicitly if we want to, as Scala allows us to define our own custom unapply matchers. Scala refers to these as extractor objects.

Http4s itself has some great examples of when we might want to use extractors. The status matcher Succesful matches response statuses within the 2xx range, with similar matchers for 4xx and 5xx too. This showcases the first style of matcher — a way of grouping similar terms in a match.

You can nest Scala’s matchers in the same way that you can nest data. For example;

1
2
3
4
5


 case Right(Some(3)) =>
// ^ ^
// | |_ Then it applies the `Option` matcher to the result.
// |
// |_ First applies the `Either` matcher

This nesting also powers the familiar list matcher.

1
2
3
4


{
 case 1 :: 2 :: Nil => // using inline syntax
 case ::(1, ::(2, ::(3, ::(Nil))) => // using actual extractor classes
}

This syntax eliminates the normal bracket syntax we see in favour of something more readable. Seeing this, you might see how the Http4s DSL takes shape. Using DSL objects (made available via the Htp4sDSL) trait we can string a series of matchers along to define our expected request structure. Checking the source code gives us an idea of how to structure our own custom matcher.

The first component of a Http4s route:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


object -> {

 /** HttpMethod extractor:
 * {{{
 * (request.method, Path(request.path)) match {
 * case Method.GET -> Root / "test.json" => ...
 * }}}
 */
 def unapply[F[_]](req: Request[F]): Some[(Method, Path)] =

...
}

It takes as input the inbound request itself and outputs two chunks split from the request; the method and the path. Through the same syntax as :: that we saw earlier; Scala can match the request as a method on the left and a path on the right. Scala also allows us to match against specific values, so we can further refine our match with an exact method. The path however gets fed to another Path matcher, which decomposes the value further.

1
2
3
4
5


object / {
 def unapply(path: Path): Option[(Path, String)] =

...
}

This path matcher consumes a Path and splits to a String component on its right-hand side. The left-hand side returns another Path so we can decompose the path string as much as we need to.

You should see a pattern emerging, and we can start implementing our own. Let’s revisit our desired syntax.

1
2
3


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
}

We can start building our matcher by looking at our preferred inputs and outputs. In this case we will need:

Input: Request[F]
Output: (Service, Request[F])

Our matcher takes an HTTP request and returns it later. This means matchers can continue matching with it. Along with that we also return our service identifier. Our implementation looks something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


object Service {
 case object Distributor

 object using {
 def unapply[F[_]](req: Request[F]): Option[(Service, Request[F])] =
 req.headers.get[`User-Agent`].collect {
 case ua if ua.value == "distributor" => (Distributor, req)
 }
 }
}

Notice the unapply method actually has a generic type parameter. The compiler doesn’t allow type-annotated pattern matchers (you can’t do case Foo[Int](ambiguousNumber) => ???). This means the type parameter needs to resolve unambiguously in the match context.

We returned a simple value, but we could do something more interesting. For example, we could;

create a bespoke DB connection using a user’s details.
configure request caching.
return a request with additional tracking information.

We can also get creative with our DSL. We could change the order of the outputs and move our syntax to the match end instead, or return multiple matches instead. Experimenting with the values you extract in your match brings up some interesting possibilities. Take Http4s’ ->> matcher as an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


 /** Extractor to match an http resource and then enumerate all supported methods:
 * {{{
 * (request.method, Path(request.path)) match {
 * case withMethod ->> Root / "test.json" => withMethod {
 * case Method.GET => ...
 * case Method.POST => ...
 * }}}
 *
 * Returns an error response if the method is not matched, in accordance with [[https://datatracker.ietf.org/doc/html/rfc7231#section-4.1 RFC7231]]
 */
def unapply[F[_] : Applicative](
 req: Request[F]
 ): Some[(PartialFunction[Method, F[Response[F]]] => F[Response[F]], Path)] =
...

This extractor yields control back to the caller. That allows the routing to share some common logic while also delegating part of that match back to the user.

This helps create powerful DSLs. They make defining request-dependent service behaviour a breeze. Generally, you should avoid custom syntax if you can help it. This technique can help simplify a lot of code, but it can make it worse too.

Common drawbacks and criticisms of custom DSLs include;

Confusing or pointlessly obtuse syntax (for example, |@| from cats).
They can hide too much. Subtle behaviours introduced through syntax can feel “magic”.
Newcomers need to learn your syntax on top of a language they might not know already.

You and your team decide how far you want to take it. A part of the art of software development includes deciding when to use these kinds of techniques.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ― Antoine de Saint-Exupéry.

Even small uses can benefit from custom matchers, for example, this…

1
2
3
4
5
6
7


{
 // domain logic leak
 case Left(Some(event)) if event.timestamp > cutOffTime =>

 // better, self-documenting
 case LiveEvent(event) =>
}

That reduces cramped matchers into something simple and self-documenting.