meteor

Get Started

meteor is a wrapper around Java AWS SDK v2 library, provides higher level API over standard DynamoDB’s actions.

Features

  • wraps AWS Java SDK V2 CRUD actions
  • stream batch actions or scan table as fs2 Stream
  • auto processes left over items
  • auto remove duplication in batch actions
  • support DynamoDB’s single table design
  • provides codec as a simple abstraction and syntax on top of AttributeValue
  • support multiple codec libraries including Dynosaur and Scanamo

Installation

Add the following to your build.sbt, see the badge on homepage for latest version. Supports Scala 2.12 and 2.13.

libraryDependencies += "io.github.d2a4u" %% "meteor-awssdk" % "LATEST_VERSION"

Modules

Scanamo Format

Note: Only version 1.0.0-M11 is supported because in my experience, this is the most stable version of Scanamo. However, because it is an older version when DynamoDB’s did not support empty String, this version of Scanamo serializes these cases: "", None and Some("") to Dynamo’s NULL. This is problematic because once the value is written down, reading it back is difficult.

libraryDependencies += "io.github.d2a4u" %% "meteor-scanamo" % "LATEST_VERSION"

Dynosaur Codecs 

libraryDependencies += "io.github.d2a4u" %% "meteor-dynosaur" % "LATEST_VERSION"

Optimisation

meteor uses AWS SDK v2 DynamoDbAsyncClient under the hood to make calls to the underline Java API. The DynamoDbAsyncClient internally create a NettyNioAsyncHttpClient with a default value for maximum connections of 50. This can be increased depending on the hardware/virtual machine configuration. There is also an alternative AWS CRT HTTP client that can be used and tuned similarly. The official documentation can be found here.

import cats.effect._
import meteor.Client
import io.netty.channel.ChannelOption
import software.amazon.awssdk.http.SdkHttpClient
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient
import software.amazon.awssdk.services.dynamodb.DynamoDbClient

val httpClientSrc = Resource.fromAutoCloseable(
  IO {
    NettyNioAsyncHttpClient
      .builder()
      .putChannelOption(ChannelOption.SO_KEEPALIVE, true)
      .maxConcurrency(100)
      .maxPendingConnectionAcquires(10000).build()
  }
)

def sdkDynamoClientSrc(http: SdkHttpClient) =
  Resource.fromAutoCloseable(
    IO {
      DynamoDbClient.builder().httpClient(http).build()
    }
  )

val clientSrc =
  for {
    http <- httpClientSrc
    sdkClient <-sdkDynamoClientSrc(http)
  } yield Client[IO](sdkClient)
Hello World →