Integration testing for Couchbase scopes and collections with Testcontainers in Spring Boot

The 7.0 release of the Couchbase introduces a new data organisation feature called Scopes and Collections are logical containers within a Couchbase bucket.

It’s helpful to think of Collections as tables within a relational database and Scopes are a set of related Collections, making them similar to an RDBMS schema.

When I started to write this post the latest version of Couchbase was 7.3, and we wanted to start a new project where we will have a few microservices.

We decided to use Kotlin as a programming language, Spring Boot for our web framework and the Couchbase as a data store for some of our services.

We know the importance of automation tests in our software delivery process and wanted to test the integration of our code to Couchbase DB too.

I believe most of you already know for doing integration tests there is a great library called Testcontainers. In a nutshell; it provides lightweight, throwaway instances of common databases or anything else that can run in a Docker container.

But as of today, the Testcontainers doesn’t have any support for the newest version of the Couchbase which has scopes and collections. Luckily it’s not a daunting task to accomplish and in this post, I’d like to show you how.

The tech stack

• Junit 5 Jupiter for test runner library
• Spring Boot (version >= 2.6.3) for web application framework
• Testcontainers modules: Testcontainers Junit Jupiter, Testcontainers Couchbase

First, we need to add the following dependencies to our pom.xml/build.gradle file:

testImplementation("org.testcontainers:couchbase:1.16.3")
testImplementation("org.testcontainers:testcontainers:1.16.3")
testImplementation("org.testcontainers:junit-jupiter:1.16.3")

Now we can write the tests and our Couchbase integration test flow will be like this:

1. First we need to create a Couchbase container from the Testcontainers library, keep in mind when we create this container we don’t have a way to create our scopes and collections yet :)
2. Since it is going to be an integration test we will tell the Spring Boot to create an application context for our test and when we do that we need to give the Couchbase DB properties created by the Testcontainers to Spring Boot, this way our application will connect to the Testcontainers Couchbase DB.
3. And then we will use the @BeforeAll annotation to annotate a setup function for our test class and in this function, we will configure our Couchbase DB with collections and scopes using Rest calls.
4. Bonus, doing a custom readiness check before the tests start.

1. Creating a Couchbase container from the Testcontainers library:

companion object {
    private val COUCHBASE_IMAGE_NAME = "couchbase"
    private val DEFAULT_IMAGE_NAME = "couchbase/server"
    private val BUCKET_NAME = "feed"
    private val DEFAULT_IMAGE =
        DockerImageName.parse(COUCHBASE_IMAGE_NAME).asCompatibleSubstituteFor(DEFAULT_IMAGE_NAME)

    @Container
    val couchbaseContainer: CouchbaseContainer =
        CouchbaseContainer(DEFAULT_IMAGE)
            .withCredentials("Administrator", "password")
            .withBucket(BucketDefinition(BUCKET_NAME).withPrimaryIndex(false))
            .withStartupTimeout(Duration.ofSeconds(60))

Here we’re creating a container with credentials and a bucket from Testcontainers.

2. Creating an application context and binding the DB properties:

@SpringBootTest
@Testcontainers
internal abstract class BaseCouchbaseTest {
    companion object {
        private val COUCHBASE_IMAGE_NAME = "couchbase"
        private val DEFAULT_IMAGE_NAME = "couchbase/server"
        private val BUCKET_NAME = "feed"
        private val SCOPE_NAME = "post"
        private val DEFAULT_IMAGE =
            DockerImageName.parse(COUCHBASE_IMAGE_NAME).asCompatibleSubstituteFor(DEFAULT_IMAGE_NAME)

        @Container
        val couchbaseContainer: CouchbaseContainer =
            CouchbaseContainer(DEFAULT_IMAGE)
                .withCredentials("Administrator", "password")
                .withBucket(BucketDefinition(BUCKET_NAME).withPrimaryIndex(false))
                .withStartupTimeout(Duration.ofSeconds(60))

        @JvmStatic
        @DynamicPropertySource
        fun bindCouchbaseProperties(registry: DynamicPropertyRegistry) {
            registry.add("spring.couchbase.connection-string") { couchbaseContainer.connectionString }
            registry.add("spring.couchbase.username") { couchbaseContainer.username }
            registry.add("spring.couchbase.password") { couchbaseContainer.password }
            registry.add("spring.data.couchbase.bucket-name") { BUCKET_NAME }
            registry.add("spring.data.couchbase.scope-name") { SCOPE_NAME }
        }

Here with the @SpringBootTest annotation we are telling that we need a test application context and the framework will provide that for us. When the application is starting up the framework will call the static method annotated with @DynamicPropertySource and in this function, we are mapping Testcontainers Couchbase container properties to the application properties. By doing that, the Spring Boot framework will know how to connect to test DB in our tests.

And also it’s worth mentioning, the @Testcontainers is a JUnit Jupiter extension to activate automatic startup and stop of containers used in a test case. The test containers extension finds all fields that are annotated with Container and calls their container lifecycle methods, like starting a container before tests or stopping it after tests.

3. Configuration of the Couchbase DB with scopes and collections:

@JvmStatic
@BeforeAll
fun setup() {
    postToCouchBase("name=post", "/pools/default/buckets/feed/scopes")
    postToCouchBase("name=posts", "/pools/default/buckets/feed/scopes/post/collections")
    postToCouchBase("name=counters", "/pools/default/buckets/feed/scopes/post/collections")
}

private fun postToCouchBase(body: String, resourcePath: String): ResponseEntity<String> {
    val httpEntity = HttpEntity(
        body,
        HttpHeaders().also {
            it.contentType = MediaType.APPLICATION_FORM_URLENCODED
            it.accept = listOf(MediaType.APPLICATION_JSON)
        }
    )

    return TestRestTemplate()
        .withBasicAuth(couchbaseContainer.username, couchbaseContainer.password)
        .postForEntity(
            "http://${couchbaseContainer.host}:${couchbaseContainer.bootstrapHttpDirectPort}$resourcePath",
            httpEntity,
            String::class.java
        )
}

As you can see above, we’re using the Couchbase REST API to configure the DB, even though the Testcontainers' current version doesn’t support the scopes and collections, we’re using the Couchbase REST API to create our microservice scope and collections. And now the DB is ready and we can start writing our integration test against the DB. So we’ve created a scope called post and two collections called posts and counters.

4. Bonus, doing a custom readiness check before the tests start:

In my case; I’m using an autoId to generate an id for my entities. E.g.

@Document
@Scope("post")
@Collection("posts")
data class PostEntity(
    @IdAttribute(order = 1)
    val autoId: Long,
    @IdAttribute(order = 0)
    val producer: String,
    @Field
    val collectionId: String,
) {
    @Id
    @GeneratedValue(delimiter = "::")
    var id: String? = null
}

And with Spring Data Couchbase it is very easy to query on this entity with Repository pattern:

interface PostEntityRepository : CouchbaseRepository<PostEntity, String>

And for generating the autoId, like sequences in the RDMS world, I created another collection and a custom repository. Since it’s just a counter and I don’t need an entity to map it, I used my existing entity repository to access to DB operations API that was already provided by Spring Data CouchbaseRepository:

@Repository
class PostCounterRepository(val postEntityRepository: PostEntityRepository) {
    private val COUNTER_COLLECTION = "counters"
    private val COUNTER_KEY = "post"

    fun nextValue(): Long =
        postEntityRepository.operations.couchbaseClientFactory.getCollection(COUNTER_COLLECTION).binary().increment(COUNTER_KEY).content()
}

And in my service:

@Service
class PostService(
    val postEntityRepository: PostEntityRepository,
    val postCounterRepository: PostCounterRepository
) {

    fun addPost(collectionCreated: CollectionCreated) =
        postEntityRepository.save(
            collectionCreated.toPostEntity(postCounterRepository.nextValue())
        ).toPostDTO()

As you can see I need a bucket, a scope, 2 collections and in one of my collections (counters) there should be a document ready when I start my application. In the integration test, we need to make sure these are ready. And so far we created our bucket, scope and collections now we need to insert a document into a collection and wait until it’s succeeded then we can start our integration tests. As you imagined already it’s not going to be different, we’re going to use the Couchbase REST API again to insert a document and implement a retry mechanism with a simple backoff. So if we go to our BaseCouchbaseTest class again, we need to modify our setup function:

@JvmStatic
@BeforeAll
fun setup() {
    runBlocking {
        launch { waitUntilDatastoreReadyOrTimedOut() }
        postToCouchBase("name=post", "/pools/default/buckets/feed/scopes")
        postToCouchBase("name=posts", "/pools/default/buckets/feed/scopes/post/collections")
        postToCouchBase("name=counters", "/pools/default/buckets/feed/scopes/post/collections")
        println("initialising the readiness check ......")
    }
    println("The readiness check of datasource is finished")
}

private suspend fun waitUntilDatastoreReadyOrTimedOut() {
    var tryCount = 0
    do {
        tryCount++
        println("The readiness check - retrying with $tryCount second(s) delay")
        val isPostCounterReady = try {
            delay(tryCount * 1_000L)
            callAndCheckIfPostCounterReady()
        } catch (e: Exception) {
            false
        }
    } while (!isPostCounterReady && tryCount <= 15)
}

private fun callAndCheckIfPostCounterReady() =
    postToCouchBase(
        "statement=INSERT INTO feed.post.posts (KEY,VALUE) VALUES (\"post\",0)",
        "/_p/query/query/service"
    )
        .let {
            it.statusCode == HttpStatus.OK
        }

Here we added a new function call in the setup function it will do a post call to insert our counter document to DB, but it’ll do that with at most 15 retries with a simple backoff. So each retry we will increase the delay to give some time to the DB to be able to handle the initialisations.

With all these changes here is the latest BaseCouchbaseTest class:

package com.trendyol.feedpost.domain.repository

import kotlinx.coroutines.delay
import kotlinx.coroutines.launch
import kotlinx.coroutines.runBlocking
import org.junit.jupiter.api.BeforeAll
import org.springframework.boot.test.context.SpringBootTest
import org.springframework.boot.test.web.client.TestRestTemplate
import org.springframework.http.HttpEntity
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpStatus
import org.springframework.http.MediaType
import org.springframework.http.ResponseEntity
import org.springframework.test.context.DynamicPropertyRegistry
import org.springframework.test.context.DynamicPropertySource
import org.testcontainers.couchbase.BucketDefinition
import org.testcontainers.couchbase.CouchbaseContainer
import org.testcontainers.junit.jupiter.Container
import org.testcontainers.junit.jupiter.Testcontainers
import org.testcontainers.utility.DockerImageName
import java.time.Duration

@SpringBootTest
@Testcontainers
internal abstract class BaseCouchbaseTest {
    companion object {
        private val COUCHBASE_IMAGE_NAME = "couchbase"
        private val DEFAULT_IMAGE_NAME = "couchbase/server"
        private val BUCKET_NAME = "feed"
        private val SCOPE_NAME = "post"
        private val DEFAULT_IMAGE =
            DockerImageName.parse(COUCHBASE_IMAGE_NAME).asCompatibleSubstituteFor(DEFAULT_IMAGE_NAME)

        @Container
        val couchbaseContainer: CouchbaseContainer =
            CouchbaseContainer(DEFAULT_IMAGE)
                .withCredentials("Administrator", "password")
                .withBucket(BucketDefinition(BUCKET_NAME).withPrimaryIndex(false))
                .withStartupTimeout(Duration.ofSeconds(60))

        @JvmStatic
        @DynamicPropertySource
        fun bindCouchbaseProperties(registry: DynamicPropertyRegistry) {
            registry.add("spring.couchbase.connection-string") { couchbaseContainer.connectionString }
            registry.add("spring.couchbase.username") { couchbaseContainer.username }
            registry.add("spring.couchbase.password") { couchbaseContainer.password }
            registry.add("spring.data.couchbase.bucket-name") { BUCKET_NAME }
            registry.add("spring.data.couchbase.scope-name") { SCOPE_NAME }
            registry.add("kafka.enable") { false }
        }

        @JvmStatic
        @BeforeAll
        fun setup() {
            runBlocking {
                launch { waitUntilDatastoreReadyOrTimedOut() }
                postToCouchBase("name=post", "/pools/default/buckets/feed/scopes")
                postToCouchBase("name=posts", "/pools/default/buckets/feed/scopes/post/collections")
                postToCouchBase("name=counters", "/pools/default/buckets/feed/scopes/post/collections")
                println("initialising the readiness check ......")
            }
            println("The readiness check of datasource is finished")
        }

        private fun postToCouchBase(body: String, resourcePath: String): ResponseEntity<String> {
            val httpEntity = HttpEntity(
                body,
                HttpHeaders().also {
                    it.contentType = MediaType.APPLICATION_FORM_URLENCODED
                    it.accept = listOf(MediaType.APPLICATION_JSON)
                }
            )

            return TestRestTemplate()
                .withBasicAuth(couchbaseContainer.username, couchbaseContainer.password)
                .postForEntity(
                    "http://${couchbaseContainer.host}:${couchbaseContainer.bootstrapHttpDirectPort}$resourcePath",
                    httpEntity,
                    String::class.java
                )
        }

        private suspend fun waitUntilDatastoreReadyOrTimedOut() {
            var tryCount = 0
            do {
                tryCount++
                println("The readiness check - retrying with $tryCount second(s) delay")
                val isPostCounterReady = try {
                    delay(tryCount * 1_000L)
                    callAndCheckIfPostCounterReady()
                } catch (e: Exception) {
                    false
                }
            } while (!isPostCounterReady && tryCount <= 15)
        }

        private fun callAndCheckIfPostCounterReady() =
            postToCouchBase(
                "statement=INSERT INTO feed.post.counters (KEY,VALUE) VALUES (\"post\",0)",
                "/_p/query/query/service"
            )
                .let {
                    it.statusCode == HttpStatus.OK
                }
    }
}

And finally the integration tests class:

package com.trendyol.feedpost.domain.repository

import com.trendyol.feedpost.models.PostFixtures.randomPostEntity
import org.apache.commons.lang3.RandomStringUtils.randomAlphabetic
import org.assertj.core.api.Assertions.assertThat
import org.junit.jupiter.api.Test
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.data.repository.findByIdOrNull

internal class PostEntityRepositoryIT : BaseCouchbaseTest() {

    @Autowired
    private lateinit var postEntityRepository: PostEntityRepository

    @Autowired
    private lateinit var postCounterRepository: PostCounterRepository

    @Test
    fun `call to nextValue should return the next value of the counter`() {
        val nextValue = postCounterRepository.nextValue()
        assertThat(nextValue).isNotNull
        assertThat(nextValue).isGreaterThanOrEqualTo(1)
    }

    @Test
    fun `save should create a document of the given post on the datastore`() {
        val autoId = postCounterRepository.nextValue()
        val producer = randomAlphabetic(5)
        val postEntity = randomPostEntity(autoId, producer)
        val savedEntity = postEntityRepository.save(postEntity)

        assertThat(savedEntity.id).isEqualTo("$producer::$autoId")
        assertThat(postEntityRepository.findByIdOrNull(savedEntity.id!!))
            .extracting("autoId", "producer", "collectionId")
            .contains(postEntity.autoId, postEntity.producer, postEntity.collectionId)
    }
}