Support android Pagination for paging3

[Robohat]

Is there an existing issue?

• I have searched existing issues

Use case

Using ObjectBox as the database for Android Apps i would like to use the pagination library paging3 from android.
See https://developer.android.com/topic/libraries/architecture/paging/v3-overview

Since ObjectBox 2.0.0 there is an implementation of LiveData<PagedList< DATAENTITYMODEL >>,
which was a very good integration with androids paging2 library back then.

Can you create a similar solution for use with the Android paging3 library?
https://developer.android.com/topic/libraries/architecture/paging/v3-overview#kts

Proposed solution

I would love to see an implementation of
Flow<PagingData< DATAENTITYMODEL >>
such that i can do, what is described here
https://developer.android.com/topic/libraries/architecture/paging/v3-paged-data#display-paged-data

Alternatives

Alternative: An implementation of PagingSource could also help to use the android paging3 library
https://developer.android.com/reference/kotlin/androidx/paging/PagingSource

[greenrobot-team]

Thanks for your suggestion! As things keep changing (think Jetpack Compose) and this is often very use case specific, I don't think we will provide this.

Maybe have a look at the source of the v2 paging source for inspiration to write a v3 one.

[ajans]

Maybe the time is ripe to implement this on the objectbox-side of things?

From my point of view for the following reasons:

1. The workaround to convert the ObjectBoxDataSource to a paging3-compatible PagingSource by using the paging3-extension asPagingSourceFactory() on the instance returned by ObjectBoxDataSource.Factory(query) works in general, but the initial loading (with a RemoteMediator mediating between a Retrofit-instance and ObjectBox) always loads three pages instead of one, producing significant unnecessary overhead. I think this bug is somewhere between PositionalDataSource and asPagingSourceFactory, so I doubt this will be fixed by Google.
2. The paging3-library from Google is still relevant and a self-made paging-implementation is non-trivial, especially with an already existing implementation based on paging3, with support for both RecyclerView- and LazyColumn-usage (I am still in the process of migration to Jetpack Compose, so this would help both).
3. Jetpack Compose support for paging3 is stable since paging3 version 3.2.0, see Google documentation on collectAsLazyPagingItems.
4. collectAsLazyPagingItems works on a Flow<PagingData< DATAENTITYMODEL >> so it is exactly what is needed to support Jetpack Compose properly.
5. Everybody would benefit from a centrally maintained version of the current paging-standard-library on android, Google's paging3-library.
6. It would make sense to deprecate ObjectBoxDataSource since paging2 is deprecated. And since it currently resides wrongly in the objectbox-dependency objectbox-android-objectbrowser, it would be practical to move it to its own dependency objectbox-android-paging3, anyway.
7. When Google removes support for paging2-DataSource in paging3, ObjectBox would not have any working support for Googles paging-libraries anymore.

[greenrobot-team]

@ajans Thanks for your interest and details! To be honest, I don't think we'll have the time to add support for this (or any other future adapters for UI frameworks). But we would be open to refer to a third party project that makes this available.

[lucf15]

@ajans here is an implementation based on LimitOffsetPagingSource by androidx which uses ObjectBox:

import androidx.paging.PagingSource
import androidx.paging.PagingState
import io.objectbox.BoxStore
import io.objectbox.kotlin.awaitCallInTx
import io.objectbox.query.Query
import io.objectbox.reactive.DataObserver
import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicLong

class LimitOffsetPagingSource<Value : Any>(
    private val boxStore: BoxStore,
    private val query: Query<Value>,
) : PagingSource<Long, Value>() {

    companion object {
        private const val INITIAL_ITEM_COUNT = -1L
        val INVALID = LoadResult.Invalid<Any, Any>()
    }

    private val observer = ThreadSafeInvalidationObserver<Value>(onInvalidated = ::invalidate)

    private val itemCount: AtomicLong = AtomicLong(INITIAL_ITEM_COUNT)

    override suspend fun load(params: LoadParams<Long>): LoadResult<Long, Value> {
        observer.registerIfNecessary(query)
        val tempCount = itemCount.get()
        return try {
            if (tempCount == INITIAL_ITEM_COUNT) {
                initialLoad(params)
            } else {
                nonInitialLoad(params, tempCount)
            }
        } catch (e: Exception) {
            LoadResult.Error(e)
        }
    }

    private suspend fun initialLoad(params: LoadParams<Long>): LoadResult<Long, Value> {
        return boxStore.awaitCallInTx {
            val tempCount = query.count()
            itemCount.set(tempCount)
            queryDatabase(
                params = params,
                query = query,
                itemCount = tempCount
            )
        }!!
    }

    private fun nonInitialLoad(
        params: LoadParams<Long>,
        tempCount: Long,
    ): LoadResult<Long, Value> {
        val loadResult = queryDatabase(
            params = params,
            query = query,
            itemCount = tempCount
        )
        @Suppress("UNCHECKED_CAST")
        return if (invalid) INVALID as LoadResult.Invalid<Long, Value> else loadResult
    }

    override fun getRefreshKey(state: PagingState<Long, Value>): Long? {
        return state.getClippedRefreshKey()
    }

    override val jumpingSupported: Boolean
        get() = true

    private fun <Value : Any> queryDatabase(
        params: LoadParams<Long>,
        query: Query<Value>,
        itemCount: Long,
    ): LoadResult<Long, Value> {
        val key = params.key ?: 0
        val limit: Long = getLimit(params, key)
        val offset: Long = getOffset(params, key, itemCount)
        val data: List<Value> = query.find(offset * 1L, limit * 1L)
        val nextPosToLoad = offset + data.size
        val nextKey =
            if (data.isEmpty() || data.size < limit || nextPosToLoad >= itemCount) {
                null
            } else {
                nextPosToLoad
            }
        val prevKey = if (offset <= 0 || data.isEmpty()) null else offset
        return LoadResult.Page(
            data = data,
            prevKey = prevKey,
            nextKey = nextKey,
            itemsBefore = offset.toInt(),
            itemsAfter = maxOf(0, itemCount - nextPosToLoad).toInt()
        )
    }

    private fun getLimit(params: LoadParams<Long>, key: Long): Long {
        val loadSize = params.loadSize * 1L
        return when (params) {
            is LoadParams.Prepend ->
                if (key < loadSize) {
                    key
                } else {
                    loadSize
                }

            else -> loadSize
        }
    }

    private fun getOffset(params: LoadParams<Long>, key: Long, itemCount: Long): Long {
        return when (params) {
            is LoadParams.Prepend ->
                if (key < params.loadSize) {
                    0
                } else {
                    key - params.loadSize
                }

            is LoadParams.Append -> key
            is LoadParams.Refresh ->
                if (key >= itemCount) {
                    maxOf(0, itemCount - params.loadSize)
                } else {
                    key
                }
        }
    }

    private fun <Value : Any> PagingState<Long, Value>.getClippedRefreshKey(): Long? {
        return when (val anchorPosition = anchorPosition) {
            null -> null
            else -> maxOf(0L, anchorPosition - (config.initialLoadSize / 2L))
        }
    }

    private class ThreadSafeInvalidationObserver<T : Any>(val onInvalidated: () -> Unit) : DataObserver<MutableList<T>> {
        private val registered: AtomicBoolean = AtomicBoolean(false)

        fun registerIfNecessary(query: Query<T>) {
            if (registered.compareAndSet(false, true)) {
                query.subscribe().onlyChanges().weak().observer(this)
            }
        }

        override fun onData(data: MutableList<T>) {
            onInvalidated()
        }
    }
}