IssueTracker

Apologies Jeremy, I realized we have a tracking ticket for the sheet APIs specifically: b/261409034 Feel free to keep this issue or mark as duplicate of the other one.

Sounds good. I will make it a dup.

One guess I have at your issue, is that a remote APPEND is getting queued behind remote REFRESH because your first generation is empty, and second APPEND is triggered since first APPEND doesn't finish invalidating next generation before it loads to the end.

BTW remote calls are triggered pro-actively as soon as PagingSource loads to the end of data, you don't necessarily need to scroll down within prefetchDistance to trigger remote load, just need to trigger loading the last item in DB, so it's possible for it to be off by at most 1 page.

One guess I have at your issue, is that a remote APPEND is getting queued behind remote REFRESH because your first generation is empty, and second APPEND is triggered since first APPEND doesn't finish invalidating next generation before it loads to the end.

How can I check if this is happening? Also, what exactly do you mean with "first generation"? The PagingSource comes from Room, and there is data in the table, so how would the first generation be empty?

BTW remote calls are triggered pro-actively as soon as PagingSource loads to the end of data, you don't necessarily need to scroll down within prefetchDistance to trigger remote load, just need to trigger loading the last item in DB, so it's possible for it to be off by at most 1 page.

What is "end of data" in the context of a Room PagingSource? The last item of the current page? Or the last item in the complete table? Why would that occur without scrolling if I have a pageSize of 4, and an initialLoadSize of 12. After a REFRESH the database would have 12 items, the RecyclerView would show 1-2 items, so why would the last item in the database be accessed, since that would be 2-3 pages further.

I will checkout the codelab code and see if I have the same problem there. Though on first glance I don't see any major differences between my code and the codelab.

Ok I ran the paging codelab, and I seem to have the same issues there (but my theories on why it happens appear to be wrong, it doesn't seem to be caused by the initialLoadSize)

In the codelab it simply calls APPEND 3 times immediately after refresh. Someone made an issue about this on Github

I slightly edited the codelab to bring it in line with my own code. I changed how the remoteKeys are stored in the db based on this Stackoverflow comment. (I thought I read this was updated in the codelab, but apparently not)

Also checked if it happens with just a network PagingSource, but then it works correctly, calling just 1 page.

I attached my edited version of the codelab.

Hi,

Any updates or insights on this?

First off, sorry it took awhile to get back to this. Let me answer your questions first and then I actually checked out your sample and have the solution below:

How can I check if this is happening? Also, what exactly do you mean with "first generation"? The PagingSource comes from Room, and there is data in the table, so how would the first generation be empty?

Generation means a instance of PagingData + PagingSource. For each emission from Flow<PagingData> Paging invokes your pagingSourceFactory to get a new PagingSource so it can basically reload and pick up any new changes since invalidation.

The first generation is typically empty in the DB + Network case, because PagingSource loads from DB and we have to wait for RemoteMediator to fetch from network and insert in DB, and these run concurrently (in case there is pre-existing cached data), and usually loading from network takes longer than loading from DB.

What is "end of data" in the context of a Room PagingSource? The last item of the current page? Or the last item in the complete table? Why would that occur without scrolling if I have a pageSize of 4, and an initialLoadSize of 12. After a REFRESH the database would have 12 items, the RecyclerView would show 1-2 items, so why would the last item in the database be accessed, since that would be 2-3 pages further.

endOfPagination happens whenever PagingSource returns null for nextKey / prevKey, in Room's case this is last item in the table. It's the last row matching the @Query you provided in your Dao. If you read the solution below though, it turns out this was a RemoteMediator implementation issue.

As for the problem in the sample...

So I took a log of the request RemoteMediator makes and what items are fetched:

RemoteMediator.load(REFRESH) apiResponse = scrcpy, ionic-framework, awesome-android-ui, okhttp, architecture-samples, retrofit, MPAndroidChart, fastlane, shadowsocks-android, glide, lottie-android, android-open-project, AndroidUtilCode, ijkplayer, zxing, android_guides, leakcanary, butterknife, lottie-web, ResumeSample, joplin, EventBus, SmartRefreshLayout, Signal-Android, iosched, architecture-components-samples, android-best-practices, RxAndroid, material-dialogs, Magisk, libgdx, skrollr, picasso, ExoPlayer, PhotoView, Telegram, Android-Universal-Image-Loader, flexbox-layout, fresco, framework7, plaid, anko, tinker, Android-CleanArchitecture, sunflower, NewPipe, CircleImageView, expo, appium, Material-Animations, ARouter, kivy, QMUI_Android, logger, Android-PickerView, MMKV, create-react-native-app, xbmc, material-components-android, greenDAO
RemoteMediator.load(APPEND) apiResponse = joplin, EventBus, SmartRefreshLayout, Signal-Android, iosched, architecture-components-samples, android-best-practices, RxAndroid, material-dialogs, Magisk, libgdx, skrollr, picasso, ExoPlayer, PhotoView, Telegram, Android-Universal-Image-Loader, flexbox-layout, fresco, framework7
RemoteMediator.load(APPEND) apiResponse = plaid, anko, tinker, Android-CleanArchitecture, sunflower, NewPipe, CircleImageView, expo, appium, Material-Animations, ARouter, kivy, QMUI_Android, logger, Android-PickerView, MMKV, create-react-native-app, xbmc, material-components-android, greenDAO
RemoteMediator.load(APPEND) apiResponse = react-native-maps, stetho, uamp, bytecode-viewer, AndroidSwipeLayout, Apktool, Matisse, stf, VirtualXposed, AndroidViewAnimations, tachiyomi, AndroidAutoSize, Android, RxTool, MaterialDrawer, VasSonic, banner, filament, SlidingMenu, androidannotations

In PagingConfig we have initialLoadSize = 60 and pageSize = 20. So it looks like our initial load loads the first 60 items from network, and in the following APPENDs we change the pageSize to 20 as expected, but we don't increment the page # correctly. Since we want to load items 60-80, Github API would treat that as page #4 with pageSize = 20.

Paging will essentially continue to request RemoteMediator loads until it either hits endOfPagination = true, or it gets enough items added to let PagingSource load the next page. In this case we have these redundant APPENDs because we aren't fetching any new data.

To resolve this, we can change nextKey calculation in GithubRemoteMediator to:

val nextKey = if (endOfPaginationReached) null else page + (pageSize / state.config.pageSize)

Which gives us the correct loads:

RemoteMediator.load(REFRESH) apiResponse = scrcpy, ionic-framework, awesome-android-ui, okhttp, architecture-samples, retrofit, MPAndroidChart, fastlane, shadowsocks-android, glide, lottie-android, android-open-project, AndroidUtilCode, ijkplayer, zxing, android_guides, leakcanary, butterknife, lottie-web, ResumeSample, joplin, EventBus, SmartRefreshLayout, Signal-Android, iosched, architecture-components-samples, android-best-practices, RxAndroid, material-dialogs, Magisk, libgdx, skrollr, picasso, ExoPlayer, PhotoView, Telegram, Android-Universal-Image-Loader, flexbox-layout, fresco, framework7, plaid, anko, tinker, Android-CleanArchitecture, sunflower, NewPipe, CircleImageView, expo, appium, Material-Animations, ARouter, kivy, QMUI_Android, logger, Android-PickerView, MMKV, create-react-native-app, xbmc, material-components-android, greenDAO
RemoteMediator.load(APPEND) apiResponse = react-native-maps, stetho, uamp, bytecode-viewer, AndroidSwipeLayout, Apktool, Matisse, stf, VirtualXposed, AndroidViewAnimations, tachiyomi, AndroidAutoSize, Android, RxTool, MaterialDrawer, VasSonic, banner, filament, SlidingMenu, androidannotations

Here's a similar fix uploaded to our Codelab: https://github.com/googlecodelabs/android-paging/pull/176

Thanks for looking into it. Incredibly obvious when you see it but something that's easily missed.

Np! Thanks for being patient, sorry it took awhile to respond!