Self-updating Paging results [160232968]

Assigned

Feature Request

Status Update

No update yet.

Description

cc...@google.com

created issue #1

Jun 30, 2020 03:52PM

Currently, Paging expects immutable gradually-loading snapshots to be returned from PagingSource, and that any content changes come through invalidate and re-load.

This works fine for Room's behavior around SQLite, or a pull-only network API that doesn't update individual items, but may not work for other cases.

The feature here would likely be a new Page type that can return a self-updating Page.

Currently:

Page<T>(data: List<T>, ...)

Self updating version could be:

UpdatingPage<T>(data: Flow<List<T>>, ...)

I'd appreciate if others could chime in about how useful this would be, and about examples of data sources that would benefit from this.

One constraint to making this work is that we'll likely need to assume these item updates don't affect separator positions (until items are dropped and reloaded).

Comments

au...@gmail.com <au...@gmail.com> #2Jun 30, 2020 04:21PM

Can you please turn on debug logging using https://developer.android.com/topic/libraries/architecture/workmanager/advanced/custom-configuration.

Set Log level to DEBUG. Send us the logs.

ri...@soundcloud.com <ri...@soundcloud.com> #3Jun 30, 2020 04:50PM

Sure thing!

Here we go:

D/WM-PackageManagerHelper: androidx.work.impl.background.systemjob.SystemJobService enabled
D/WM-Schedulers: Created SystemJobScheduler and enabled SystemJobService
D/WM-ForceStopRunnable: Performing cleanup operations.
D/WM-PackageManagerHelper: androidx.work.impl.background.systemalarm.RescheduleReceiver enabled
D/WM-SystemJobScheduler: Scheduling work ID 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd Job ID 74
D/WM-GreedyScheduler: Starting work for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: Processor: processing 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-WorkerWrapper: Starting work for REDACTED.Application$ForegroundWorker
I/WM-Processor: Moving WorkSpec (2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd) to the foreground
D/WM-WorkerWrapper: REDACTED.Application$ForegroundWorker returned a Success {mOutputData=Data {}} result.
I/WM-WorkerWrapper: Worker result SUCCESS for Work [ id=2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd, tags={ REDACTED.Application$ForegroundWorker } ]
D/WM-Processor: No more foreground work. SystemForegroundService is already stopped
D/WM-GreedyScheduler: Cancelling work ID 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: Processor stopping background work 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: WorkerWrapper could not be found for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-StopWorkRunnable: StopWorkRunnable for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd; Processor.stopWork = false
D/WM-SystemJobService: onStartJob for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: Processor: processing 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-WorkerWrapper: Status for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd is SUCCEEDED; not doing any work
D/WM-WorkerWrapper: REDACTED.Application$ForegroundWorker is not in ENQUEUED state. Nothing more to do.
I/WM-SystemFgDispatcher: Started foreground service Intent { act=ACTION_START_FOREGROUND cmp=REDACTED/androidx.work.impl.foreground.SystemForegroundService (has extras) }
D/WM-SystemFgDispatcher: Notifying with (id: 42, workSpecId: 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd, notificationType: 0)
D/WM-SystemFgDispatcher: Notifying with (id: 42, workSpecId: 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd, notificationType: 0)
D/WM-Processor: Processor 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd executed; reschedule = false
D/WM-SystemJobService: 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd executed on JobScheduler
D/WM-SystemJobService: onStopJob for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: Processor 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd executed; reschedule = false
D/WM-SystemJobService: 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd executed on JobScheduler
D/WM-Processor: Processor stopping background work 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-Processor: WorkerWrapper could not be found for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd
D/WM-StopWorkRunnable: StopWorkRunnable for 2d2cab8b-beb4-45c4-8d04-9d85e5aa4ecd; Processor.stopWork = false

ma...@marcardar.com <ma...@marcardar.com> #4Aug 17, 2020 04:01PM

Just tried 2.3.4 which claims to resolve the issue 150553353 but the issue described here is not resolved (despite this one marked as a duplicate for some reason).

Logs:

D/WM-PackageManagerHelper: androidx.work.impl.background.systemjob.SystemJobService enabled
D/WM-Schedulers: Created SystemJobScheduler and enabled SystemJobService
D/WM-ForceStopRunnable: Performing cleanup operations.
D/WM-PackageManagerHelper: androidx.work.impl.background.systemalarm.RescheduleReceiver enabled
D/WM-SystemJobScheduler: Scheduling work ID 2d44c9de-0e3c-4518-b960-c282ba924638 Job ID 299
D/WM-GreedyScheduler: Starting work for 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: Processor: processing 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-WorkerWrapper: Starting work for REDACTED.Application$ForegroundWorker
I/WM-Processor: Moving WorkSpec (2d44c9de-0e3c-4518-b960-c282ba924638) to the foreground
D/WM-WorkerWrapper: REDACTED.Application$ForegroundWorker returned a Success {mOutputData=Data {}} result.
I/WM-WorkerWrapper: Worker result SUCCESS for Work [ id=2d44c9de-0e3c-4518-b960-c282ba924638, tags={ REDACTED.Application$ForegroundWorker } ]
D/WM-Processor: No more foreground work. SystemForegroundService is already stopped
D/WM-GreedyScheduler: Cancelling work ID 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: Processor stopping background work 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: WorkerWrapper could not be found for 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-StopWorkRunnable: StopWorkRunnable for 2d44c9de-0e3c-4518-b960-c282ba924638; Processor.stopWork = false
D/WM-SystemJobService: onStartJob for 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: Processor: processing 2d44c9de-0e3c-4518-b960-c282ba924638
I/WM-SystemFgDispatcher: Started foreground service Intent { act=ACTION_START_FOREGROUND cmp=REDACTED/androidx.work.impl.foreground.SystemForegroundService (has extras) }
D/WM-SystemFgDispatcher: Notifying with (id: 42, workSpecId: 2d44c9de-0e3c-4518-b960-c282ba924638, notificationType: 0)
D/WM-SystemFgDispatcher: Notifying with (id: 42, workSpecId: 2d44c9de-0e3c-4518-b960-c282ba924638, notificationType: 0)
D/WM-WorkerWrapper: Status for 2d44c9de-0e3c-4518-b960-c282ba924638 is SUCCEEDED; not doing any work
D/WM-WorkerWrapper: REDACTED.Application$ForegroundWorker is not in ENQUEUED state. Nothing more to do.
D/WM-Processor: Processor 2d44c9de-0e3c-4518-b960-c282ba924638 executed; reschedule = false
D/WM-SystemJobService: 2d44c9de-0e3c-4518-b960-c282ba924638 executed on JobScheduler
D/WM-SystemJobService: onStopJob for 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: Processor stopping background work 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-Processor: WorkerWrapper could not be found for 2d44c9de-0e3c-4518-b960-c282ba924638
D/WM-StopWorkRunnable: StopWorkRunnable for 2d44c9de-0e3c-4518-b960-c282ba924638; Processor.stopWork = false
D/WM-Processor: Processor 2d44c9de-0e3c-4518-b960-c282ba924638 executed; reschedule = false
D/WM-SystemJobService: 2d44c9de-0e3c-4518-b960-c282ba924638 executed on JobScheduler

ka...@gmail.com <ka...@gmail.com> #5Aug 18, 2020 04:28PM

We marked it duplicate because if your Worker ran long enough, it would trigger the other bug which had a similar sticky Notification. However, you have a combination of problems one of which we fixed in 2.3.4.

Your RxWorker implementation has a bug. You are calling setForegroundAsync(...), but you are not waiting for completion. setForegroundAsync(...) returns a ListenableFuture that you should be waiting for.

What happens is that your Worker running on a dedicated Executor is racing against setForegroundAsync() and in some cases it ends up being called after your Worker returns a Result.success().

du...@google.com <du...@google.com> #6Aug 18, 2020 04:57PM

Yep, the call does return the Future but get is called on it. I was under an impression that it blocks the execution until the Future delivers a result. Isn’t that the case here?

nd...@gmail.com <nd...@gmail.com> #7Apr 29, 2021 03:45PM

Yes, that is how you should be blocking on the ListenableFuture. When I last looked at your code, you were not doing it correctly.

[Deleted User] <[Deleted User]> #8Jun 2, 2021 03:33PM

Sorry, I’m confused. The sample in the opening comment has this:

Single
    .fromCallable {
        setForegroundAsync(ForegroundInfo(42, createNotification())).get().toOptional()
    }
    .map { Result.success() }

What it does (from my POV):

changes the worker to be foregrounded using setForegroundAsync;
blocks until the setForegroundAsync ListenableFuture provides it;
emits a successful worker result.

Is there a bug in this implementation? I’m confused because #5 states that there is a bug in the sample but #7 confirms that blocking via get() should work.

Message last modified on Jun 2, 2021 03:52PM

ne...@gmail.com <ne...@gmail.com> #9Jul 27, 2021 11:02AM

Are you calling setForegroundAsync() in your constructor ?

c4...@gmail.com <c4...@gmail.com> #10Sep 17, 2021 02:24AM

Comment has been deleted.

Message last modified on Sep 17, 2021 02:26AM

al...@pierlis.com <al...@pierlis.com> #11Oct 4, 2021 12:27PM

WorkManager has multiple schedulers (in-process, JobScheduler, the AlarmManager based scheduler etc.).

What's happening in your case, is that the in-process Scheduler is picking up the RxWorker implementation and your Worker completes successfully by returning an instance of Result.

However, around the same time JobScheduler also wants to execute the same WorkRequest. For us to be able to de-dupe the execution, we create an instance of the Worker and check to see if it needs to be scheduled. As your logs point out, we correctly identify that the Worker is done.

D/WM-WorkerWrapper: Status for 2d44c9de-0e3c-4518-b960-c282ba924638 is SUCCEEDED; not doing any work
D/WM-WorkerWrapper: REDACTED.Application$ForegroundWorker is not in ENQUEUED state. Nothing more to do.

However in your RxWorker implementation the side-effect of calling setForegroundAsync() in createWork() is that a call to notify() happens despite your Worker not having to run at all. The reason this happens is because there is no easy way for you to be able to tell that the Worker actually started running (because that happens only when an underlying subscription is created).

We made this a lot more ergonomic in 2.4.0-alpha02 (soon to be released). Attempts to notify are ignored when a Worker is done.

In your RxWorker for a workaround around the notifications, you could do something like:

Single
    .fromCallable {
        val workInfo = WorkManager.getInstance(context).getWorkInfoById(getId()).get()
        if (workInfo != null && !workInfo.getState().isFinished()) {
          setForegroundAsync(ForegroundInfo(42, createNotification())).get().toOptional()
        }
    }
    .map { Result.success() }

Going forward, we automatically handle this for you.

ge...@gmail.com <ge...@gmail.com> #12Oct 13, 2021 01:13AM

Awesome, thanks for the detailed explanation! I’ll look forward to the 2.4.0.

Sorry for bothering you with this. It was a bit weird to follow the documentation and still having an issue.

du...@google.com <du...@google.com> #13Oct 13, 2021 06:15PM

No problem. In 2.4.0-alpha02 you can just call setForegroundAsync() and not have to worry about calling get(). We do the rest :)

cc...@google.com <cc...@google.com> Mar 22, 2022 03:36PM

Reassigned to du...@google.com.

du...@google.com <du...@google.com> Aug 4, 2022 04:20PM

Reassigned to cl...@google.com.

cl...@google.com <cl...@google.com> Aug 24, 2022 05:56PM

Status: New

co...@gmail.com <co...@gmail.com> #14Sep 4, 2022 06:09AM

完全需要这个功能，我觉得这应该是一个基本的功能，但现在想实现这个基本功能却如此的痛苦

pu...@gmail.com <pu...@gmail.com> #15May 10, 2023 01:21PM

Seems like paging 3 library was just implemented without any regard to what end user wants to do with it?
Now, for user to interact something on the view, which would call the viewmodel to POST something on backend and based on the result, change the UI a bit is impossible, without actually invalidating the list and calling the backend again to update the current view. No, I am not using any Local storage for this. Our source of truth is backend. This is so silly

qw...@gmail.com <qw...@gmail.com> #16May 29, 2023 06:30AM

Abandoned, this is a garbage library, when you need to insert data, or clear data, it is very troublesome

cl...@google.com <cl...@google.com> #17May 29, 2023 05:54PM

Re: comment#16 What kind of issues are you facing? Feel free to file a separate bug and I can help you there.

aa...@gmail.com <aa...@gmail.com> #18Jun 20, 2023 12:42PM

Re:

comment#17

use case: in case of horizontal recycler view inside vertical recycler view (e.g. OTT like app).
Pages can be fetched from one source But in some horizontal list source can be different like: continue watch / my list .. and just to update one horizontal rv we need to invalidate whole paged data. Isn't it unnecessary load on low end devices like TV ?

cl...@google.com <cl...@google.com> #19Jun 20, 2023 06:42PM

Re comment#18

For your use case, I would highly suggest using different adapters for each nested horizontal RV. This is crucial so that

Paging can individually track scroll state of each horizontal row
You only need to invalidate data for one row

Or you're probably already doing that and your implementation would invalidate all inner PagingSource whenever the outer PagingSource is invalidated?

But I understand where you're coming from and self-updating Paging is still something we intend to address.

Message last modified on Jun 20, 2023 10:37PM

aa...@gmail.com <aa...@gmail.com> #20Jun 21, 2023 06:14AM

comment#19

Definitely update can be done by using individual row's paging source.

looks like I missed - Use Case:
suppose user can add or remove items in MyList (lets assume we have MyList horizontal row at position x in our vertical list - where vertical list has its own network backed paging source - which tells us to show MyList at position x )
1. when all items are removed from MyList we need to remove MyList from our vertical list.
2. when first item is added in MyList we need to add MyList in vertical list at same position.

For these we need to make change in vertical list.
so how this is possible ?

cl...@google.com <cl...@google.com> #21Jun 21, 2023 07:47PM

I see. I understand your use case. To emphasize, we still want to implement self-updating Paging. This is not abandoned.

an...@gmail.com <an...@gmail.com> #22Aug 10, 2023 02:47AM

Hi, I'd like to add one more possible use case to consider.

I am involved in Tusky, a client for "Mastodon" (think Twitter but Linux). We are attempting to switch to paging3 for our displayed lists (timelines of posts, lists of notifications). We have one particular feature which seems to be incompatible with paging3 currently:

Imagine you read some posts on Mastodon, you set down your phone, you come back 12 hours later. The app goes to load more posts, but discovers there are many, many posts, more than would be convenient to display. Instead of loading the next page of posts, the app places a button saying "LOAD MORE" above the previously loaded posts. Above that it puts the newest page of posts. If you tap "LOAD MORE", the app can then load page [PRESENT]-1 or [PAST]+1 above or below the "LOAD MORE" button (and if more pages still remain between present and past, stick another "LOAD MORE" button in between.

This is a pretty common idiom, but it seems to be impossible in Paging3 because there is no way to insert pages "in the middle" between two other pages. It only seems to be possible to add to the head or the tail, or to invalidate the entire list (from our perspective a heavyweight operation). The only solution we've found to this is to simply drop the "LOAD MORE" feature, which may have been a good idea for other reasons, but it's too bad if we're having to drop features because paging3 does not support them rather than because we specifically decided the program would be better about this.

(In addition to this, as with other posters in this thread, we are also struggling with the ability to update a single row out of the dataset, something we need because posts on Mastodon can be deleted or edited by their posters after we have already displayed them. We have a solution for this but so far it doesn't work so great.)

le...@gmail.com <le...@gmail.com> #23Oct 3, 2023 06:35PM

I am using sqldelight OffsetQueryPagingSource for the Pager. When removing an item from the recycler view, the removal animation is lagging (DefaultItemAnimator). This is because we update the DB twice in a short period. I can mitigate this by ensuring we update the DB only once for this use case, but the DB update can happen asynchronously. Another way to "fix" the lag is adding debounce(500L) to the paging data flow, but not liking it. I hope this feature will let me fix the lagging issue, but if you have any other suggestions, please let me know.

pa...@gmail.com <pa...@gmail.com> #24Dec 4, 2023 10:44PM

what is the timeline of this?? its one of the most requested features

cl...@google.com <cl...@google.com> Feb 14, 2024 06:45PM

Assigned to cl...@google.com.

db...@gmail.com <db...@gmail.com> #25Aug 12, 2024 12:18PM

Is there an ETA on this potentially? Since the last activity date was in February this year? We're very interested to know if there is a workaround or potential update coming for this.

ma...@gmail.com <ma...@gmail.com> #26Sep 2, 2024 11:19AM

I would like to know about this, too.