IssueTracker

As per issue 175247132 I tried the latest snapshot of 3.0.0 and the issue reported in 175247132 is still not fixed.

I have replied in the issue that was marked as duplicate, but I have no understanding whether duplicates are ignored so echoing part of it to here. In my scenario I cannot have placeholders, thus this fix will not work as per the CL description and for that I also believe it should not have been marked as duplicate of this one and keep it visible instead.

If there will be no intent on solving this issue for the scenario where placeholders are not enabled then it completely invalidates that use case scenario given the developer will be left with absolutely no means to retain scroll position in the list when a fragment is resumed/restored and that seems like a monumental issue to me.

I've replied in the other thread, but to be clear this ticket is not yet fully resolved.

I'm subscribed to all updates in paging issues, including closed ones so happy to continue helping you there, but I'd like to continue using this ticket to track progress on full saved state support.

Hi, this bug is not closed w/ that CL so we know there is still work to do here. Keeping a single bug makes it easier to maintain and duplicate issues are still linked here.

I'll check your application to see why it is not working as you are using cachedIn hence it should work for your use case (except when application is restarted).

Ah so sorry, I was under the impression the fix had been finalized, apologies for my mistake.

ok so the problem in your sample is unrelated.

In your code, every time getItems is called, you are creating a new Pager hence the cachedIn is useless. You need to create it once and re-use for cachedIn to be useful.

That being said, I don't blame you because cachedIn receives a lifecycle hence you probably felt the obligation to create it lazily with a lifecycle (and view models do not have a lifecycle but they do have a CoroutineScope which is necessary for cachedIn). We should probably add an overload that will receive a view model instead of a lifecycle. Meanwhile, you have two quick fixes possible:

public final LiveData<PagingData<ItemEntity>> pagingDataLiveData;
    public FirstFragmentViewModel(@NonNull Application application) {
        super(application);
        myRepository = new MyRepository(application);
        pagingDataLiveData = PagingLiveData.cachedIn(
                PagingLiveData.getLiveData(new Pager<>(pagingConfig, myRepository::getItems)),
                ProcessLifecycleOwner.get().getLifecycle()
        );
    }

but i wouldn't ship that because it uses the process's lifecycle (so won't be cleaned after fragment goes away)

Instead, a proper fix for now would be to write:

public final LiveData<PagingData<ItemEntity>> pagingDataLiveData;
    public FirstFragmentViewModel(@NonNull Application application) {
        super(application);
        myRepository = new MyRepository(application);
        pagingDataLiveData = PagingLiveData.cachedIn(
                PagingLiveData.getLiveData(new Pager<>(pagingConfig, myRepository::getItems)),
                ViewModelKt.getViewModelScope(this)
        );
    }

which will require you to add a dependency on viewmodelktx (implementation 'androidx.lifecycle:lifecycle-viewmodel-ktx:<version>') until we add a public api for cachedIn to receive a view model instead.

Btw, there is another unrelated problem in your code. You are caching the binding which will cause leaks. You should always create the binding when onCreateView is called and get rid of it after onDestroyView.

Also, you should use the view lifecycle instead of fragment lifecycle while observing. Otherwise, you'll keep accumulating observers that will try to modify the same adapter and race each-other.

Similarly, you cannot reuse the adapter.

Here is a possible fix for that (not necessarily the only way, but fixes your problems):

public class FirstFragment extends Fragment {
    FirstFragmentViewModel viewModel;

    @Override
    public void onCreate(@Nullable @org.jetbrains.annotations.Nullable Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        viewModel = new ViewModelProvider(this).get(FirstFragmentViewModel.class);
    }

    @Nullable
    @Override
    public View onCreateView(
        @NonNull LayoutInflater inflater,
        @Nullable ViewGroup container,
        @Nullable Bundle savedInstanceState
    ) {
        FirstFragmentAdapter adapter = new FirstFragmentAdapter();
        FragmentFirstBinding binding = FragmentFirstBinding.inflate(inflater, container, false);
        binding.recyclerView.setLayoutManager(new LinearLayoutManager(getContext(), LinearLayoutManager.VERTICAL, false));
        binding.recyclerView.setAdapter(adapter);

        viewModel.pagingDataLiveData.observe(
                getViewLifecycleOwner(),
                itemEntityPagingData -> adapter.submitData(getViewLifecycleOwner().getLifecycle(), itemEntityPagingData)
        );
        return binding.rootContainer;
    }
}

One last thing you need to fix in your sample is that every time activity is rotated, you are creating a new fragment which prevents state restoration. You need to avoid pushing the new fragment if it already exists. A quick fix for that is:

public void openFragment(
        final Fragment fragment,
        final boolean addToBackStack
    ) {

        final String fragmentTag;
        final FragmentManager fragmentManager;

        fragmentTag = fragment.getClass().toString();
        fragmentManager = getSupportFragmentManager();
        if (fragmentManager.findFragmentByTag(fragmentTag) != null) {
            return;
        }
        .....

I would highly recommend using the navigation library to avoid dealing with these issues and it will do the right things for you.

These changes will fix your common case, it still won't handle activity being re-created for process death but for the rest of cases, it should work fine.

created https://b/175332619 for the view model scope extension.

I am very much grateful to your help and your colleague as well.

I followed your steps and it worked perfectly; the state is 100% kept now.

I had to use the ViewModelKt.getViewModelScope(this) instead of the activity (it was suggested in the other issue as an option, although this one was still enforced too) and the ViewModelKt.getViewModelScope(this) worked best.

I had no idea ini-on-null was open to memory leaks 🤦🏻‍♂️

I followed that pattern as per suggestions in StackOverflow discussions in regards to not being needed to initialize them again if they are still kept in memory, so I did that and found no issues. It was also what helped me work around some problems that I had at the time, which I believe might have been because of what we are discussing here actually. So now that this is "fixed", at least for me, I will have to update the project for where I am making use of this ini-on-null pattern to prevent the issues you described.

Also the duplicate fragment on rotation was a serious problem I had in the beginning of the project and could not find a solution for it other than disabling screen rotation all-together. I am not gonna dismiss what you suggested, I will definitely implement that check in the manager, but for sure I am planning on implementing the navigation components to replace this.

The only problem I have with the navigation components so far is for the scenario where I am using the toolbar inside the fragments themselves (unlike outside, globally unique) its state is not being restore correctly. For example, problems like the one mentioned in this question: https://stackoverflow.com/questions/63695127/

However that is way beyond this issue now and something I'll look into it better when I start implementing it, hopefully it was an issue that has since been solved.

Other than that I am so very thankful for you help, again, and so glad to finally get this one behind me, it was taking so much of my time trying to work around it to the point where I was implementing my own made-up state restoration policy just for this.

Seems like the state is also lost if I add any transformation to the returned data in adapter.submitData(getViewLifecycleOwner().getLifecycle(), itemEntityPagingData), for example if I just want to insert separators using PagingData.insertSeparators. Do I also have to cache these or is it a different process entirely?

Ignore my last question, I just noticed this is because the data being loaded in first fragment is also being changed when I open the second fragment. I am assuming that what this is doing is causing the data in the first fragment to be "flagged" as changed when I returned to it as a result of related data being changed/updated in the second fragment.

This will be a tricky one to work around because when I don't update any data after opening the second fragment and return to the first fragment everything is ok, the state of the list is kept intact.

This would be an example scenario of a message list in first fragment and when you click on a message it opens its details in the second fragment, but because I don't have the full details of the message when I open the second fragment then I have to load them from the server and update the database with the result. Since the message in the first fragment and the message details in the second share common columns/data it might be causing this issue again and I don't know what is the best workaround for this one. I imagine making a single table just for message details and a completely different table just for the messages would mitigate this problem, but then keeping the data synchronized across tables would become a monumental task.

I thought maybe having completely separate repositories for each of the fragments would be enough, but as I found out it is not.

yea changing data is challenging :/ When you are in the list and data changes, we are able to calculate a diff for RecyclerView to find the same items. It gets tricky on restore where we don't have a list to diff from :( so it becomes more like a saved state.

I'm glad we were able to help with the basic case at least.

We are trying to figure out what kind of APIs / helpers we can implement to aid with more complicated cases (like changing lists, dropped pages etc) and honestly we don't yet have a good solution that is helpful & generic enough.

So if you try custom restore, would you mind sharing that implementation / algorithm with us? It would help us greatly in discovering use cases.

I would not dare showing any glimpse of the Frankenstein I was trying to build, it was a mash of multiple methods extracted from the framework internals and I was not going anywhere reliably because there are so many package-private accesses that I needed I was almost recreating the whole thing from scratch.

I'm currently stuck with a strange dilemma of a recyclerview inside a viewpager2 inside a fragment. For some reason the state of that recyclerview is not being restored correctly, but when I create a sample test case with the same structure, but way more basic, the state is being kept without any issue whatsoever. I made sure I was passing the getChildFragmentManager() and getViewLifecycleOwner().getLifecycle() to the FragmentStateAdapter and also tried with requireActivity() and just this. This one might be just a matter of debugging, I have a feeling that I cannot get it to restore the state correctly because on my project I have Transformations.map inside the PagingLiveData.cachedIn to be able to trigger a new data fetch when the user searches for something, but on my sample test case I am using the same straight forward code presented in this issue.

As for your point, and if my suggestion based on a limited knowledge is any worth, I'd say that if it is so hard to work around the inability to DIFF_* the list during restore then provide a mechanism where we, the developers, inject somewhere for the time being the same DIFF we use in the adapter. It would be a band-aid on the problem, it would fix the issue for us and in the meantime you can find a proper/better method for automating this.

Something like recyclerView.registerDiffCallback(adapter.DIFF_CALLBACK); tying the diff into the recyclerview's state restoration logic.

I am now having an even bigger problem with this coupled with Transformations.switchMap. While I am making changes that are reflected in the first page of the list everything works without any issues, but when I scroll to any page other than the first and make changes the list reloads itself completely, throwing the user to the first page regardless of where he was before.

The changes are applied correctly still, but the behavior is completely state destructive.

See attached recording exemplifying the problem.

The code is based on the sample provided in a duplicate issue and improved based on #9 suggestions in this issue. Sample app is attached and the relevant SC portions are as follows:

FirstFragmentViewModel

    final LiveData<PagingData<ItemEntity>> pagingDataLiveData;
    // this is just a reference list that stores whichever
    // items have been selected and changes to this list are
    // being used to signal/dispatch the main list above in
    // order to update the items themselves with the new changes
    final MutableLiveData<List<ItemEntity>> selectedItems;

    public FirstFragmentViewModel(@NonNull Application application) {

        super(application);

        selectedItems = new MutableLiveData<>(new ArrayList<>());

        MyRepository myRepository = new MyRepository(application);
        PagingConfig pagingConfig = new PagingConfig(2, 10, false, 15);
        Pager<Integer, ItemEntity> pager = new Pager<>(pagingConfig, myRepository::getItems);
        // here is where I am not sure if this is done right because
        // if I cache the `PagingLiveData.getLiveData(pager)` then it
        // never updates whenever `selectedItems` changes
        LiveData<PagingData<ItemEntity>> switchMap = Transformations.switchMap(selectedItems, input -> PagingLiveData.getLiveData(pager));
        this.pagingDataLiveData = PagingLiveData.cachedIn(switchMap, ViewModelKt.getViewModelScope(this));

    }

FirstFragment

    @Nullable
    @Override
    public View onCreateView(
        @NonNull LayoutInflater inflater,
        @Nullable ViewGroup container,
        @Nullable Bundle savedInstanceState
    ) {

        FirstFragmentAdapter adapter = new FirstFragmentAdapter();
        adapter.setOnItemClickListener(viewModel::toggleItem);
        FragmentFirstBinding binding = FragmentFirstBinding.inflate(inflater, container, false);
        binding.recyclerView.setLayoutManager(new LinearLayoutManager(getContext(), LinearLayoutManager.VERTICAL, true));
        binding.recyclerView.setAdapter(adapter);

        viewModel.pagingDataLiveData.observe(
            getViewLifecycleOwner(),
            itemEntityPagingData -> adapter.submitData(
                getViewLifecycleOwner().getLifecycle(),
                itemEntityPagingData.map(itemEntity ->
                    new ItemData(viewModel.isSelected(itemEntity), itemEntity)
                )
            )
        );

        return binding.rootContainer;

    }

The interaction is very basic; the item alpha value changes when it is clicked.

The sample app follows attached.

Am I doing this wrong? I followed examples found online and derived from that because the available documentation does not present an explanation on how to do this correctly.

Well, I ended up stumbling across MediatorLiveData and I would never have thought this would be the appropriate use for it by no means. Every time I read about it, it is always related to loading states combined with a single stream of data. Apparently it can take any sources we want, so as soon as I coupled the two lists in my sample project it worked right away without any issues, incredible.

Just in case this might be useful for any future reader here is what I did:

FirstFragmentViewModel

    public FirstFragmentViewModel(@NonNull Application application) {

        super(application);

        selectedItems = new MutableLiveData<>(new ArrayList<>());

        MyRepository myRepository = new MyRepository(application);
        PagingConfig pagingConfig = new PagingConfig(2, 10, false, 15);
        Pager<Integer, ItemEntity> pager = new Pager<>(pagingConfig, myRepository::getItems);
        LiveData<PagingData<ItemEntity>> liveData = PagingLiveData.getLiveData(pager);
        LiveData<PagingData<ItemEntity>> pagingDataLiveData = PagingLiveData.cachedIn(liveData, ViewModelKt.getViewModelScope(this));

        // we use `transformed` to feed the adapter using `adapter.submitData(...)`
        // in place of the `pagingDataLiveData` we were using before so we can
        // combine two different streams into one stream and respond to item changes
        // without affecting the list state in the recyclerview
        transformed = new MediatorLiveData<>();
        transformed.addSource(pagingDataLiveData, this::transformData);
        transformed.addSource(selectedItems, this::updateSelectedItems);

    }

    private void transformData(@NonNull PagingData<ItemEntity> itemEntityPagingData) {
        // here we convert the stream we receive from the source (either
        // server or room DB) to the type of data we want to feed to the
        // recyclerview with the correct state if we already have enough
        // information. for ex. if this is the first time we are loading
        // then none of the items will be selected, but if we are loading
        // more pages of items (user can be scrolling up or down) then we
        // might already have some items that are selected and are out of
        // view, this ensures they wil remain selected if the user had
        // already selected them
        transformed.setValue(itemEntityPagingData.map(
            itemEntity -> new FirstFragment.ItemData(isSelected(itemEntity), itemEntity)
        ));
    }

    private void updateSelectedItems(List<ItemEntity> itemEntities) {
        // this is a simple update process, we never transform it here
        // since we want to keep the integrity of the original list
        // and only update the state of the items whenever they are
        // loaded or reloaded
        // NOTE should we be concerned with ConcurrentModificationException
        // here since we are transforming the list directly?
        transformed.setValue(transformed.getValue().map(itemData -> {
            itemData.selected = isSelected(itemData);
            return itemData;
        }));
    }

FirstFragment

    @Nullable
    @Override
    public View onCreateView(
        @NonNull LayoutInflater inflater,
        @Nullable ViewGroup container,
        @Nullable Bundle savedInstanceState
    ) {

        FirstFragmentAdapter adapter = new FirstFragmentAdapter();
        adapter.setOnItemClickListener(viewModel::toggleItem);
        FragmentFirstBinding binding = FragmentFirstBinding.inflate(inflater, container, false);
        binding.recyclerView.setLayoutManager(new LinearLayoutManager(getContext(), LinearLayoutManager.VERTICAL, true));
        binding.recyclerView.setAdapter(adapter);

        viewModel.transformed.observe(
            getViewLifecycleOwner(),
            // simply feed the adapter directly, any transformations
            // have already been handled by the ViewModel so we are
            // now receiving a stream with the final data information
            itemEntityPagingData -> adapter.submitData(getViewLifecycleOwner().getLifecycle(), itemEntityPagingData)
        );

        return binding.rootContainer;

    }

One concern that I have is the note I added in the comment in the ViewModel regarding the possibility of a CME occurring, but that's outside the scope of this issue and it is bound to happen I know it will happen in due time.

So far no amount of workarounds has resolved this issue for me. I either somehow never update the data of a list while that list is not visible, or the list scroll position will always be lost when it is restored after data has "changed", even if said that never actually changes, a simple "update" where no data changes is enough to cause this problem.

Seeing that at the moment this is apparently not something that has much focus in fixing, I have no option other than completely drop this and use static lists, which will be a nightmare in terms of memory usage when they become bigger over time on the users' devices, but it is still better than using paging like this and provide the user with a broken user experience.

you cannot change the data you send to paging because it will break diffing (we need the original version of the item to be able to diff properly).

Have you tried just copying items instead, or at least do it if its' selected has changed?

• note to maintainers: I've moved this back into paging 3.0 hotlist so it stays in the radar even though we don't have any more changes planned for saved state. Will move this out after the case at hand is resolved.

• I have a search result list, that is being fed with results from the Room database using PagingDataAdapter.
• I scroll down to a random item, and click it which in turn opens a new fragment with a list of data associated with that item selection.
• In that fragment the data is also being populated via PagingDataAdapter.
• I scroll in that fragment to see more data, eventually it hits the end of the list and requests more data from the server.
• I receive more data from the server, the fragment updates correctly and I keep scrolling until.
• I close that fragment, which returns me to the previous result list.
• That result list position is now lost because there was more data loaded in the previous fragment, which is the issue described in this case.

Both the result list and the fragment data need to be able to load more data if the user scrolls to the end of their lists. This is a very basic scenario. A contact list, where you open the details of each contact and return back to the contact list, or a conversation list, where you open the conversation and return back, or an email list, etc.

When you say that I "cannot change the data" how exactly are you proposing that I do that in this scenario? Because, as it is, the only workaround I can think of is instead of having one database with X tables, I would need X databases with just 1 table, with each and every single database representing an isolated data collection strictly for each and every fragment that displays it just so I can update data and not having it interfere with lists from different fragments that are in the backstack.

No documentation available provides any guide on how to do this correctly either, or at least any that is available is not that clear about it.

Don't take this the wrong way, I appreciate any sort of help here, it's just so far I am on my own trying to get this to work properly and failing at it every time.

That result list position is now lost because there was more data loaded in the previous fragment, which is the issue described in this case.

This should not be a problem as long as the page size is large enough such that previous items are still in the new list (at least 1 of them so that we can figure out the relation between two versions). If that happens but UI still jump, that is a bug we should look at (can you share a repro for it?). If items get re-ordered etc, that might be a problem because we simply cannot know what needs to show up in UI. We fixed some bugs there in beta01 so may have also been already fixed if you've not updated or might still be broken. Btw there is some followup work happening here: (prototypes) https://android-review.googlesource.com/c/platform/frameworks/support/+/1600592 https://android-review.googlesource.com/c/platform/frameworks/support/+/1621325/1/paging/runtime/src/main/java/androidx/paging/NullPaddedDiffing.md

Also, I'm assuming you don't use the same PagedList between two fragments as they will race each other on ranges they load, make sure you create a paged list for each fragment.

When you say that I "cannot change the data" how exactly are you proposing that I do that in this scenario? Because, as it is, the only workaround I can think of is instead of having one database with X tables, I would need X databases with just 1 table, with each and every single database representing an isolated data collection strictly for each and every fragment that displays it just so I can update data and not having it interfere with lists from different fragments that are in the backstack.

When I said "don't modify the item", i meant the value in memory.

So when an item is selected, if you go and set its .selected value, when we diff, differ won't detect the selected property has changed because it is literally the same ItemData instance in both lists. You are free to change the value in the database as Room will always load a new instance of it.

Wrt documentation, we do actually have a project to rewrite and extend paging docs. It is just difficult to write comprehensive docs while the project is in active development (causes a lot of churn) so we tend to do those as we get closer to stable release rather than alphas.

You are right to get frustrated, we do as well :) I just want to make sure we properly understand the use case and either provide a solution or accept it as a problem we cannot solve in 3.0. This is also why repro apps are super useful so that we can debug and understand where things are failing instead of guessing.

I will try to attach a repro sample next week instead of trying to describe it even further, I understand that is way better for you guys than continuing my descriptions which may be over simplistic, leaving possible important information behind.

I've been very busy going around this still, and so far the ONLY thing that has worked flawlessly is to reuse the adapter, which is weird since in #9 I'm told I cannot reuse adapters.

So far reusing adapters has been the only solution that has managed to keep the recyclerview scroll position perfectly when I return to the fragment.

I tried to find any documentation where it explained, or at least warned against reusing adapters, but I found absolutely none. I also tried playing with adapter state restoration strategies, but they provided no benefit at all for this.

I did manage to not "modify" the data better, by moving the data modifications prior to the .cachedIn() call (it wasn't before) and I also tested with just a basic clean stream of data with no manipulations, and the behaviors were exactly the same.

Why can't I reuse adapters? I cannot find any explanation/justification anywhere, and if I do not reuse it then the scroll position gets all bogged up.

I just updated the paging library to version 3.0.0-beta02 and my problem is completely gone. The list is restoring in the right scroll position, regardless of how many pages it scrolled (as in it is working correctly even when it wasn't positioned inside the first page of items). I actually tested it multiple times thinking that I was not testing it correctly since I could not believe it was working perfectly now, but indeed it is.

Note to anyone reading: Make sure you initiate and set your adapter at a lifecycle prior to when the activity or fragment are already displayed, or else it will be a problem if you do it later than that point and the list might show an erratic behavior. Do it in the onCreate() or onCreateView().

Thank you so very much, I believe this means that this issue can be marked as fixed.

Actually, if you have time, can you try with latest snapshot from androidx.dev ?

We did some major changes in placeholder diff handling that might affect your case and we would like to know that we didn't break your case :)

Last time I tried a snapshot and went back to the previous version, Android Studio never stopped telling me to update to the snapshot 🙆‍♂️ Can you provide with instructions, or point me to where I can find how I can stop that from happening, and I'll gladly try the latest snapshot

:o Android Studio should've never recommended you to use snapshot, that seems very wrong.

I was referring to snapshots here: https://androidx.dev/

Basically, you add this maven to your project:

maven { url 'https://androidx.dev/snapshots/builds/[buildId]/artifacts/repository'
}

where buildId is something you grab from android.dev (usually from the url of latest build, e.g. as if writing this it would be 7218523)

allprojects {
    repositories {
         maven { url 'https://androidx.dev/snapshots/builds/7218523/artifacts/repository'
}
}

then for your paging version, you set it to 3.0.0-SNAPSHOT.

If that studio bug happens again, would you mind filing a bug on studio, that seems like a bad bug.

Thanks!

Ahh yigit beat me to this :)

You can also use maven { url 'https://androidx.dev/snapshots/latest/artifacts/repository' } which redirects to latest build.

Does Android Studio tell you to update to SNAPSHOT even if you remove the above maven url?

Thanks to both of you, I will give it a try. And to answer that last question, that is correct. The only way I got it fixed was to delete the entire project locally, and pulling it from remote again.

I have tested the snapshot, used the version 7218984 and so far the issue remains fixed.

Unfortunately the problem that I mentioned before with Android Studio is still present, which means my project is now permanently bugged suggesting me to update the library to the snapshot version (see attached screenshot).

I did clear the build, I also did the clear cache and restart, but it keeps suggesting me to update to snapshot version regardless.

I've created issue 183137869 for that snapshot bug.

thanks!