IssueTracker

How should this work "else if (parent instanceof BottomSheetDialogFragment)"?

Yeah, I just realized that it's not as simple as in the case of the DrawerLayout. When using a DrawerLayout, people mostly have something like this:

<android.support.v4.widget.DrawerLayout xmlns:android="http://schemas.android.com/apk/res/android"
                                        xmlns:app="http://schemas.android.com/apk/res-auto"
                                        android:id="@+id/drawer_layout"
                                        android:layout_width="match_parent"
                                        android:layout_height="match_parent"
                                        android:fitsSystemWindows="true">

    <FrameLayout
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:id="@+id/header_container"/>

    <android.support.design.widget.NavigationView
        android:id="@+id/nav_view"
        android:layout_width="wrap_content"
        android:layout_height="match_parent"
        android:layout_gravity="start"
        app:headerLayout="@layout/main_nav_header"
        android:fitsSystemWindows="true"
        app:menu="@menu/main_drawercontent"/>

</android.support.v4.widget.DrawerLayout>

Obviously here the NavigationView is a direct child of the DrawerLayout, which is why the implementation we already have for handling the closing of the navigation drawer works like it does, by checking the NavigationView's parent.
But when using a BottomSheetDialogFragment, we can't rely on the NavigationView being a direct child of the BottomSheetDialogFragment. Since it's mostly a regular Fragment, people may use a layout like this (simplified):

<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".view.BottomDrawerFragment">

    <!-- Here we'd have views typically found in the navigation drawer,
         such as a user profile picture or their name -->

    <com.google.android.material.navigation.NavigationView
        android:id="@+id/bottom_drawer_navigation"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="bottom"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/divider"
        app:menu="@menu/bottom_drawer_menu" />
</androidx.constraintlayout.widget.ConstraintLayout>

My suggestion wouldn't work with this, as the NavigationView's parent is not the BottomSheetDialogFragment, but the ConstraintLayout. So we can't use the parent to detect when we should dismiss a BottomSheetDialogFragment.

A potential solution would probably have to use a different approach. Maybe in setupWithNavController() you could pass a reference to the BottomSheetDialogFragment directly, in addition to the NavigationView and NavController. Something similar happens already, for example when using a Toolbar that should be kept in sync also, the method call is NavigationUI.setupWithNavController(collapsingToolbarLayout, toolbar, navController) for that.
So we could end up having an implementation of setupWithNavController just for our case with the BottomSheetDialogFragment that goes something like this:

public static void setupWithNavController(
        @NonNull BottomSheetDialogFragment bottomSheetDialogFragment,
        @NonNull final NavigationView navigationView,
        @NonNull final NavController navController) {
    navigationView.setNavigationItemSelectedListener(
            new NavigationView.OnNavigationItemSelectedListener() {
                @Override
                public boolean onNavigationItemSelected(@NonNull MenuItem item) {
                    boolean handled = onNavDestinationSelected(item, navController, true);
                    if (handled) {
                        bottomSheetDialogFragment.dismiss();
                    }
                    return handled;
                }
            });
    final WeakReference<NavigationView> weakReference = new WeakReference<>(navigationView);
    navController.addOnNavigatedListener(new NavController.OnNavigatedListener() {
        @Override
        public void onNavigated(@NonNull NavController controller,
                @NonNull NavDestination destination) {
            NavigationView view = weakReference.get();
            if (view == null) {
                controller.removeOnNavigatedListener(this);
                return;
            }
            Menu menu = view.getMenu();
            for (int h = 0, size = menu.size(); h < size; h++) {
                MenuItem item = menu.getItem(h);
                item.setChecked(matchDestination(destination, item.getItemId()));
            }
        }
    });
}

And we'd use it in our example like so:

public class BottomDrawerFragment extends BottomSheetDialogFragment {
    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) {
        View view = inflater.inflate(R.layout.fragment_bottom_drawer, container, false);
        NavigationView navigationView = view.findViewById(R.id.bottom_drawer_navigation);
        NavigationUI.setupWithNavController(this, navigationView, Navigation.findNavController(getActivity(), R.id.nav_host_fragment));
        return view;
    }
}

You nailed it man. Thanks for your investigation!

Are there any cases where you'd have a NavigationView in a BottomSheet of any type and not expect the bottom sheet to be hidden when you tap on a NavigationView item?

If you'd always expect it to be hidden, we could use BottomSheetBehavior.from(navigationView) to find the bottom sheet, avoiding the need to create another setup method.

Not that I can think of. According to the Material Design spec, there are standard bottom sheets which remain open while the user interacts with the main UI region or the sheet itself, and modal bottom sheets which must be dismissed to interact with the underlying content.
Examples for standard bottom sheets are playback controls inside a music player or the location information in Google Maps, while modal bottom sheets provide a set of choices and are an alternative to inline menus.

This makes it pretty clear that modal bottom sheets should be used for any kind of menu, and modal bottom sheets are always dismissed by tapping a menu item or action. That being said, it's not unlikely that someone is going to use a NavigationView inside a standard bottom sheet at some point. While the spec doesn't explicitly prohibit that, it's clearly not the intention, so I wouldn't consider this case in the implementation.

With that in mind, I think using BottomSheetBehavior.from(navigationView) is a good solution.

Fixed in https://android-review.googlesource.com/c/platform/frameworks/support/+/743749