IssueTracker

How should this work "else if (parent instanceof BottomSheetDialogFragment)"?

Yeah, I just realized that it's not as simple as in the case of the DrawerLayout. When using a DrawerLayout, people mostly have something like this:

<android.support.v4.widget.DrawerLayout xmlns:android="http://schemas.android.com/apk/res/android"
                                        xmlns:app="http://schemas.android.com/apk/res-auto"
                                        android:id="@+id/drawer_layout"
                                        android:layout_width="match_parent"
                                        android:layout_height="match_parent"
                                        android:fitsSystemWindows="true">

    <FrameLayout
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:id="@+id/header_container"/>

    <android.support.design.widget.NavigationView
        android:id="@+id/nav_view"
        android:layout_width="wrap_content"
        android:layout_height="match_parent"
        android:layout_gravity="start"
        app:headerLayout="@layout/main_nav_header"
        android:fitsSystemWindows="true"
        app:menu="@menu/main_drawercontent"/>

</android.support.v4.widget.DrawerLayout>

Obviously here the NavigationView is a direct child of the DrawerLayout, which is why the implementation we already have for handling the closing of the navigation drawer works like it does, by checking the NavigationView's parent.
But when using a BottomSheetDialogFragment, we can't rely on the NavigationView being a direct child of the BottomSheetDialogFragment. Since it's mostly a regular Fragment, people may use a layout like this (simplified):

<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".view.BottomDrawerFragment">

    <!-- Here we'd have views typically found in the navigation drawer,
         such as a user profile picture or their name -->

    <com.google.android.material.navigation.NavigationView
        android:id="@+id/bottom_drawer_navigation"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="bottom"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/divider"
        app:menu="@menu/bottom_drawer_menu" />
</androidx.constraintlayout.widget.ConstraintLayout>

My suggestion wouldn't work with this, as the NavigationView's parent is not the BottomSheetDialogFragment, but the ConstraintLayout. So we can't use the parent to detect when we should dismiss a BottomSheetDialogFragment.

A potential solution would probably have to use a different approach. Maybe in setupWithNavController() you could pass a reference to the BottomSheetDialogFragment directly, in addition to the NavigationView and NavController. Something similar happens already, for example when using a Toolbar that should be kept in sync also, the method call is NavigationUI.setupWithNavController(collapsingToolbarLayout, toolbar, navController) for that.
So we could end up having an implementation of setupWithNavController just for our case with the BottomSheetDialogFragment that goes something like this:

public static void setupWithNavController(
        @NonNull BottomSheetDialogFragment bottomSheetDialogFragment,
        @NonNull final NavigationView navigationView,
        @NonNull final NavController navController) {
    navigationView.setNavigationItemSelectedListener(
            new NavigationView.OnNavigationItemSelectedListener() {
                @Override
                public boolean onNavigationItemSelected(@NonNull MenuItem item) {
                    boolean handled = onNavDestinationSelected(item, navController, true);
                    if (handled) {
                        bottomSheetDialogFragment.dismiss();
                    }
                    return handled;
                }
            });
    final WeakReference<NavigationView> weakReference = new WeakReference<>(navigationView);
    navController.addOnNavigatedListener(new NavController.OnNavigatedListener() {
        @Override
        public void onNavigated(@NonNull NavController controller,
                @NonNull NavDestination destination) {
            NavigationView view = weakReference.get();
            if (view == null) {
                controller.removeOnNavigatedListener(this);
                return;
            }
            Menu menu = view.getMenu();
            for (int h = 0, size = menu.size(); h < size; h++) {
                MenuItem item = menu.getItem(h);
                item.setChecked(matchDestination(destination, item.getItemId()));
            }
        }
    });
}

And we'd use it in our example like so:

public class BottomDrawerFragment extends BottomSheetDialogFragment {
    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) {
        View view = inflater.inflate(R.layout.fragment_bottom_drawer, container, false);
        NavigationView navigationView = view.findViewById(R.id.bottom_drawer_navigation);
        NavigationUI.setupWithNavController(this, navigationView, Navigation.findNavController(getActivity(), R.id.nav_host_fragment));
        return view;
    }
}

You nailed it man. Thanks for your investigation!

Are there any cases where you'd have a NavigationView in a BottomSheet of any type and not expect the bottom sheet to be hidden when you tap on a NavigationView item?

If you'd always expect it to be hidden, we could use BottomSheetBehavior.from(navigationView) to find the bottom sheet, avoiding the need to create another setup method.