Status Update
Comments
bl...@google.com <bl...@google.com> #2
Kotlin 1.3.5.0
lifecyle = 2.2.0-rc02
implementation "androidx.lifecycle:lifecycle-viewmodel-ktx:${versions.lifecycle}"
implementation "androidx.lifecycle:lifecycle-extensions:${versions.lifecycle}"
implementation "androidx.lifecycle:lifecycle-common-java8:${versions.lifecycle}"
class VideoModel(val app: Application) : AndroidViewModel(app) {
// do something
}
dh...@google.com <dh...@google.com> #3
This is lifecyle-viewmodel-kts build dependency, where lifecycle-viewmodel is added (as API).
This is lifecycle-viewmodel build, and there proguard rule is applied to all build types.
and finally, this is the configuration:
```
-keepclassmembers,allowobfuscation class * extends androidx.lifecycle.AndroidViewModel {
<init>(android.app.Application);
}
```
Because of that rule, at least instance initializer of your class that extends AndroidViewModel would be kept as long as that class itself is used/kept somehow. That's one notable difference between `-keepclassmemebers` v.s. other keep directives such as `-keep` or `-keepclasseswithmembers`.
But those rules are coming from the library, so I assume it's been there for a while (though you're using lifecycle 2.2.0-rc02, is it something you recently upgraded?) Then, the difference we should track is why `VideoModel` was kept at AGP 3.5.2 and is not now at AGP 3.6.0-beta04.
Could you add the following rule to your `proguard-rules.pro` and build your app again with AGP 3.5.2?
```
-whyareyoukeeping class **.VideoModel
```
To make sure rules from lifecycle-viewmodel is included as well, could you also add the following to literally print all configurations?
```
-printconfiguration all-configurations.txt
```
If you think it reveals too much details of your apps, you can send it to jsjeon@google.com privately, instead of posting it here. Thanks.
se...@google.com <se...@google.com> #4
I added printconfiguration all-configurations.txt then compare on AS all-configurations-3.5.2.txt vs all-configurations-3.6.0-beta04.txt. Proguard config of ViewModel was consumed.
The different:
all-configurations-3.6.0-beta04.txt
# When editing this file, update the following files as well:
# - META-INF/com.android.tools/proguard/
# - META-INF/proguard/
all-configurations-3.5.2.txt
# Files in this directory will be ignored starting with Android Gradle Plugin 3.6.0+
# When editing this file, update the following files as well for AGP 3.6.0+:
# - META-INF/com.android.tools/proguard/
# - META-INF/com.android.tools/r8-upto-1.6.0/
and all-configurations-3.6.0-beta04.txt missed that was added in all-configurations-3.5.2.txt.
This the part missing:
# This is a configuration file for ProGuard.
#
#
# Starting with version 2.2 of the Android plugin for Gradle, this file is distributed together with
# the plugin and unpacked at build-time. The files in $ANDROID_HOME are no longer maintained and
# will be ignored by new version of the Android plugin for Gradle.
# Optimization is turned off by default. Dex does not like code run
# through the ProGuard optimize steps (and performs some
# of these optimizations on its own).
# Note that if you want to enable optimization, you cannot just
# include optimization flags in your own project configuration file;
# instead you will need to point to the
# "proguard-android-optimize.txt" file instead of this one from your
# project.properties file.
-dontoptimize
-dontusemixedcaseclassnames
-dontskipnonpubliclibraryclasses
-verbose
# Preserve some attributes that may be required for reflection.
-keepattributes *Annotation*,Signature,InnerClasses,EnclosingMethod
-keep public class com.google.vending.licensing.ILicensingService
-keep public class com.android.vending.licensing.ILicensingService
-keep public class com.google.android.vending.licensing.ILicensingService
-dontnote com.android.vending.licensing.ILicensingService
-dontnote com.google.vending.licensing.ILicensingService
-dontnote com.google.android.vending.licensing.ILicensingService
# For native methods, see
-keepclasseswithmembernames class * {
native <methods>;
}
# Keep setters in Views so that animations can still work.
-keepclassmembers public class * extends android.view.View {
void set*(***);
*** get*();
}
# We want to keep methods in Activity that could be used in the XML attribute onClick.
-keepclassmembers class * extends android.app.Activity {
public void *(android.view.View);
}
# For enumeration classes, see
-keepclassmembers enum * {
public static **[] values();
public static ** valueOf(java.lang.String);
}
-keepclassmembers class * implements android.os.Parcelable {
public static final ** CREATOR;
}
-keepclassmembers class **.R$* {
public static <fields>;
}
# Preserve annotated Javascript interface methods.
-keepclassmembers class * {
@android.webkit.JavascriptInterface <methods>;
}
# The support libraries contains references to newer platform versions.
# Don't warn about those in case this app is linking against an older
# platform version. We know about them, and they are safe.
-dontnote android.support.**
-dontnote androidx.**
-dontwarn android.support.**
-dontwarn androidx.**
# This class is deprecated, but remains for backward compatibility.
-dontwarn android.util.FloatMath
# Understand the @Keep support annotation.
-keep class android.support.annotation.Keep
-keep class androidx.annotation.Keep
-keep @android.support.annotation.Keep class * {*;}
-keep @androidx.annotation.Keep class * {*;}
-keepclasseswithmembers class * {
@android.support.annotation.Keep <methods>;
}
-keepclasseswithmembers class * {
@androidx.annotation.Keep <methods>;
}
-keepclasseswithmembers class * {
@android.support.annotation.Keep <fields>;
}
-keepclasseswithmembers class * {
@androidx.annotation.Keep <fields>;
}
-keepclasseswithmembers class * {
@android.support.annotation.Keep <init>(...);
}
-keepclasseswithmembers class * {
@androidx.annotation.Keep <init>(...);
}
# These classes are duplicated between android.jar and org.apache.http.legacy.jar.
-dontnote org.apache.http.**
-dontnote android.net.http.**
# These classes are duplicated between android.jar and core-lambda-stubs.jar.
-dontnote java.lang.invoke.**
I have another project that has the same config but it hadn't the problem with AGP 3.6.0-beta04. I will try to investigate more.
I think it can the same this problem before:
Below is the full log
E/TAG: java.lang.RuntimeException: Cannot create an instance of class com.myapp.screenrecorder.taskmedia.video.s
at androidx.lifecycle.M.a(ViewModelProvider.java:9)
at androidx.lifecycle.I.a(SavedStateViewModelFactory.java:4)
at androidx.lifecycle.Q.a(ViewModelProvider.java:8)
at com.myapp.screenrecorder.taskmedia.video.b.invoke(com.android.tools.r8.jetbrains.kotlin-style lambda group:1)
at kotlin.l.getValue(Lazy.kt:2)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.e(VideoFragment.kt:1)
at com.myapp.screenrecorder.taskmedia.video.g.c(VideoFragment.kt:11)
at kotlin.coroutines.jvm.internal.a.a(ContinuationImpl.kt:3)
at kotlinx.coroutines.K.a(Dispatched.kt:18)
at com.myapp.simplefilechooser.g.a(Unknown Source:20)
at kotlinx.coroutines.a.a(AbstractCoroutine.kt:21)
at kotlinx.coroutines.d.a(Unknown Source:16)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.d0(VideoFragment.kt:1)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.a(VideoFragment.kt:18)
at androidx.fragment.app.T.a(FragmentStateManager.java:51)
at androidx.fragment.app.K.a(FragmentManager.java:176)
at androidx.fragment.app.K.a(FragmentManager.java:426)
at androidx.fragment.app.K.a(FragmentManager.java:373)
at androidx.fragment.app.K.c(FragmentManager.java:38)
at androidx.fragment.app.K.c(FragmentManager.java:22)
at androidx.fragment.app.K.b(FragmentManager.java:49)
at androidx.fragment.app.K.c(FragmentManager.java:42)
at androidx.fragment.app.k.e(Fragment.java:6)
at androidx.fragment.app.T.a(FragmentStateManager.java:56)
at androidx.fragment.app.K.a(FragmentManager.java:177)
at androidx.fragment.app.K.h(FragmentManager.java:4)
at androidx.fragment.app.K.a(FragmentManager.java:257)
at androidx.fragment.app.K.b(FragmentManager.java:47)
at androidx.fragment.app.K.c(FragmentManager.java:42)
at androidx.fragment.app.w.a(FragmentController.java:9)
at androidx.fragment.app.m.onStart(FragmentActivity.java:5)
at androidx.appcompat.app.v.onStart(AppCompatActivity.java:1)
at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1392)
at android.app.Activity.performStart(Activity.java:7157)
at android.app.ActivityThread.handleStartActivity(ActivityThread.java:2952)
at android.app.servertransaction.TransactionExecutor.performLifecycleSequence(TransactionExecutor.java:180)
at android.app.servertransaction.TransactionExecutor.cycleToPath(TransactionExecutor.java:165)
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:142)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:70)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1823)
at android.os.Handler.dispatchMessage(Handler.java:107)
at android.os.Looper.loop(Looper.java:198)
at android.app.ActivityThread.main(ActivityThread.java:6729)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
Caused by: java.lang.NoSuchMethodException: <init> [class android.app.Application]
at java.lang.Class.getConstructor0(Class.java:2327)
at java.lang.Class.getConstructor(Class.java:1725)
at androidx.lifecycle.M.a(ViewModelProvider.java:5)
at androidx.lifecycle.I.a(SavedStateViewModelFactory.java:4)
at androidx.lifecycle.Q.a(ViewModelProvider.java:8)
at com.myapp.screenrecorder.taskmedia.video.b.invoke(com.android.tools.r8.jetbrains.kotlin-style lambda group:1)
at kotlin.l.getValue(Lazy.kt:2)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.e(VideoFragment.kt:1)
at com.myapp.screenrecorder.taskmedia.video.g.c(VideoFragment.kt:11)
at kotlin.coroutines.jvm.internal.a.a(ContinuationImpl.kt:3)
at kotlinx.coroutines.K.a(Dispatched.kt:18)
at com.myapp.simplefilechooser.g.a(Unknown Source:20)
at kotlinx.coroutines.a.a(AbstractCoroutine.kt:21)
at kotlinx.coroutines.d.a(Unknown Source:16)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.d0(VideoFragment.kt:1)
at com.myapp.screenrecorder.taskmedia.video.VideoFragment.a(VideoFragment.kt:18)
at androidx.fragment.app.T.a(FragmentStateManager.java:51)
at androidx.fragment.app.K.a(FragmentManager.java:176)
at androidx.fragment.app.K.a(FragmentManager.java:426)
at androidx.fragment.app.K.a(FragmentManager.java:373)
at androidx.fragment.app.K.c(FragmentManager.java:38)
at androidx.fragment.app.K.c(FragmentManager.java:22)
at androidx.fragment.app.K.b(FragmentManager.java:49)
at androidx.fragment.app.K.c(FragmentManager.java:42)
at androidx.fragment.app.k.e(Fragment.java:6)
at androidx.fragment.app.T.a(FragmentStateManager.java:56)
at androidx.fragment.app.K.a(FragmentManager.java:177)
at androidx.fragment.app.K.h(FragmentManager.java:4)
at androidx.fragment.app.K.a(FragmentManager.java:257)
at androidx.fragment.app.K.b(FragmentManager.java:47)
at androidx.fragment.app.K.c(FragmentManager.java:42)
at androidx.fragment.app.w.a(FragmentController.java:9)
at androidx.fragment.app.m.onStart(FragmentActivity.java:5)
at androidx.appcompat.app.v.onStart(AppCompatActivity.java:1)
at android.app.Instrumentation.callActivityOnStart(Instrumentation.java:1392)
at android.app.Activity.performStart(Activity.java:7157)
at android.app.ActivityThread.handleStartActivity(ActivityThread.java:2952)
at android.app.servertransaction.TransactionExecutor.performLifecycleSequence(TransactionExecutor.java:180)
at android.app.servertransaction.TransactionExecutor.cycleToPath(TransactionExecutor.java:165)
at android.app.servertransaction.TransactionExecutor.executeLifecycleState(TransactionExecutor.java:142)
at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:70)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1823)
at android.os.Handler.dispatchMessage(Handler.java:107)
at android.os.Looper.loop(Looper.java:198)
at android.app.ActivityThread.main(ActivityThread.java:6729)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)
ma...@google.com <ma...@google.com> #5
mi...@google.com <mi...@google.com> #6
The finally file size is 4,179KB. I don't know why before file size is 3804KB, maybe the cause is the part missing of proguard.
Thank you!
dh...@google.com <dh...@google.com> #7
As jsjeon@google.com mentioned in
-keepclassmembers,allowobfuscation class * extends androidx.lifecycle.AndroidViewModel {
<init>(android.app.Application);
}
should come from the aar, but it does not show up in your -printconfiguration output, which seems strange.
To test I created a "Basic Activity" app, and added
dependencies {
implementation "androidx.lifecycle:lifecycle-viewmodel-ktx:2.2.0-rc02"
implementation "androidx.lifecycle:lifecycle-extensions:2.2.0-rc02"
implementation "androidx.lifecycle:lifecycle-common-java8:2.2.0-rc02"
}
and when I add -printconfiguration the rule above is in the file - you should see that as well.
It would be interesting to understand why this does not work. I am not that familiar with androidx.lifecycle, but I don't think you should need the rule '-keep class **VideoModel'. I wonder if you could try the following without that rule:
1. Add the rule '-whyareyoukeeping class **VideoModel' and build release. The output from -whyareyoukeeping goes to stdout for the build.
2. Look at the mapping file app/build/outputs/mapping/release/mapping.txt for the build from 1), and look for you ViewModel class.
3. If the above still crash at runtime try the rule '-keep,allowobfuscation class **VideoModel'
4. Look at the mapping file app/build/outputs/mapping/release/mapping.txt for the build from 3), and look for you ViewModel class.
je...@google.com <je...@google.com> #8
I build all cases with AGP 3.6.0-beta04 and this the result:
-keepclassmembers,allowobfuscation class * extends androidx.lifecycle.AndroidViewModel {
<init>(android.app.Application);
}
or '-whyareyoukeeping class **VideoModel
But it still crashed at runtime because it missed this in the mapping.txt
android.app.Application app -> c
1:1:void <init>(android.app.Application):17:17 -> <init>
.........................................................
When adding rule '-keep,allowobfuscation class **VideoModel'. It worked fine because it added full.
db...@google.com <db...@google.com> #9
So, again, from
> 1. Add the rule '-whyareyoukeeping class **VideoModel' and build release. The output from -whyareyoukeeping goes to stdout for the build.
This actually meant, building with AGP 3.5.2, not AGP 3.6.0-beta04 (because we knew beta version is not keeping something on that hierarchy).
> 3. If the above still crash at runtime try the rule '-keep,allowobfuscation class **VideoModel'
But, this one meant AGP 3.6.0-beta04, in order to see if renaming matters v.s. it's literally missing to trace fundamental components. And, that's what you already tested, thanks.
------
But... you can hold on doing another build and experiment, since I feel like we're chasing a wrong target. The reason we kept asking you to chase why your **.ViewModel was kept or not was, mainly because that seems like the one we're missing according to
E/TAG: java.lang.RuntimeException: Cannot create an instance of class com.myapp.screenrecorder.taskmedia.video.s
at androidx.lifecycle.M.a(ViewModelProvider.java:9)
at androidx.lifecycle.I.a(SavedStateViewModelFactory.java:4)
at androidx.lifecycle.Q.a(ViewModelProvider.java:8)
and the other is:
Caused by: java.lang.NoSuchMethodException: <init> [class android.app.Application]
at java.lang.Class.getConstructor0(Class.java:2327)
at java.lang.Class.getConstructor(Class.java:1725)
at androidx.lifecycle.M.a(ViewModelProvider.java:5)
Indeed, there are many reflective instance creations in ViewModelProvider, e.g.,
------
More interestingly, according to
which I believe came from:
Not seeing `-dontoptimize` at 3.6.0-beta04.txt means it's using
Maybe that came from other dependent libraries. Then, I guess the issue was just hidden even with AGP 3.5.2
------
According to
class MyActivity : AppCompatActivity() {
override fun onCreate(savedInstanceState: Bundle?) {
// Create a ViewModel the first time the system calls an activity's onCreate() method.
// Re-created activities receive the same MyViewModel instance created by the first activity.
val model = ViewModelProviders.of(this)[MyViewModel::class.java]
model.getUsers().observe(this, Observer<List<User>>{ users ->
// update UI
})
}
}
and there, type reference, e.g., MyViewModel::class.java, made that type being kept, and then trigger -keepclassmembers rules with <init>. That's how ViewModelProvider can reflectively instantiate users' view model. Could you let us know how your **.VideoModel is used?
je...@google.com <je...@google.com> #10
ViewModelProviders.of is deprecated so I am using
class VideoFragment : Fragment() {
private val videoModel: VideoModel by lazy(LazyThreadSafetyMode.NONE) {
ViewModelProvider(this).get(VideoModel::class.java)
}
............................
private fun loadVideos() {
val videos = videoModel.getVideos()
}
}
class VideoModel(val app: Application) : AndroidViewModel(app) {
suspend fun getVideos(): List<MediaItem> = withContext(ioDispatcher) {
val appSettings = AppSettings.getInstance(app)
if (appSettings.locationStorageType == 0) {
traverseInternalStorage(appSettings.videoDirectory)
} else {
traverseRemovableStorage(appSettings.externalStorageUri?.toUri())
}
}
..................
}
My config in build.gradle
buildTypes {
release {
minifyEnabled true
shrinkResources true
proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
manifestPlaceholders += [crashlyticsEnabled: true]
}
}
gradle.properties
org.gradle.parallel=true
org.gradle.caching=true
android.enableJetifier=true
android.useAndroidX=true
android.enableR8.fullMode=true
android.debug.obsoleteApi=true
kapt.incremental.apt=true
Now my proguard-rules.pro config
-keep public class * implements com.bumptech.glide.module.GlideModule
-keep public class * extends com.bumptech.glide.AppGlideModule
-keep public enum com.bumptech.glide.load.resource.bitmap.ImageHeaderParser$** {
**[] $VALUES;
public *;
}
-keep class com.coremedia.iso.** { *; }
-keep class com.googlecode.mp4parser.** { *; }
-keep class com.mp4parser.** { *; }
-keep class org.mp4parser.aspectj.internal.** { *; }
-keep class org.mp4parser.aspectj.lang.** { *; }
-keep class org.mp4parser.aspectj.runtime.** { *; }
-dontwarn com.googlecode.mp4parser.authoring.tracks.mjpeg.OneJpegPerIframe
# For Kotlin coroutines
# ServiceLoader support
-keepnames class kotlinx.coroutines.internal.MainDispatcherFactory {}
-keepnames class kotlinx.coroutines.CoroutineExceptionHandler {}
-keepnames class kotlinx.coroutines.android.AndroidExceptionPreHandler {}
-keepnames class kotlinx.coroutines.android.AndroidDispatcherFactory {}
# Most of volatile fields are updated with AFU and should not be mangled
-keepclassmembernames class kotlinx.** {
volatile <fields>;
}
#
-keep @interface kotlin.coroutines.jvm.internal.** { *; }
# In-app
-keep class com.android.vending.billing.**
-keepattributes *Annotation*
-keepattributes SourceFile,LineNumberTable
-keep public class * extends java.lang.Exception
# Avoid runtime app crash
-keep,allowobfuscation **VideoModel
AGP 3.5.2 I didn't added any rule for VideoModel it worked fine.
AGP 3.6.0-beta04 I need to add rule -keep,allowobfuscation **VideoModel to the app doesn't error at runtime.
I have another project it using the AndroidViewModel and same code but when build with AGP 3.6.0-beta04, it worked as AGP 3.5.2 (didn't need to add a proguard rule for ViewModel).
I think it had a problem for my project. Because this didn't happen on all project.
Thank for your supporting!
cj...@google.com <cj...@google.com> #11
ma...@google.com <ma...@google.com> #12
Thanks!
se...@google.com <se...@google.com> #13
se...@google.com <se...@google.com> #14
Actually, my soluation at
I will remove the comment.
ma...@google.com <ma...@google.com> #15
jr...@google.com <jr...@google.com> #16
Copied over from Chrome OS Chatty Eng per seewaifu@'s request.
fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects
error: Cannot fetch chromiumos/platform/ec from https://chromium.googlesource.com/chromiumos/platform/ec
Hit the above error again in a different checkout and confirmed the following fixed unblocked me (after backing up any local changed, of course):
rm -rf .repo/project-objects/chromiumos/platform/ec.git
rm -rf .repo/projects/src/platform/ec.git
repo sync src/platform/ec
This sounds like what manojgupta@ did in
mu...@google.com <mu...@google.com> #17
jo...@google.com <jo...@google.com> #18
jo...@google.com <jo...@google.com> #19
jr...@google.com <jr...@google.com> #20
100% agree with you, jorgelo@. I keep hitting this as well...
# EDIT: These commands didn't fix anything. Another repo sync reverted the local git repo changes.
ib...@google.com <ib...@google.com> #21
sync is pretty complete disaster for me as well:
fatal: bad object refs/remotes/cros/intel-wifi/v4.14-Core31-merge
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/kernel did not send all necessary objects
error: Cannot fetch chromiumos/third_party/kernel from https://chromium.googlesource.com/chromiumos/third_party/kernel
fatal: bad object refs/tags/v2.6.38.1
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/kernel did not send all necessary objects
error: Cannot fetch chromiumos/third_party/kernel from https://chromium.googlesource.com/chromiumos/third_party/kernel
fatal: bad object refs/remotes/cros/upstream/dri3-auto
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/mesa did not send all necessary objects
error: Cannot fetch chromiumos/third_party/mesa from https://chromium.googlesource.com/chromiumos/third_party/mesa
Checking out: 4% (45/1028) external/gob/nest-internal/platform/networkmanager-linuxerror: Cannot checkout chromiumos/third_party/kernel: ManifestInvalidRevisionError: revision refs/heads/merge/continuous/chromeos-kernelupstream-5.15-rc7 in chromiumos/third_party/kernel not found
error: in `sync`: revision refs/heads/merge/continuous/chromeos-kernelupstream-5.15-rc7 in chromiumos/third_party/kernel not found
I have also not sync'ed my repo for a long period of time (weeks? months?). I really think the age of the repo / time-since-last-sync is related here.
Fortunately, I don't have branches I need to keep, so erase-the-whole-thing-and-restart is an option for me.
jo...@google.com <jo...@google.com> #23
mu...@google.com <mu...@google.com> #24
https://paste.googleplex.com/5867083372953600 worked for me.
Not for me, unfortunately. I still see numerous failures.
error: Cannot fetch aosp/platform/system/core/libutils from https://chromium.googlesource.com/aosp/platform/system/core/libutils
remote: Total 0 (delta 0), reused 0 (delta 0)
fatal: bad object ec19e4ac7f06ca452c63e09619918a8a628ce5e3
error: persistent-https://chrome-internal.git.corp.google.com/chromeos/project/jacuzzi/kappa did not send all necessary objects
...
chromeos/project/octopus/fleex:
remote: Total 0 (delta 0), reused 0 (delta 0)
fatal: bad object ec5322a01bd2fb44d6a3985f914f191b53ed29ff
error: persistent-https://chrome-internal.git.corp.google.com/chromeos/project/octopus/fleex did not send all necessary objects
...
remote: Total 6 (delta 2), reused 6 (delta 2)
fatal: bad object ecbfa0ba4ac0bcce20ff52bdfd92ea667b38c658
error: persistent-https://chromium.git.corp.google.com/chromiumos/platform/touch_updater did not send all necessary objects
and many more
jo...@google.com <jo...@google.com> #25
ib...@google.com <ib...@google.com> #26
Didn't work for me either:
iby@iby3:~/chromiumos/src$ repo forall -v -c git clean-refs
[skipping the no-delete repos]
/ssd/iby/chromiumos/src/platform/cr50
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/platform/ec
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/platform/release-firmware/fpmcu-bloonchipper
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/platform/release-firmware/fpmcu-dartmonkey
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/platform/release-firmware/fpmcu-nami
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/platform/release-firmware/fpmcu-nocturne
Deleting stale ref: refs/remotes/cros/firmware-zork-13421.B-master
/ssd/iby/chromiumos/src/third_party/intel-wifi-fw-dump
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
/ssd/iby/chromiumos/src/third_party/mesa
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
/ssd/iby/chromiumos/src/third_party/mesa-amd
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
/ssd/iby/chromiumos/src/third_party/mesa-debian
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
/ssd/iby/chromiumos/src/third_party/mesa-freedreno
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
/ssd/iby/chromiumos/src/third_party/mesa-img
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
/ssd/iby/chromiumos/src/third_party/mesa-iris
Deleting stale ref: refs/remotes/cros/upstream/dri3-auto
iby@iby3:~/chromiumos/src$ repo sync -j 20
[stuff]
remote: Total 104969 (delta 85267), reused 104886 (delta 85267)
fatal: bad object refs/tags/v2.6.38.1
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/kernel did not send all necessary objects
error: Cannot fetch chromiumos/third_party/kernel from https://chromium.googlesource.com/chromiumos/third_party/kernel
[stuff]
remote: Total 104969 (delta 85267), reused 104886 (delta 85267)
fatal: bad object refs/remotes/cros/intel-wifi/v4.14-Core31-merge
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/kernel did not send all necessary objects
error: Cannot fetch chromiumos/third_party/kernel from https://chromium.googlesource.com/chromiumos/third_party/kernel
[stuff]
Garbage collecting: 100% (12/12), done in 0.080s
Checking out: 2% (28/1028) chromeos/vendor/verisilicon/falconlite-kernel-v5.4error: Cannot checkout chromiumos/third_party/kernel: ManifestInvalidRevisionError: revision refs/heads/merge/continuous/chromeos-kernelupstream-5.15-rc7 in chromiumos/third_party/kernel not found
Checking out: 84% (864/1028), done in 4.118s
error: in `sync -j 20`: revision refs/heads/merge/continuous/chromeos-kernelupstream-5.15-rc7 in chromiumos/third_party/kernel not found
Note: For the not a git repository
error: The intel-wifi-fw-dump
was one of the directories I removed an hour or two ago. I removed it, redid the repo sync
, tried your script, redid repo sync
with the above errors. SO, maybe the intermediate repo sync just didn't refetch it due to other errors?
ib...@google.com <ib...@google.com> #27
What's the easiest way to do an erase-the-whole-repo-and-start-over?
I remember there are warning about not trying to delete chroot with rm
.
jo...@google.com <jo...@google.com> #28
ib...@google.com <ib...@google.com> #29
In the end, I just recreated the whole thing:
cros_sdk --delete
cd ..
sudo rm -rf chromiumos
mkdir chromiumos
cd chromiumos
repo init -u https://chrome-internal.googlesource.com/chromeos/manifest-internal
repo sync -j4
And the errors went away
cj...@google.com <cj...@google.com> #30
I don't have a great answer for a path forward here; reassigning to the current oncall.
an...@google.com <an...@google.com> #31
Agree that this is probably related to the age of the checkout. On a cloudtop (newer checkout), I do not get this error, on my desktop (older checkout), I do get this error:
$ repo sync chromiumos/platform/ec
...
remote: Finding sources: 99% (10139/10241)
remote: Finding sources: 100% (10241/10241)
remote: Total 10241 (delta 6635), reused 10232 (delta 6635)
fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects
error: Cannot fetch chromiumos/platform/ec from https://chromium.googlesource.com/chromiumos/platform/ec
I'll do a bit more digging on why this happens.
an...@google.com <an...@google.com> #32
I remember that the zork fw branch was changed at some point. It is now firmware-zork-13434.B-master
on the remote, not firmware-zork-13421.B-master
, as appears locally. So deleting refs on the remote probably causes problems locally.
Remote only has 13434 (
$ git branch -a | grep firmware-zork
remotes/cros/firmware-zork-13421.B-cr50_stab
remotes/cros/firmware-zork-13421.B-master
remotes/cros/firmware-zork-13434.B-cr50_stab
remotes/cros/firmware-zork-13434.B-master
an...@google.com <an...@google.com> #33
Running repo sync with the --prune
tag fixes for me:
$ repo sync chromiumos/platform/ec --prune
Fetching: 100% (7/7), done in 5.914s
src/platform/cr50: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/ec: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/ish: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/release-firmware/fpmcu-bloonchipper: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/release-firmware/fpmcu-dartmonkey: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/release-firmware/fpmcu-nami: Shared project chromiumos/platform/ec found, disabling pruning.
src/platform/release-firmware/fpmcu-nocturne: Shared project chromiumos/platform/ec found, disabling pruning.
Garbage collecting: 100% (7/7), done in 0.047s
src/third_party/zephyr/main/v2.6: Deleting obsolete checkout.
Checking out: 100% (7/7), done in 1.422s
repo sync has finished successfully.
A bit more documentation on the tag.
- From
repo sync -h
:--prune delete refs that no longer exist on the remote
. - Also docs on the
git fetch
page (I assumerepo sync
is basically calling this):https://git-scm.com/docs/git-fetch .
As expected, the old remote ref is deleted after:
$ git branch -a | grep firmware-zork
remotes/cros/firmware-zork-13434.B-cr50_stab
remotes/cros/firmware-zork-13434.B-master
an...@google.com <an...@google.com> #34
I'm not sure if this will fix every type of error described in this bug and related threads, but it at least seems to at least fix the fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error on a repo for me, and should fix related errors where a remote ref no longer exists.
I'd advise reading
jo...@google.com <jo...@google.com> #35
The collective time lost by this is probably in the SWE-weeks at this point.
an...@google.com <an...@google.com> #36
cc'ing current git admin oncalls.
Can only git admins delete a remote ref? We could only allow deleting a ref if strictly necessary (probably already the policy), and include a PSA in the playbook.
ju...@google.com <ju...@google.com> #37
I believe only git admins can and no one else. Adding ajp@ incase he knows of anyone else who can do this. The
va...@google.com <va...@google.com> #38
that's not accurate for CrOS repos. we have branch creator powers, and those powers also allow branch deletion.
i didn't think git fetch
failed like this when remote deleted a ref. but looks like repo is just reflecting the git behavior.
$ cd src/platform/ec
$ git fetch cros
remote: Total 6907 (delta 4643), reused 6888 (delta 4643)
Receiving objects: 100% (6907/6907), 3.37 MiB | 26.96 MiB/s, done.
Resolving deltas: 100% (4643/4643), completed with 1455 local objects.
fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error: https://chromium.googlesource.com/chromiumos/platform/ec did not send all necessary objects
i'm somewhat inclined to have repo use --prune
behavior by default. really seems like the more seasonable default.
an...@google.com <an...@google.com> #39
I think it could also be set in the global git config repo
?
If --prune
is never destructive, enabling it by default or suggesting it when this error occurs seems reasonable.
va...@google.com <va...@google.com> #40
users are free to set fetch.prune in their ~/.gitconfig
if they want, but repo will never touch those files by design. those are user config settings and repo is only allowed to touch things in its local repo client checkout.
i don't know that i characterize --prune
as not destructive, but i think the majority of repo users just don't care about the nuances. they're working with a local source state so that they can push changes to the remote, and random server refs don't matter much in this mode.
i think this would help people recover from issues we've had in the past where refs were incompatible with filesystem paths. e.g. refs/heads/foo
changing to refs/heads/foo/bar
.
mu...@google.com <mu...@google.com> #41
--prune
is not a panacea here; in particular it hasn't resolved the issue for me for (at least) chromeos/third_party/intel-fsp/edk2-platforms
. I keep seeing the following, even after running repo sync
on this repo with --prune
:
chromeos/third_party/intel-fsp/edk2-platforms:
remote: Total 0 (delta 0), reused 0 (delta 0)
fatal: bad object ec1372de107a5a7b1852b6e2ff1d01cde6dcf271
error: persistent-https://chrome-internal.git.corp.google.com/chromeos/third_party/intel-fsp/edk2-platforms did not send all necessary objects
edit: this is when running repo sync --prune chromeos/third_party/intel-fsp/edk2-platforms
va...@google.com <va...@google.com> #42
i suspect there is more than one bug that ends up with similar errors, so going to focus on one at a time
mu...@google.com <mu...@google.com> #43
Is there another bug I can follow for the other cause(s)? Despite syncing all repos with --prune
, I still get this error on a very large number of repos
va...@google.com <va...@google.com> #44
feel free to file a new one with -j1 logs after running --prune
i've pushed --prune default to staging for CrOS, but since it's Friday, it'll wait until Monday to push to stable
mi...@google.com <mi...@google.com> #45
Please do hold off as I believe the last time we tested using --prune
it broke the cache builders. I haven't pursued it much further as we resolved the underlying issues with builds, but I want to ensure we are not breaking builders to fix local repos.
-- Mike
mu...@google.com <mu...@google.com> #46
feel free to file a new one with -j1 logs after running --prune
Thanks, filed
rr...@google.com <rr...@google.com> #47
Try my script again:
I added the bad ref
pattern to sed. I didn't actually run into the bad object
error so I haven't tested it.
mu...@google.com <mu...@google.com> #48
EDIT: This was wrong; the new clean-refs didn't actually do anything. See
(00:00)(12:20:56) (dbus-feature) mutexlox@deadlox:/chromium_work/src/platform2/featured$ repo forall -v -c git clean-refs
<snipped large output>
(00:09)(12:21:13) (dbus-feature) mutexlox@deadlox:/chromium_work/src/platform2/featured$ pushd ../../third_party/adhd
/chromium_work/src/third_party/adhd /chromium_work/src/platform2/featured
(00:00)(12:21:16) ((8a1a1308...)) mutexlox@deadlox:/chromium_work/src/third_party/adhd$ git fetch --all -v
Fetching cros
POST git-upload-pack (102 bytes)
POST git-upload-pack (gzip 1066 to 579 bytes)
POST git-upload-pack (gzip 2666 to 1367 bytes)
POST git-upload-pack (gzip 5866 to 2974 bytes)
remote: Total 37 (delta 11), reused 32 (delta 11)
Unpacking objects: 100% (37/37), 47.66 KiB | 1.36 MiB/s, done.
fatal: bad object ec7ced6904430fc36873ee87cd2a70e60b24358b
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/adhd did not send all necessary objects
error: Could not fetch cros
(1) (00:02)(12:21:20) ((8a1a1308...)) mutexlox@deadlox:/chromium_work/src/third_party/adhd$ git clean-refs
/chromium_work/src/third_party/adhd
(00:00)(12:21:43) ((8a1a1308...)) mutexlox@deadlox:/chromium_work/src/third_party/adhd$ git fetch --all -v
Fetching cros
POST git-upload-pack (102 bytes)
POST git-upload-pack (gzip 1066 to 579 bytes)
POST git-upload-pack (gzip 2666 to 1367 bytes)
POST git-upload-pack (gzip 5866 to 2974 bytes)
remote: Total 37 (delta 11), reused 32 (delta 11)
Unpacking objects: 100% (37/37), 47.66 KiB | 1.08 MiB/s, done.
fatal: bad object ec7ced6904430fc36873ee87cd2a70e60b24358b
error: persistent-https://chromium.git.corp.google.com/chromiumos/third_party/adhd did not send all necessary objects
error: Could not fetch cros
mu...@google.com <mu...@google.com> #49
It seems like that ref isn't listed at all in git show-ref
, so the script won't do anything:
$ git show-ref | grep ec7ced6904430fc36873ee87cd2a70e60b24358b
(1) (00:01)(12:26:43) ((8a1a1308...)) mutexlox@deadlox:/chromium_work/src/third_party/adhd$
rr...@google.com <rr...@google.com> #50
Ah, that is a different failure case. That one legit looks like GoB didn't send the object.
On my system:
~/chromiumos/src/third_party/adhd$ git log ec7ced6904430fc36873ee87cd2a70e60b24358b --oneline | head
ec7ced69 CRAS: audio_thread - Log the non-empty state in sleep event
53b4d7d3 CRAS: iodev_list - Adjust input ui gain range
27be363f CRAS: hfp_info - Support 24 bytes HCI SCO packet
va...@google.com <va...@google.com> #51
i think the most important part is that the object is reachable. and it looks like it should be -- it's in main among many others.
but it's been in there for over a year, so why doesn't Miriam's local repo have this ? was it created with shallow/depth at some point ? does git log m/main
work ?
mu...@google.com <mu...@google.com> #52
was it created with shallow/depth at some point ?
not afaik
does git log m/main work ?
No. It prints a good number but then it gets to a bad commit and blows up:
$ git log m/main --oneline
<snip>
e469a74d (cros/stabilize-14189.B) sound_card_init/cras_tests: Fix PlaybackBufferStream stucks
ff7ca87e sound_card_init: remove Cargo.lock
e02bb938 cras: Add async API
error: Could not read ecdf9184b9fb779a50b0290768efd2ed1ca37d43
fatal: Failed to traverse parents of commit 294b50a2044964e5cca01f6585fdcc01ce66adaa
mu...@google.com <mu...@google.com> #53
Should we move discussion of this failure mode into
Description
fatal: bad object refs/remotes/cros/firmware-zork-13421.B-master
error:
error: Cannot fetch chromiumos/platform/ec from
fatal: bad object refs/tags/v2.6.38.1
error:
error: Cannot fetch chromiumos/third_party/kernel from
fatal: bad object refs/remotes/cros/upstream/dri3-auto
error:
error: Cannot fetch chromiumos/third_party/mesa from