IssueTracker

Hello,

I understand your issue is that the upload using the Python libraries to GS buckets and loading a table to BigQuery is comparatively slower than the upload using the commands gsutil and bq. Please let me know if I have misunderstood.

Let me clarify that the comparision should be done this way: Python's blob.upload_from_file() vs gsutil command and Python's load_table_from_file() vs bq command. Once that is clear, I would like to ask you for the codes that you use so I can reproduce the situation myself and get further insights. Please remove all the personal information from your codes before sharing them.

I will wait for your response,

Manuel Alaman
Google Cloud Big Data Support Barcelona

You are correct.

Attached python script will generate a test csv file and conduct the python client test. Please find and replace all occurrences of `UPDATE_THIS` text.

It also has the DDL query you'll need to use to create the BQ table before you run the script.

Additionally, it has the exact bq command you'll need to test the bq CLI utility against the same file.

I just tested again after creating this using python 3.6.9, google-cloud-bigquery 2.20.0, and BigQuery CLI 2.0.69 (most recent versions). I still see the same performance difference (~ 4MBps upload from the python client, vs ~70MBps upload for the same file to the same table using BigQuery CLI.

Let me know if you need anything else.

Hey there any update on this?

Hi Kevin,

We are still investigating the issue. At this point we obtained [1] for the script and [2] for the bq command, where the “Upload complete” was achieved in about 11 seconds.
Further updates will be published here.


[1]
2021-06-30 06:55:01,496 root                       test_uploads                  INFO: Beginning load job...
2021-06-30 06:57:08,662 root                       test_uploads                  INFO: Job ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
2021-06-30 06:57:08,662 root                       test_uploads                  INFO: BQ load job complete without error!

[2]
Upload complete.
Waiting on bqjob_XXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX_X ... (48s) Current status: DONE

Hi there, has there been any progress on this? Should I move this over to an Issue at https://github.com/googleapis/google-cloud-python?

This was investigated and has a fix pending. See the following

https://github.com/googleapis/google-resumable-media-python/issues/238
https://github.com/googleapis/python-bigquery/issues/775
https://github.com/googleapis/python-bigquery/pull/799

I am still waiting for the fix. No workaround yet.

This is now fixed in Jetifier beta01 and will be available in future version of AGP.

In the meantime you can work-around this issue by adding this to your gradle script:
buildscript {
   dependencies {
       classpath 'com.android.tools.build.jetifier:jetifier-processor:1.0.0-beta01'
   }
}

Thank you!

Sorry jetifier beta01 was not binary compatible with alpha10.

Please try:
buildscript {
   dependencies {
       classpath 'com.android.tools.build.jetifier:jetifier-processor:1.0.0-beta02'
   }
}

Just FYI this workaround only works if DataBinding is not enabled.  Simply adding the following in the app's build.gradle will throw an exception:

android {
    dataBinding {
        enabled true
    }
}

Exception:

Execution failed for task ':app:kaptGenerateStubsDebugKotlin'.
> Could not resolve all files for configuration ':app:kapt'.
   > Failed to transform file 'jetifier-core-1.0.0-alpha10.jar' to match attributes {artifactType=processed-jar} using transform JetifyTransform
      > Failed to transform '/Users/nkotula/.gradle/caches/modules-2/files-2.1/com.android.tools.build.jetifier/jetifier-core/1.0.0-alpha10/9eb7027c383061de12f93aae7a22cbeb97832d2a/jetifier-core-1.0.0-alpha10.jar' using Jetifier. Reason: The given artifact contains a string literal with a package reference 'android/support/v4' that cannot be safely rewritten. Libraries using reflection such as annotation processors need to be updated manually to add support for androidx.. (Run with --stacktrace for more details.) To disable Jetifier, set android.enableJetifier=false in your gradle.properties file.

Steps to reproduce:

 1.  Start new Android project with all the default settings (include Kotlin support)
 2.  Add the workaround from above into the top-level build.gradle file.
 3.  Sync & Rebuild the project to confirm it is working.
 4.  Enable DataBinding as described above.  Rebuilding will now fail with this exception.

It looks like it's trying to Jetify the older version of the Jetifier which is strange.  I've tried clearing out Gradle cache both in my home directory and in the project directory, invalidating Android Studio cache, and manually wiping out any build artifacts before a rebuild, and none of that worked.

This issue is also present on both the stable and latest canary channels of Android Studio.

My environment (stable channel):

Android Studio 3.2.1
Build #AI-181.5540.7.32.5056338, built on October 8, 2018
JRE: 1.8.0_152-release-1136-b06 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.14

My environment (canary channel):

Android Studio 3.3 Beta 2
Build #AI-182.4892.20.33.5078385, built on October 18, 2018
JRE: 1.8.0_152-release-1248-b01 x86_64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
macOS 10.14

Hi @pa...@google.com , Can you look into the above issue. It seem to be blocker at the moment for us too.

The original issue will be fixed in AGP 3.3.0-beta03 and 3.4.0-alpha03 (released on Nov 7).

@13: I've filed Issue 119135578, please follow the progress there.

If you still run into issues other than 119135578, please let us know. Thanks!