Fixed
Status Update
No update yet.
Comments
br...@google.com
[Deleted User] <[Deleted User]>
Hello,
I understand your issue is that the upload using the Python libraries to GS buckets and loading a table to BigQuery is comparatively slower than the upload using the commands gsutil and bq. Please let me know if I have misunderstood.
Let me clarify that the comparision should be done this way: Python's blob.upload_from_file() vs gsutil command and Python's load_table_from_file() vs bq command. Once that is clear, I would like to ask you for the codes that you use so I can reproduce the situation myself and get further insights. Please remove all the personal information from your codes before sharing them.
I will wait for your response,
Manuel Alaman
Google Cloud Big Data Support Barcelona
I understand your issue is that the upload using the Python libraries to GS buckets and loading a table to BigQuery is comparatively slower than the upload using the commands gsutil and bq. Please let me know if I have misunderstood.
Let me clarify that the comparision should be done this way: Python's blob.upload_from_file() vs gsutil command and Python's load_table_from_file() vs bq command. Once that is clear, I would like to ask you for the codes that you use so I can reproduce the situation myself and get further insights. Please remove all the personal information from your codes before sharing them.
I will wait for your response,
Manuel Alaman
Google Cloud Big Data Support Barcelona
[Deleted User] <[Deleted User]>
You are correct.
Attached python script will generate a test csv file and conduct the python client test. Please find and replace all occurrences of `UPDATE_THIS` text.
It also has the DDL query you'll need to use to create the BQ table before you run the script.
Additionally, it has the exact bq command you'll need to test the bq CLI utility against the same file.
I just tested again after creating this using python 3.6.9, google-cloud-bigquery 2.20.0, and BigQuery CLI 2.0.69 (most recent versions). I still see the same performance difference (~ 4MBps upload from the python client, vs ~70MBps upload for the same file to the same table using BigQuery CLI.
Let me know if you need anything else.
Attached python script will generate a test csv file and conduct the python client test. Please find and replace all occurrences of `UPDATE_THIS` text.
It also has the DDL query you'll need to use to create the BQ table before you run the script.
Additionally, it has the exact bq command you'll need to test the bq CLI utility against the same file.
I just tested again after creating this using python 3.6.9, google-cloud-bigquery 2.20.0, and BigQuery CLI 2.0.69 (most recent versions). I still see the same performance difference (~ 4MBps upload from the python client, vs ~70MBps upload for the same file to the same table using BigQuery CLI.
Let me know if you need anything else.
hu...@cuva.eu
Hey there any update on this?
br...@google.com
Hi Kevin,
We are still investigating the issue. At this point we obtained [1] for the script and [2] for the bq command, where the “Upload complete” was achieved in about 11 seconds.
Further updates will be published here.
[1]
2021-06-30 06:55:01,496 root test_uploads INFO: Beginning load job...
2021-06-30 06:57:08,662 root test_uploads INFO: Job ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
2021-06-30 06:57:08,662 root test_uploads INFO: BQ load job complete without error!
[2]
Upload complete.
Waiting on bqjob_XXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX_X ... (48s) Current status: DONE
We are still investigating the issue. At this point we obtained [1] for the script and [2] for the bq command, where the “Upload complete” was achieved in about 11 seconds.
Further updates will be published here.
[1]
2021-06-30 06:55:01,496 root test_uploads INFO: Beginning load job...
2021-06-30 06:57:08,662 root test_uploads INFO: Job ID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
2021-06-30 06:57:08,662 root test_uploads INFO: BQ load job complete without error!
[2]
Upload complete.
Waiting on bqjob_XXXXXXXXXXXXXXXXX_XXXXXXXXXXXXXXXX_X ... (48s) Current status: DONE
hu...@cuva.eu
Hi there, has there been any progress on this? Should I move this over to an Issue at https://github.com/googleapis/google-cloud-python ?
sp...@gmail.com
Since this is being investigated in github, having this issue also here seems like a duplicate. Let's close this and follow the fix on github.
pu...@gmail.com
We are eagerly waiting for this fix. Can you please give approximate date for next release.
pu...@gmail.com
Thank you Dianaa. Any timeline on when you're planning to release new version of SDK with this fix?
pu...@gmail.com
Dianaaa, is this issue fixed in latest Version 2.2 ? Please let us know we are waiting for this resolution.
br...@google.com
pu...@gmail.com
Can someone please let me know when this issue is getting fixed and next version release date?
di...@google.com
ga...@google.com
This should be fixed with the 2.3 release. It dropped iOS 7 support and thus allowed the use of the newer core data threading model which doesn't trigger the iOS 10 regression.
sa...@arrivy.com
Appearing again on OS version 17 as well.
Description
demonstration page if at all possible, or attach code.
1. Enable CoreData concurrency debugging: add `-com.apple.CoreData.ConcurrencyDebug 1` to Scheme arguments
2. Open a map view
3. The map begins to render then the app crashes with:
* thread #10: tid = 0x72c1c, 0x0000000108fd9124 CoreData`+[NSManagedObjectContext __Multithreading_Violation_AllThatIsLeftToUsIsHonor__] + 4, queue = 'SQLQueue 0x7fe39060a0f0 for Objects.sqlite', stop reason = EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)
* frame #0: 0x0000000108fd9124 CoreData`+[NSManagedObjectContext __Multithreading_Violation_AllThatIsLeftToUsIsHonor__] + 4
frame #1: 0x0000000108fd9525 CoreData`_PFAssertSafeMultiThreadedAccess_impl + 645
frame #2: 0x0000000108f597a8 CoreData`-[NSManagedObjectContext objectRegisteredForID:] + 328
frame #3: 0x0000000109057d99 CoreData`-[NSSQLiteConnection updateRow:forRequestContext:] + 169
frame #4: 0x00000001090fdaab CoreData`_writeChangesForSaveRequest + 1579
frame #5: 0x00000001090ff433 CoreData`_executeSaveChangesRequest + 291
frame #6: 0x0000000108f8e9c6 CoreData`-[NSSQLSaveChangesRequestContext executeRequestUsingConnection:] + 38
frame #7: 0x00000001090a4eb7 CoreData`__52-[NSSQLDefaultConnectionManager handleStoreRequest:]_block_invoke + 215
frame #8: 0x000000010eaa50cd libdispatch.dylib`_dispatch_client_callout + 8
frame #9: 0x000000010ea8230a libdispatch.dylib`_dispatch_barrier_sync_f_invoke + 340
frame #10: 0x00000001090a4d8c CoreData`-[NSSQLDefaultConnectionManager handleStoreRequest:] + 236
frame #11: 0x00000001090ac61f CoreData`-[NSSQLCoreDispatchManager routeStoreRequest:] + 351
frame #12: 0x000000010902f7d9 CoreData`-[NSSQLCore dispatchRequest:withRetries:] + 233
frame #13: 0x000000010902a89f CoreData`-[NSSQLCore processSaveChanges:forContext:] + 191
frame #14: 0x0000000108f21e4d CoreData`-[NSSQLCore executeRequest:withContext:error:] + 909
frame #15: 0x00000001090155d9 CoreData`__65-[NSPersistentStoreCoordinator executeRequest:withContext:error:]_block_invoke + 4169
frame #16: 0x000000010900d966 CoreData`-[NSPersistentStoreCoordinator _routeHeavyweightBlock:] + 390
frame #17: 0x0000000108f2187e CoreData`-[NSPersistentStoreCoordinator executeRequest:withContext:error:] + 654
frame #18: 0x0000000108f4baba CoreData`-[NSManagedObjectContext save:] + 3658
frame #19: 0x00000001079f92b8 Wingtip`__72-[GMSObjectDataCache storeObjectNamesAndData:version:completionHandler:]_block_invoke + 400
frame #20: 0x000000010ea7b980 libdispatch.dylib`_dispatch_call_block_and_release + 12
frame #21: 0x000000010eaa50cd libdispatch.dylib`_dispatch_client_callout + 8
frame #22: 0x000000010ea82fb8 libdispatch.dylib`_dispatch_queue_serial_drain + 569
frame #23: 0x000000010ea83b9f libdispatch.dylib`_dispatch_queue_invoke + 1073
frame #24: 0x000000010ea8407f libdispatch.dylib`_dispatch_queue_override_invoke + 683
frame #25: 0x000000010ea863b7 libdispatch.dylib`_dispatch_root_queue_drain + 720
frame #26: 0x000000010ea8608b libdispatch.dylib`_dispatch_worker_thread3 + 123
frame #27: 0x000000010ee54746 libsystem_pthread.dylib`_pthread_wqthread + 1299
frame #28: 0x000000010ee54221 libsystem_pthread.dylib`start_wqthread + 13
It looks like [GMSObjectDataCache storeObjectNamesAndData:version:completionHandler:] is accessing an NSManagedObjectContext from an incorrect thread.
Operating system version: iOS 10.0
Google Maps SDK for iOS version: 2.1.0
Hardware model: iOS Simulator iPhone 7
*********************************************************
For developers viewing this issue: please click the 'star' icon to be
notified of future changes, and to let us know how many of you are
interested in seeing it resolved.
*********************************************************