Not able to select fields inside struct in pyspark dataframe with Spark version 3.3.0 and Apache Hudi optional component 0.12.0 [293281647]

Assigned

Bug

Status Update

No update yet.

Description

mi...@echo-analytics.io

created issue #1

Jul 26, 2023 03:33PM

I am running a dataproc pyspark job on gcp to read data from hudi table (parquet format) into pyspark dataframe. Below is the output of printSchema() on pyspark dataframe.

root
 |-- columnA: string (nullable = true)
 |-- columnB: struct (nullable = true)
 |    |-- subColumn1: string (nullable = true)
 |    |-- subColumn2: string (nullable = true)
 |    |-- ...
 |    |-- subColumnN: string (nullable = true)
 |-- columnC: string (nullable = true)
 |-- ...
 |-- columnN: string (nullable = true)

I am not able to select struct type columns and their fields (ex: columnB.subColumn1), it gives below error. Though other string type columns works fine (ex: columnA).

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 516): java.lang.ClassCastException: class org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class org.apache.spark.sql.vectorized.ColumnarBatch (org.apache.spark.sql.catalyst.expressions.UnsafeRow and org.apache.spark.sql.vectorized.ColumnarBatch are in unnamed module of loader 'app')

Comments

mi...@echo-analytics.io <mi...@echo-analytics.io> #2Jul 26, 2023 03:35PM

Hi , Could you please clarify the issue description or share any screen shot of the problem where you are facing issue ?

nr...@google.com <nr...@google.com> Jul 27, 2023 08:23AM

Assigned to nr...@google.com.

nr...@google.com <nr...@google.com> #3Jul 28, 2023 05:25AM

Hi, its a feature request. Can you change it to feature request?
I am requesting a dataproc image version that supports spark 3.4

https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-release-2.2

mi...@echo-analytics.io <mi...@echo-analytics.io> #4Aug 2, 2023 07:04AM

Hello,

Thank you for reaching out to us with your request.

We have duly noted your feedback and will thoroughly validate it. While we cannot provide an estimated time of implementation or guarantee the fulfillment of the feature request , please be assured that your input is highly valued. Your feedback enables us to enhance our products and services.

nr...@google.com <nr...@google.com> #5Aug 11, 2023 06:45AM

Reassigned to gc...@google.com.

Hello,

Thanks for reaching out to us!

This issue request is acknowledged and we will validate it further. Though we can't provide an ETA on this issue nor guarantee their implementation, rest assured that your feedback is always taken very seriously, as it allows us to improve our products.

Thanks

Issue 293281647