NoSuchMethodError on Spark 3.1 in Databricks

[mithun1979]

Hi,
We are trying to run TPC-DS queries on spark-sql-perf on Spark 3.1.
We tried in Databricks runtimes 7.3 LTS and 8.1.
We are using the JAR - spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar
We are getting the following error:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation.tableIdentifier()Lorg/apache/spark/sql/catalyst/TableIdentifier;
logs.txt

Please find the detailed logs attached.
Would you be able to help with guidance on this issue.

Thanks and Regards
Saurav Chakraborty

[mithun1979]

Just wanted to mention that we are running the following code. Its runs fine of Databricks runtion 5.5 LTS (Spark 2.4) with the JAR spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar.

import com.databricks.spark.sql.perf.tpcds.TPCDS

val tpcds = new TPCDS (sqlContext = sqlContext)
def queries = {
val filtered_queries = query_filter match {
case Seq() => tpcds.tpcds2_4Queries
case _ => tpcds.tpcds2_4Queries.filter(q => query_filter.contains(q.name))
}
if (randomizeQueries) scala.util.Random.shuffle(filtered_queries) else filtered_queries
}

val experiment = tpcds.runExperiment(
queries,
iterations = iterations,
resultLocation = resultLocation,
forkThread = true)

println(experiment.toString)
experiment.waitForFinish(timeout6060)