[SPARK-57262][SQL][WEBUI] Job description derived from a query should respect spark.sql.redaction.string.regex

[sarutak]

What changes were proposed in this pull request?

This PR changes SparkSQLDriver.scala to redact a query before setJobDescription.

Why are the changes needed?

In the current implementation, when a query is executed through SparkSQLDriver, redaction is done in SQLExecution.scala so the description in the table on the top of /SQL/execution is redacted.

But the description in the table on the /jobs page and the one in the table on the bottom of /SQL/execution page are not redacted.

NOTE:
Even after this PR is merged, when a job description is set manually using sc.setJobDescription, the description displayed in the /jobs page and the one on the bottom of /SQL/execution page are not redacted though the one on the top of SQL/execution page is redacted.

$ bin/spark-shell -c spark.sql.redaction.string.regex="secret.*=.*"
scala> val s = "SELECT * FROM (SELECT 'secret=1')"
scala> sc.setJobDescription(s)
scala> sql(s).show()
+--------+
|secret=1|
+--------+
|secret=1|
+--------+

description in /jobs page

description in /SQL/execution (top)

description in /SQL/execution (bottom)

This is consistent with the previous behavior and not a regression. There is no simple way to redact them and doing it is out of scope of this PR.

Does this PR introduce any user-facing change?

Yes.

How was this patch tested?

Added new test and confirmed the test SQL execution description should respect spark.sql.redaction.string.regex added in #56358 passed.
Also confirmed descriptions are redacted in UI.

$ bin/spark-sql --conf spark.sql.redaction.string.regex="secret.*=.*"
spark-sql (default)>  CREATE TABLE test1(secret string);
spark-sql (default)> SELECT * FROM test1 WHERE secret=1;

Was this patch authored or co-authored using generative AI tooling?

Kiro CLI / Claude

[dongjoon-hyun]

Thank you for resubmission. BTW, we don't need a new test coverage for this, @sarutak ? I thought you will reuse your previous test case, SparkSQLDriverSuite.scala.

[sarutak]

Oh, forgot to include the new test. Thanks @dongjoon-hyun .

[dongjoon-hyun]

+1, LGTM (Pending CIs). Thank you, @sarutak !

[dongjoon-hyun]

Ur, @sarutak . While checking once more before merging, I found a little strange description. Is the PR description correct?

NOTE:
Even after this PR is merged, when a job description is set manually using sc.setJobDescription, the description displayed in the /jobs page and the one on the bottom of /SQL/execution page are not redacted though the one on the top of SQL/execution page is redacted.

[sarutak]

Yes, it's correct, @dongjoon-hyun. Please check the screenshot in the description.
As I mentioned in the description too, it's consistent with the previous behavior and not a regression. There is no simple way to redact them and doing it, and I think users should not set a text which includes confidential information as a description by themselves.

[dongjoon-hyun]

Thank you for your confirmation. Let me check this once more Today, @sarutak .

[dongjoon-hyun]

I finished the verification and fully agree with your assessment. Only SQL tab has the redacted result. We are good to go. Thank you for waiting for me. Feel free to merge this, @sarutak .

[sarutak]

Merged to master/branch-4.x/branch-4.2. Thank you all for reviewing.