[SPARK-54125][SQL] Support SHOW DATABASES AS JSON

[vinodkc]

What changes were proposed in this pull request?

Support SHOW DATABASES ... [AS JSON] to optionally display output in JSON format.

Why are the changes needed?

Users need a programmatic, machine-readable way to list databases for automation and tooling purposes. Currently, SHOW DATABASES only provides tabular text output which is difficult to parse reliably in scripts and applications.

Does this PR introduce any user-facing change?

Yes. New syntax is available:

SHOW DATABASES [LIKE pattern] AS JSON;

eg:

scala>  sql("SHOW databases").show(false)
+---------+
|namespace|
+---------+
|default  |
|dev_db   |
|prod_db  |
|showdb   |
|test_db1 |
|test_db2 |
+---------+


scala> sql("SHOW databases AS JSON").show(false)
+----------------------------------------------------------------------------------------------------------------------------------+
|json_metadata                                                                                                                     |
+----------------------------------------------------------------------------------------------------------------------------------+
|{"namespaces":[{"name":"default"},{"name":"dev_db"},{"name":"prod_db"},{"name":"showdb"},{"name":"test_db1"},{"name":"test_db2"}]}|
+----------------------------------------------------------------------------------------------------------------------------------+

How was this patch tested?

Added tests show-databases.sql

Was this patch authored or co-authored using generative AI tooling?

No.

[vinodkc]

CC @asl3 , @cloud-fan , @gengliangwang

[cloud-fan]

This is a bit tricky. The current SHOW NAMESPACES has queryable output already, like sql("SHOW NAMESPACES").orderBy("namespace"). It's unclear what's the benefit of the AS JSON version.