Spark 2.4 to Spark 3.2 Refactoring

When migrating from Spark 2.4 to Spark 3.x, there are significant changes to executing Dataset/ Dataframe APIs, DDL statements, and UDF functions.

See the Apache Spark migration documentation when migrating to Spark 3.x.