apache-spark-sql tutorials

Partitions not being pruned in simple SparkSQL queries

Sep 13, 2022

Using TestHiveContext/HiveContext in unit tests

Jun 29, 2021

apache-spark hive apache-spark-sql hivecontext

Not able to fetch result from hive transaction enabled table through spark-sql

Oct 20, 2022

hadoop apache-spark hive apache-spark-sql

How to write dataframe (obtained from hive table) into hadoop SequenceFile and RCFile?

Oct 16, 2022

apache-spark apache-spark-sql spark-dataframe

The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx--------- (on Linux)

Jan 18, 2020

apache-spark hive apache-spark-sql spark-dataframe hiveql

Using .where() on pyspark.sql.functions.max().over(window) on Spark 2.4 throws Java exception

Aug 22, 2022

apache-spark exception pyspark apache-spark-sql

one-hot encode of multiple string categorical features using Spark DataFrames

Jun 21, 2022

python apache-spark pyspark apache-spark-sql bigdata

Aggregate while dropping duplicates in pyspark

Jul 02, 2022

dataframe apache-spark pyspark apache-spark-sql databricks

How to extract complex JSON structures using Apache Spark 1.4.0 Data Frames

Nov 21, 2022

apache-spark apache-spark-sql

Apache Spark: In SparkSql, are sql's vulnerable to Sql Injection [duplicate]

Apr 05, 2022

hadoop apache-spark hive apache-spark-sql bigdata

rank() function usage in Spark SQL

Sep 04, 2018

java apache-spark apache-spark-sql window-functions rank

How to convert the group by function to data frame

Nov 21, 2022

scala apache-spark apache-spark-sql

How can you update values in a dataset?

Aug 25, 2022

apache-spark apache-spark-sql

How to add sparse vectors after group by, using Spark SQL?

Sep 16, 2022

python apache-spark machine-learning apache-spark-sql pyspark-sql

How to compute statistics on a streaming dataframe for different type of columns in a single query?

Sep 24, 2022

scala apache-spark apache-spark-sql spark-structured-streaming

Pyspark: java.lang.OutOfMemoryError: GC overhead limit exceeded

Nov 08, 2022

apache-spark pyspark apache-spark-sql

How to write dataframe with duplicate column name into a csv file in pyspark

Sep 05, 2022

apache-spark pyspark apache-spark-sql apache-spark-2.0

Spark - Non-time-based windows are not supported on streaming DataFrames/Datasets;

Sep 14, 2022

java apache-spark apache-spark-sql spark-streaming

Why does Spark groupBy.agg(min/max) of BigDecimal always return 0?

Nov 11, 2022

apache-spark apache-spark-sql bigdecimal

How do explicit table partitions in Databricks affect write performance?

Jun 26, 2022

amazon-s3 hive apache-spark-sql databricks delta-lake

New posts in apache-spark-sql