apache-spark tutorials and guides

java.lang.NoClassDefFoundError: Could not initialize class when launching spark job via spark-submit in scala code

Dec 26, 2021

multi-processing with spark(PySpark) [duplicate]

Aug 27, 2019

python apache-spark pyspark spark-dataframe python-multiprocessing

How to manually set group.id and commit kafka offsets in spark structured streaming?

Nov 07, 2022

apache-spark apache-kafka spark-structured-streaming spark-kafka-integration

Use of lit() in expr()

Nov 19, 2022

scala apache-spark apache-spark-sql databricks

How to set group.id for consumer group in kafka data source in Structured Streaming?

Nov 14, 2022

apache-spark apache-kafka spark-structured-streaming spark-kafka-integration

Can SPARK use multicore properly?

Jan 05, 2020

multithreading apache-spark multicore

Pass array as an UDF parameter in Spark SQL

Oct 31, 2017

scala apache-spark dataframe apache-spark-sql user-defined-functions

How does Spark on Yarn store shuffled files?

Aug 30, 2022

apache-spark

Setting spark classpaths on EC2: spark.driver.extraClassPath and spark.executor.extraClassPath

Nov 09, 2022

hadoop apache-spark classpath maven-3

Basic Spark example not working

Oct 19, 2022

apache-spark

winutils.exe chmod command doesn't set permission

Oct 13, 2022

windows hadoop apache-spark hive

How to iterate scala wrappedArray? (Spark)

Apr 22, 2022

scala apache-spark apache-spark-sql

sparkSession/sparkContext can not get hadoop configuration

Nov 07, 2022

hadoop apache-spark

How to create Spark Dataset or Dataframe from case classes that contains Enums

Nov 26, 2017

scala apache-spark apache-spark-sql

Spark 2.0 implicit encoder, deal with missing column when type is Option[Seq[String]] (scala)

Nov 14, 2019

scala apache-spark apache-spark-dataset

Cumulate arrays from earlier rows (PySpark dataframe)

Aug 25, 2022

apache-spark dataframe pyspark apache-spark-sql

Dropping empty DataFrame partitions in Apache Spark

Aug 28, 2022

scala apache-spark apache-spark-sql

How to merge pyspark and pandas dataframes

Apr 24, 2019

python pandas apache-spark pyspark

What is Project node in execution query plan?

Sep 07, 2022

apache-spark apache-spark-sql

How to get the size of an RDD in Pyspark?

Sep 08, 2022

apache-spark pyspark

New posts in apache-spark