apache-spark tutorials and guides

How to match Dataframe column names to Scala case class attributes?

Mar 13, 2022

What does stage mean in the spark logs?

Mar 05, 2022

mapreduce apache-spark apache-spark-sql pyspark

Spark Job running on Yarn Cluster java.io.FileNotFoundException: File does not exits , eventhough the file exits on the master node

Aug 11, 2019

hadoop apache-spark hadoop-yarn spark-streaming

pyspark Do python processes on an executor node share broadcast variables in ram?

Oct 02, 2022

python apache-spark pyspark shared-memory

cannot resolve xyz given input columns error when creating Spark dataset

Jul 29, 2017

apache-spark

Creating indices for each group in Spark dataframe

Mar 27, 2022

apache-spark apache-spark-sql

java.lang.NoClassDefFoundError: Could not initialize class when launching spark job via spark-submit in scala code

Dec 26, 2021

java scala apache-spark apache-spark-sql spark-dataframe

multi-processing with spark(PySpark) [duplicate]

Aug 27, 2019

python apache-spark pyspark spark-dataframe python-multiprocessing

How to manually set group.id and commit kafka offsets in spark structured streaming?

Nov 07, 2022

apache-spark apache-kafka spark-structured-streaming spark-kafka-integration

Use of lit() in expr()

Nov 19, 2022

scala apache-spark apache-spark-sql databricks

How to set group.id for consumer group in kafka data source in Structured Streaming?

Nov 14, 2022

apache-spark apache-kafka spark-structured-streaming spark-kafka-integration

Can SPARK use multicore properly?

Jan 05, 2020

multithreading apache-spark multicore

Pass array as an UDF parameter in Spark SQL

Oct 31, 2017

scala apache-spark dataframe apache-spark-sql user-defined-functions

How does Spark on Yarn store shuffled files?

Aug 30, 2022

apache-spark

Setting spark classpaths on EC2: spark.driver.extraClassPath and spark.executor.extraClassPath

Nov 09, 2022

hadoop apache-spark classpath maven-3

Basic Spark example not working

Oct 19, 2022

apache-spark

winutils.exe chmod command doesn't set permission

Oct 13, 2022

windows hadoop apache-spark hive

How to iterate scala wrappedArray? (Spark)

Apr 22, 2022

scala apache-spark apache-spark-sql

sparkSession/sparkContext can not get hadoop configuration

Nov 07, 2022

hadoop apache-spark

How to create Spark Dataset or Dataframe from case classes that contains Enums

Nov 26, 2017

scala apache-spark apache-spark-sql

New posts in apache-spark