apache-spark tutorials and guides

Spark Dataset and java.sql.Date

Nov 03, 2022

Spark pulling data into RDD or dataframe or dataset

Dec 17, 2020

hadoop apache-spark apache-spark-sql spark-dataframe data-ingestion

Pyspark simple re-partition and toPandas() fails to finish on just 600,000+ rows

Mar 04, 2022

apache-spark memory pyspark distributed-computing bigdata

Spark error: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Dec 06, 2021

scala apache-spark

Spark is inventing his own AWS secretKey

Nov 09, 2022

amazon-web-services apache-spark amazon-s3 http-status-code-403 access-keys

Yarn slave nodes are not communicating with master node?

May 18, 2022

hadoop apache-spark hadoop-yarn

Project_Bank.csv is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [110, 111, 13, 10]

Nov 17, 2022

mysql csv apache-spark parquet spark-shell

Is there any way to get the output of Spark's Dataset.show() method as a string?

Oct 26, 2022

apache-spark apache-spark-sql

How to pivot streaming dataset?

Apr 08, 2021

apache-spark spark-structured-streaming apache-spark-2.0

UDF cause warning: CachedKafkaConsumer is not running in UninterruptibleThread (KAFKA-1894)

Oct 25, 2022

apache-spark pyspark apache-kafka apache-spark-sql spark-streaming

How can I force spark/hadoop to ignore the .gz extension on a file and read it as uncompressed plain text?

May 21, 2022

scala hadoop apache-spark gzip

pyspark equivalence of `df.loc`?

Mar 27, 2022

python pandas apache-spark dataframe pyspark

Calling a rest service from Spark

Sep 23, 2022

scala apache-spark rest

Does Spark support BigInteger type?

Aug 23, 2019

java scala apache-spark apache-spark-sql

Failed to execute user defined function($anonfun$9: (string) => double) on using String Indexer for multiple columns

Jan 04, 2022

scala apache-spark apache-spark-mllib

Spark: Prevent shuffle/exchange when joining two identically partitioned dataframes

Mar 04, 2022

apache-spark join pyspark apache-spark-sql pyspark-dataframes

How to set hive.metastore.warehouse.dir in HiveContext?

May 09, 2022

apache-spark apache-spark-sql spark-hive

Spark SQL grouping: Add to group by or wrap in first() if you don't care which value you get.;

Nov 23, 2018

sql group-by apache-spark udf

How to extract rules from decision tree spark MLlib

Jun 21, 2021

apache-spark apache-spark-mllib

Custom log4j appender in spark executor

Mar 29, 2022

apache-spark log4j

New posts in apache-spark