thebluephantom

thebluephantom has asked 25 questions and find answers to 41 problems.

Stats

About

Senior IT Architect, E2E-Solution Architect, Data Architect, Big Data Architect.

TOGAF certified. Academic and practical team player.

Spark / Big Data programmer on occasion, using Scala in Databricks style, not that Monad academic approach.

thebluephantom questions

Spark pivot invokes Job even though pivot is not an Action
High Concurrency Clusters in Databricks
spark.sql.shuffle.partitions of 200 default partitions conundrum
Node Name not showing in neo4j
Scala Spark contains vs. does not contain

thebluephantom answers

Is 'load' command in spark an action or transformation?
Spark Dataframes - derive single row containing non-null values per key from multiple such rows
Losing entries when inner-joining data to a left-joined DataFrame in Spark Structured Streaming
Delta table versioning while writing from a Spark structured streaming job
Using Spark window with more than one partition when there is no obvious partitioning column
How to use salting technique for joining data frames having skewed data
SparkContext can only be used on the driver
Convert spark dataframe to Delta table on azure databricks - warning
In which situations are the stages of DAG skipped?
how to understand each part of the name of a parquet file