Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

New posts in apache-spark

Why does Scala compiler fail with "no ': _*' annotation allowed here" when Row does accept varargs?

Scala Error: Could not find or load main class in both Scala IDE and Eclipse

How to configure Apache Spark random worker ports for tight firewalls?

Where is the Spark UI on Google Dataproc?

How to convert ArrayType to DenseVector in PySpark DataFrame?

Executing separate streaming queries in spark structured streaming

Unable to run a basic GraphFrames example

unexpected type: <class 'pyspark.sql.types.DataTypeSingleton'> when casting to Int on a ApacheSpark Dataframe

Link Spark with iPython Notebook

How to fix "java.io.NotSerializableException: org.apache.kafka.clients.consumer.ConsumerRecord" in Spark Streaming Kafka Consumer?

Efficient way to read specific columns from parquet file in spark

apache-spark parquet

How to overwrite entire existing column in Spark dataframe with new column?

Read whole text files from a compression in Spark

Full outer join in pyspark data frames

apache-spark pyspark

when to use mapParitions and mapPartitionsWithIndex?

apache-spark pyspark

How to add column with constant in Spark-java data frame

java apache-spark

How do I get the last item from a list using pyspark?

Dynamically rename multiple columns in PySpark DataFrame

Converting a dataframe into JSON (in pyspark) and then selecting desired fields

SparkException: Values to assemble cannot be null