Heatmap colors not working in plotly

Question

I'm trying to print out a grid of heatmaps using plotly for R in Shiny. I want to give them a custom color scale, but it's not behaving how I want it to. When I use the colors= option when building my plotly chart, it seems to use the distribution of values, rather than the zmin and zmax that I gave it to assign colors.

In my example code below, you can see that I give each plot the same colorscale (colorScale) using the colors= option. This works how I expect when I have a well distributed set of data, as in the first, second, and fourth row of plots.

However, in the third row, where the plots have very skewed data, you can see that the scales look different than everything else - they have the blue and red, but skip over the white in the middle, instead having purple.

In my actual code, this is causing a big problem for a chart that has a lot of values around the middle, with some extremes on either end - I want those values in the middle to appear white, to show that there was no change, but instead they are purple, making it harder to pick out the important values (the extreme ones).

Is there a way to force the color assignment to behave how I want it to?

Thanks, Cliff

server.R

if (!require("pacman")) install.packages("pacman")

pacman::p_load(shiny,data.table,plotly)


colorScale <- colorRamp(c("darkblue","cornflowerblue","white","sandybrown","firebrick"))

nCodeNames <- c("a","b","c","d","e","f","g","h","i","j","k","l")
means = c(rnorm(600,0,2.5),runif(600,-5,5),runif(130,-4,-3.9),runif(70,4.5,5),rnorm(150,-3),rnorm(50,4),rnorm(180,-2.5),runif(20,4.93,4.98),runif(300,-4,3),rnorm(300,3.5))



dt <- data.table(age=rep(rep(c(11:20),times=20),times=12),composite=rep(rep(c(81:100),each=10),times=12),mean=means,n_code=rep(nCodeNames,each=200))

sub<-dt[n_code=="a"]

shinyServer(function(input, output) {

for(Ncode in nCodeNames){
  local({
    ncode = Ncode
    output[[paste0("grid",ncode)]] <- renderPlotly({
      sub <- dt[n_code == ncode]
      p <- plot_ly(data=sub, x=~age, y=~composite, z=~mean, type="heatmap", zmin=-5,zmax=5, 
                   colors = colorScale, colorbar=list(thickness="15"))%>%
           layout(title=ncode,xaxis=list(type="category",tickvals=c(11,15,20)),yaxis=list(title="",ticks=""))
    })
  })
}
})

ui.R

if (!require("pacman")) install.packages("pacman")

pacman::p_load(shiny, plotly)

nCodeNames <- c("a","b","c","d","e","f","g","h","i","j","k","l")


shinyUI(navbarPage(
  "E-N Matrics: Proportion of E-Code Resulting in each N-Code",

  tabPanel("Grid",

            lapply(c(1:4), function(i) fluidRow(
              lapply(c(1:3), function(j) column(4, plotlyOutput(paste0("grid",nCodeNames[(i-1)*3+j]))))
             ))

           #fluidRow(column(4,plotlyOutput(paste0("grid",nCodeNames[(1-1)*3+1]))),column(4,plotly))
  )
  ))

Answer 1

I encountered similar issues with the colorscale in R plotly heatmap. When the data for the z argument has screwed distribution, only a few colors specified in the colorscale are used by plotly.

I found a work-around solution by creating a new variable according to the quantiles of the original variable, and pass it to the z argument. Here is the R code for the general idea. You will need to customize it to make it work for a specific problem.

library(plotly)
library(RColorBrewer)

# create a dataframe where z has a skewed distribution
set.seed(1)
df = data.frame(x = rep(1:50, 20) , y = rep(1:20,each =50), z = rgamma(1000, 2, 0.5))

# check distribution of z 
plot_ly(data = df, x = ~z, type = "histogram")%>% 
    layout(title = "histogram of z")  

# original heatmap
# pass the column z with screwed distribution to z argument
plot_ly(data=df, x=~x, y=~y, z=~z, type="heatmap",  
        colors = "Spectral") %>% 
    layout(title = "original heatmap")

# some data processing work 

# find unique quantiles of z 
quantiles = unique(quantile(df$z, seq(0,1,0.1)))

# create a dummy column z1 of discrete values using the quantiles as cut off
# the ideas is to arrage the data to subgroups of roughly the same size 
df$z1= cut(df$z, breaks =  c(quantiles[1]-1,quantiles[-1]), right = TRUE, labels = FALSE)

# check distribution of z1 
plot_ly(data = df, x = ~z1, type = "histogram")%>% 
    layout(title = "histogram of z1")


# new heatmap 
# passes the new column z1 to z argument 
plot_ly(data=df, x=~x, y=~y, z=~z1, type="heatmap",  
        # make sure hovering over displays original z
        text =~z, hoverinfo = "text", 
        # use the color palettes from RColorBrewer, 
        # or your customized colorscale
        colors = "Spectral",  
        # map the label of the colorbar back to the quantiles
        colorbar=list(tickmode="array", tickvals = 1:(length(quantiles)-1), ticktext = round(quantiles,2)[-1], title = "z")) %>% 
layout(title = "new heat map")

Below are the original heatmap and the new heatmap generated by plotly. The new heatmap uses more colors from "Spectral" palette to differetiate between smaller values.

Hope this helps!

---

Update on April 3,2017

I opened a request on the R plotly repository for capability to transform the color scale.

https://github.com/ropensci/plotly/issues/920