Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

ElasticSearch - Malformed Query (RoR)

I'm trying to set up an existing project using elasticsearch and ruby-on-rails. I can get the project up and running however elasticsearch queries return various error message for me and none of my colleagues:

[400] {"error":{"root_cause":[{"type":"parsing_exception","reason":"[and] query malformed, no start_object after query name","line":1,"col":896}],"type":"parsing_exception","reason":"[and] query malformed, no start_object after query name","line":1,"col":896},"status":400}

This obviously leads me to believe it's something to do with my setup. I've installed the same version of them (2.4), however even the setup command that was supplied with the project:

rake environment elasticsearch:import:all DIR=app/models FORCE=y

gives me a timeout error. I've checked and the service is running. Any help as to what could be wrong with my set up would be greatly appreciated as I'm at a dead end for 2 days on this.

Thanks in advance.

EDIT: I don't think the issue is with the query itself as this works for my colleagues however it is as follows:

{: date_range => {: from => Mon,
    01 Aug 2016 00: 00: 00 UTC + 00: 00,
    : to => Mon,
    31 Jul 2017 00: 00: 00 UTC + 00: 00
  }, : activities_only => nil
}, @pupils = [# < Pupil id: 1042, first_name: "Mark", last_name: "Mekhaiel", date_of_birth: "2017-03-19", level: 0, group_id: 95, created_at: "2017-07-31 08:59:59", updated_at: "2017-07-31 09:48:27", username: "m.mekhaiel1", picture: 0, color: 0, archived: false, sub_group_id: nil, score: 20, deleted_at: nil > ], @search = # < PupilCreditSearch: 0x007fc961801308 @interval = : month, @page = 1, @per = 25, @pupil_id = [1042], @date_range = {: from => Mon,
  01 Aug 2016 00: 00: 00 UTC + 00: 00,
  : to => Mon,
  31 Jul 2017 00: 00: 00 UTC + 00: 00
}, @queries = [# < Search::Query: 0x007fc964791bb8 @query = nil, @fields = [], @type = : multi_match, @options = {: fields => [],
  : type => : cross_fields
} > ], @filters = [# < Search::Filter: 0x007fc964756db0 @name = : pupil_id, @value = [1042], @type = : terms, @options = {} > ], @aggregations = {: years_created => # < Search::Aggregation: 0x007fc9647a32f0 @type = : date_histogram,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_histogram,
    : params => {: interval => : year
    }
  } > ,
  : months_created => # < Search::Aggregation: 0x007fc9647a30c0 @type = : date_histogram,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_histogram,
    : params => {: interval => : month
    }
  } > ,
  : date_ranges => # < Search::Aggregation: 0x007fc9647a2800 @type = : date_range,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_range,
    : params => {: keyed => true,
      : ranges => [{: from => "now/w",
        : to => "now+1w/w",
        : key => : this_week
      }, {: from => "now/M",
        : to => "now+1M/M",
        : key => : this_month
      }, {: from => "now-1w/w",
        : to => "now-2w/w",
        : key => : last_week
      }, {: from => "now-1M/M",
        : to => "now-2M/M",
        : key => : last_month
      }, {: from => "2016-09-01",
        : to => "2017-08-31",
        : key => : academic_year
      }]
    }
  } > ,
  : groups => # < Search::Aggregation: 0x007fc9647a2260 @type = : terms,
  @field = : pupil_id,
  @options = {: aggs => {: all_dates => # < Search::Aggregation: 0x007fc9647a22d8 @type = : date_histogram,
      @field = : created_at,
      @options = {: params => {: interval => : month
        },
        : aggs => {: total_points => # < Search::Aggregation: 0x007fc9647a2440 @type = : sum,
          @field = : points,
          @options = {} > ,
          : running_total => # < Search::Aggregation: 0x007fc9647a2350 @type = : cumulative_sum,
          @field = nil,
          @options = {: params => {: buckets_path => "total_points"
            }
          } >
        }
      } >
    }
  } > ,
  : options => # < Search::Aggregation: 0x007fc9647a1518 @type = : global,
  @field = nil,
  @options = {: aggs => {: filtered => # < Search::Aggregation: 0x007fc9647a1590 @type = : filter,
      @field = nil,
      @options = {: params => {: and => [{: range => {: created_at => {: gte => Mon,
                01 Aug 2016 00: 00: 00 UTC + 00: 00,
                : lt => Mon,
                31 Jul 2017 00: 00: 00 UTC + 00: 00
              }
            }
          }, {: terms => {: pupil_id => [1042]
            }
          }]
        },
        : aggs => {: categories => # < Search::Aggregation: 0x007fc9647a1b80 @type = : terms,
          @field = "category",
          @options = {: ignore => true
          } > ,
          : topic_ids => # < Search::Aggregation: 0x007fc9647a19a0 @type = : terms,
          @field = "topic_id",
          @options = {: ignore => true
          } > ,
          : learning_objective_ids => # < Search::Aggregation: 0x007fc9647a17c0 @type = : terms,
          @field = "learning_objective_id",
          @options = {: ignore => true
          } > ,
          : completed_activities => # < Search::Aggregation: 0x007fc9647a1608 @type = : terms,
          @field = : pupil_id,
          @options = {: aggs => {: activities => # < Search::Aggregation: 0x007fc9647a1680 @type = : filter,
              @field = nil,
              @options = {: params => {: and => [{: terms => {: category => [0, 1, 2, 3]
                    }
                  }]
                }
              } >
            }
          } >
        }
      } >
    }
  } >
}, @request = # < Search::Request: 0x007fc964753548 @params = {: query => {: bool => {: must => [{: match_all => {}
      }],
      : filter => [{: terms => {: pupil_id => [1042]
        }
      }]
    }
  },
  : aggs => {: years_created => {: date_histogram => {: field => : created_at,
        : interval => : year
      }
    },
    : months_created => {: date_histogram => {: field => : created_at,
        : interval => : month
      }
    },
    : date_ranges => {: date_range => {: field => : created_at,
        : keyed => true,
        : ranges => [{: from => "now/w",
          : to => "now+1w/w",
          : key => : this_week
        }, {: from => "now/M",
          : to => "now+1M/M",
          : key => : this_month
        }, {: from => "now-1w/w",
          : to => "now-2w/w",
          : key => : last_week
        }, {: from => "now-1M/M",
          : to => "now-2M/M",
          : key => : last_month
        }, {: from => "2016-09-01",
          : to => "2017-08-31",
          : key => : academic_year
        }]
      }
    },
    : groups => {: terms => {: field => : pupil_id
      },
      : aggs => {: all_dates => {: date_histogram => {: field => : created_at,
            : interval => : month
          },
          : aggs => {: total_points => {: sum => {: field => : points
              }
            },
            : running_total => {: cumulative_sum => {: buckets_path => "total_points"
              }
            }
          }
        }
      }
    },
    : options => {: global => {},
      : aggs => {: filtered => {: filter => {: and => [{: range => {: created_at => {: gte => Mon,
                  01 Aug 2016 00: 00: 00 UTC + 00: 00,
                  : lt => Mon,
                  31 Jul 2017 00: 00: 00 UTC + 00: 00
                }
              }
            }, {: terms => {: pupil_id => [1042]
              }
            }]
          },
          : aggs => {: categories => {: terms => {: field => "category"
              }
            },
            : topic_ids => {: terms => {: field => "topic_id"
              }
            },
            : learning_objective_ids => {: terms => {: field => "learning_objective_id"
              }
            },
            : completed_activities => {: terms => {: field => : pupil_id
              },
              : aggs => {: activities => {: filter => {: and => [{: terms => {: category => [0, 1, 2, 3]
                      }
                    }]
                  }
                }
              }
            }
          }
        }
      }
    }
  }
} >> , @options = # < ReportOptions: 0x007fc96b663258 @search = # < PupilCreditSearch: 0x007fc961801308 @interval = : month, @page = 1, @per = 25, @pupil_id = [1042], @date_range = {: from => Mon,
  01 Aug 2016 00: 00: 00 UTC + 00: 00,
  : to => Mon,
  31 Jul 2017 00: 00: 00 UTC + 00: 00
}, @queries = [# < Search::Query: 0x007fc964791bb8 @query = nil, @fields = [], @type = : multi_match, @options = {: fields => [],
  : type => : cross_fields
} > ], @filters = [# < Search::Filter: 0x007fc964756db0 @name = : pupil_id, @value = [1042], @type = : terms, @options = {} > ], @aggregations = {: years_created => # < Search::Aggregation: 0x007fc9647a32f0 @type = : date_histogram,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_histogram,
    : params => {: interval => : year
    }
  } > ,
  : months_created => # < Search::Aggregation: 0x007fc9647a30c0 @type = : date_histogram,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_histogram,
    : params => {: interval => : month
    }
  } > ,
  : date_ranges => # < Search::Aggregation: 0x007fc9647a2800 @type = : date_range,
  @field = : created_at,
  @options = {: field => : created_at,
    : type => : date_range,
    : params => {: keyed => true,
      : ranges => [{: from => "now/w",
        : to => "now+1w/w",
        : key => : this_week
      }, {: from => "now/M",
        : to => "now+1M/M",
        : key => : this_month
      }, {: from => "now-1w/w",
        : to => "now-2w/w",
        : key => : last_week
      }, {: from => "now-1M/M",
        : to => "now-2M/M",
        : key => : last_month
      }, {: from => "2016-09-01",
        : to => "2017-08-31",
        : key => : academic_year
      }]
    }
  } > ,
  : groups => # < Search::Aggregation: 0x007fc9647a2260 @type = : terms,
  @field = : pupil_id,
  @options = {: aggs => {: all_dates => # < Search::Aggregation: 0x007fc9647a22d8 @type = : date_histogram,
      @field = : created_at,
      @options = {: params => {: interval => : month
        },
        : aggs => {: total_points => # < Search::Aggregation: 0x007fc9647a2440 @type = : sum,
          @field = : points,
          @options = {} > ,
          : running_total => # < Search::Aggregation: 0x007fc9647a2350 @type = : cumulative_sum,
          @field = nil,
          @options = {: params => {: buckets_path => "total_points"
            }
          } >
        }
      } >
    }
  } > ,
  : options => # < Search::Aggregation: 0x007fc9647a1518 @type = : global,
  @field = nil,
  @options = {: aggs => {: filtered => # < Search::Aggregation: 0x007fc9647a1590 @type = : filter,
      @field = nil,
      @options = {: params => {: and => [{: range => {: created_at => {: gte => Mon,
                01 Aug 2016 00: 00: 00 UTC + 00: 00,
                : lt => Mon,
                31 Jul 2017 00: 00: 00 UTC + 00: 00
              }
            }
          }, {: terms => {: pupil_id => [1042]
            }
          }]
        },
        : aggs => {: categories => # < Search::Aggregation: 0x007fc9647a1b80 @type = : terms,
          @field = "category",
          @options = {: ignore => true
          } > ,
          : topic_ids => # < Search::Aggregation: 0x007fc9647a19a0 @type = : terms,
          @field = "topic_id",
          @options = {: ignore => true
          } > ,
          : learning_objective_ids => # < Search::Aggregation: 0x007fc9647a17c0 @type = : terms,
          @field = "learning_objective_id",
          @options = {: ignore => true
          } > ,
          : completed_activities => # < Search::Aggregation: 0x007fc9647a1608 @type = : terms,
          @field = : pupil_id,
          @options = {: aggs => {: activities => # < Search::Aggregation: 0x007fc9647a1680 @type = : filter,
              @field = nil,
              @options = {: params => {: and => [{: terms => {: category => [0, 1, 2, 3]
                    }
                  }]
                }
              } >
            }
          } >
        }
      } >
    }
  } >
}, @request = # < Search::Request: 0x007fc964753548 @params = {: query => {: bool => {: must => [{: match_all => {}
      }],
      : filter => [{: terms => {: pupil_id => [1042]
        }
      }]
    }
  },
  : aggs => {: years_created => {: date_histogram => {: field => : created_at,
        : interval => : year
      }
    },
    : months_created => {: date_histogram => {: field => : created_at,
        : interval => : month
      }
    },
    : date_ranges => {: date_range => {: field => : created_at,
        : keyed => true,
        : ranges => [{: from => "now/w",
          : to => "now+1w/w",
          : key => : this_week
        }, {: from => "now/M",
          : to => "now+1M/M",
          : key => : this_month
        }, {: from => "now-1w/w",
          : to => "now-2w/w",
          : key => : last_week
        }, {: from => "now-1M/M",
          : to => "now-2M/M",
          : key => : last_month
        }, {: from => "2016-09-01",
          : to => "2017-08-31",
          : key => : academic_year
        }]
      }
    },
    : groups => {: terms => {: field => : pupil_id
      },
      : aggs => {: all_dates => {: date_histogram => {: field => : created_at,
            : interval => : month
          },
          : aggs => {: total_points => {: sum => {: field => : points
              }
            },
            : running_total => {: cumulative_sum => {: buckets_path => "total_points"
              }
            }
          }
        }
      }
    },
    : options => {: global => {},
      : aggs => {: filtered => {: filter => {: and => [{: range => {: created_at => {: gte => Mon,
                  01 Aug 2016 00: 00: 00 UTC + 00: 00,
                  : lt => Mon,
                  31 Jul 2017 00: 00: 00 UTC + 00: 00
                }
              }
            }, {: terms => {: pupil_id => [1042]
              }
            }]
          },
          : aggs => {: categories => {: terms => {: field => "category"
              }
            },
            : topic_ids => {: terms => {: field => "topic_id"
              }
            },
            : learning_objective_ids => {: terms => {: field => "learning_objective_id"
              }
            },
            : completed_activities => {: terms => {: field => : pupil_id
              },
              : aggs => {: activities => {: filter => {: and => [{: terms => {: category => [0, 1, 2, 3]
                      }
                    }]
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
like image 267
Mark Avatar asked Aug 02 '17 10:08

Mark


1 Answers

The problem is with one of the aggregations, namely:

:filter => { :and => [...] }

...which leads me to conclusion that you actually may be running elasticsearch 5 and not 2.4, on your machine, because this syntax got dropped in version 5 in favour of:

:filter => { :bool => { :must: [...] } }

Verify the elasticsearch version with curl localhost:9200 (or the actual host/port your app is connecting to).

like image 102
Michał Szajbe Avatar answered Oct 05 '22 08:10

Michał Szajbe