AWS Athena - GENERIC_INTERNAL_ERROR: Number of partition values does not match number of filters

Question

I'm querying a table in Athena that is giving the error: GENERIC_INTERNAL_ERROR: Number of partition values does not match number of filters

I was able to query it earlier, but added another partition (AWS glue job) to try and optimize joins I will be doing in the query later. I did re-run the crawler to update the table to reflect the partitions.

I'm wondering what this error actually means.

I tried recrawling the data, but no luck. Next up I'm wiping out the underlying S3 data and going to reprocess the entire set from scratch and then re-crawl it.

SELECT *
FROM mydb.mytable
LIMIT 10

I'd like to understand what the error means and if there is a way around it other than reprocessing the entire raw data set and re-crawling it.

Answer 1

I had the same issue today. In my case, it was because some of the underlying data in s3 had been removed, so the number of partitions in the Glue table didn't match the number of files in s3.

To fix it, I just deleted the tables in Glue, then re-ran the crawler.