Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

elasticsearch: how to free store size after deleting documents

On my elasticsearch server: total documents: 3 million, total size: 3.6G Then, I delete about 2.8 millions documents: total documents: about 0.13 million, total size: 3.6G

I have deleted the documents, how should I free the size of the documents?

like image 617
Michael Avatar asked Dec 16 '13 10:12

Michael


People also ask

Does deleting index delete documents Elasticsearch?

Deleting an index deletes its documents, shards, and metadata. It does not delete related Kibana components, such as data views, visualizations, or dashboards.

How do I delete a document in Elasticsearch?

You use DELETE to remove a document from an index. You must specify the index name and document ID. You cannot send deletion requests directly to a data stream. To delete a document in a data stream, you must target the backing index containing the document.


4 Answers

Deleting documents only flags these as deleted, so they would not be searched. To reclaim disk space, you have to optimize the index:

curl -XPOST 'http://localhost:9200/_optimize?only_expunge_deletes=true'

documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-optimize.html

The documentation has moved to: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html

Update

Starting with Elasticsearch 2.1.x, optimize is deprecated in favor of forcemerge. The API is the same, only the endpoint did change.

curl -XPOST 'http://localhost:9200/_forcemerge?only_expunge_deletes=true'
like image 131
knutwalker Avatar answered Oct 19 '22 05:10

knutwalker


In the current elasticsearch version(7.5),

  1. To optimize all indices:

    POST /_forcemerge?only_expunge_deletes=true

  2. To optimize single index

    POST /twitter/_forcemerge?only_expunge_deletes=true , where twitter is the index

  3. To optimize several indices

    POST /twitter,facebook/_forcemerge?only_expunge_deletes=true , where twitter and facebook are the indices

Reference: https://www.elastic.co/guide/en/elasticsearch/reference/7.5/indices-forcemerge.html#indices-forcemerge

like image 39
Sri Harsha Arangi Avatar answered Oct 19 '22 03:10

Sri Harsha Arangi


knutwalker's answer is correct. However if you are using AWS ElasticSearch and want to free storage space, this will not quite work.

On AWS the index to forgemerge must be specified in the URL. It can include wildcards as is common with index rotation.

curl -XPOST 'https://something.es.amazonaws.com/index-*/_forcemerge?only_expunge_deletes=true'

AWS publishes a list of ElasticSearch API differences.

like image 6
Steve E. Avatar answered Oct 19 '22 05:10

Steve E.


I just want to note that the 7.15 docs for the Force Merge API include this warning:

Force merge should only be called against an index after you have finished writing to it. Force merge can cause very large (>5GB) segments to be produced, and if you continue to write to such an index then the automatic merge policy will never consider these segments for future merges until they mostly consist of deleted documents. This can cause very large segments to remain in the index which can result in increased disk usage and worse search performance.

So you should shut down writes to the index before beginning.

like image 1
Noumenon Avatar answered Oct 19 '22 05:10

Noumenon