Backing up and Restoring ES Indices Using Curator
The best way to manage elasticsearch indices is to use Curator. It ships with both API and CLI tool. For this article we are going to use the CLI. Curator has a huge list of features but we are going to focus on snapshot and restore for this article.
First things first, lets install the curator. You can easily do that by running the command below
pip install elasticsearch-curator
Setting things Up
Once the curator is installed, we need to give it a path where it will store the backups. In order to do that, open up the file
/etc/elasticsearch/elasticsearch.yml and add following
Make sure that the path that you set above is valid, otherwise you will get exception.
Next up, we need to setup the config file for curator that is
curator.yml. By default the curator assumes that the config file is placed at
~/.curator/curator.yml. But we can override this by passing our custom location using the
--config option. For now, let’s create the file at the default location.
~/.curator/curator.yml file and and the below
client: hosts: - "ES_HOST" port: "ES_PORT" use_ssl: False ssl_no_validate: False timeout: 30 logging: loglevel: CRITICAL
Make sure to replace
ES_PORTwith your elasticsearch host and port.
curator.yml is configured, we need to create a repository. Repository is something that will be holding our backup. In order to do that, run the below command
es_repo_mgr create fs --repository es_snapshot_name --location es_snapshot_path --compression true
Make sure to replace
es_snapshot_namewith your backup name and
es_snapshot_pathwith the path that you assigned earlier to
Now that we have the setup read, the next step is to create our action file that will be used to create the backup. This file contains the three key parts i.e.
Actions are the tasks which curator can perform on your indices. These actions will run in sequence and each action is assigned a unique number.
Options are settings used by actions. For complete list of actions check following.
Filters are the way to select only the indices you want. Check following link for more detail.
Here is how our action file looks like
actions: 1: action: snapshot description: "Create snapshot" options: repository: "es_snapshot_name" continue_if_exception: False wait_for_completion: True filters: - filtertype: pattern kind: regex value: ".*$"
Lets tear down the above
action_backup.yml file. There’s only one action and this action is responsible for taking the backup. Replace
es_snapshot_name with the name that you used earlier while running
es_repo_mgr command. In the filters, I have used the filter type
pattern which will only get the indicies matching the specified pattern. Now this filter type pattern have different kinds. The kind that we have specified here is
regex and since we want to match all the indices, we are going to use the value of
action_backup.yml is ready then run below command to take backup
In order to restore our backup, put the below in
actions: 1: action: close description: "Close indices before restoring snapshot" options: continue_if_exception: True ignore_empty_list: True filters: - filtertype: pattern kind: regex value: ".*$" 2: action: restore description: "Restore all indices in the most recent snapshot with state SUCCESS" options: repository: "es_snapshot_name" name: indices: wait_for_completion: True filters: - filtertype: state state: SUCCESS 3: action: open description: "Open indices after restoring snapshot" filters: - filtertype: pattern kind: regex value: ".*$"
As you can see, the restore action file has 3 actions. First action with number
1 is for closing the indices so that we can halt any actions that are being performed in order to proceed with the restore. In task numbered
2 we are doing the actual restore and in task
3 we are re-opening the closed indices so that we can use them for search.
Now our backup file is ready, so use following command to restore the backup
And that about wraps it up. If you would like to learn more, find the original docs here. Feel free to leave your comments and feedback down below.