Filebeat : data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path

Question

Looking at the logs in one of the filebeat pods i can see this:

2021-01-04T10:10:52.754Z        DEBUG   [add_cloud_metadata]    add_cloud_metadata/providers.go:129     add_cloud_metadata: fetchMetadata ran for 2.351101ms
2021-01-04T10:10:52.754Z        INFO    [add_cloud_metadata]    add_cloud_metadata/add_cloud_metadata.go:93     add_cloud_metadata: hosting provider type detected as openstack, metadata={"ava
ilability_zone":"us-east-1c","instance":{"id":"i-08f536567bd9945df","name":"ip-10-101-2-178.ec2.internal"},"machine":{"type":"m5.2xlarge"},"provider":"openstack"}
2021-01-04T10:10:52.755Z        DEBUG   [processors]    processors/processor.go:120     Generated new processors: add_cloud_metadata={"availability_zone":"us-east-1c","instance":{"id":"i-08f5
36567bd9945df","name":"ip-10-101-2-178.ec2.internal"},"machine":{"type":"m5.2xlarge"},"provider":"openstack"}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]]    
2021-01-04T10:10:52.755Z        INFO    instance/beat.go:392    filebeat stopped.
2021-01-04T10:10:52.755Z        ERROR   instance/beat.go:956    Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (pat
h.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

as you can see the filebeat stopped with an error :

data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

After searching the problem in github/forum i found this :
https://discuss.elastic.co/t/data-path-already-locked-by-another-beat/219852/4

Which looks like my problem, Im using the default filebeat-kubernetes.yaml , and there is no information in ELK / Filebeats docs on how to add unique paths in the filebeat-kubernetes.yaml
where do i add them and how do i make them unique? Thanks

Answer 1

I had the same problem. It means that your data path (/var/lib/filebeats) are locked by another filebeat instance. So execute sudo systemctl stop filebeat (in my case) to ensure that you don't have running filebeat and then run filebeat with sudo filebeat -e which prints logs in console

I also tried link, that you shared, but it didn't help me. Here another solutions, may be it would help you: https://discuss.elastic.co/t/data-path-already-locked-by-another-beat/219852/2

Answer 2

In addition to @Anton's answer, In one of the scenarios, I had a lock file in the data path. This could be /var/lib/filebeat/filebeat.lock depending on the configuration. Delete the file and run sudo filebeat -e