Add @timestamp field in ElasticSearch with Python

Question

I'm using Python to add entries in a local ElasticSearch (localhost:9200)

Currently, I use this method:

def insertintoes(data):
"""
Insert data into ElasicSearch
:param data: dict
:return:
"""
timestamp = data.get('@timestamp')
logstashIndex = 'logstash-' + timestamp.strftime("%Y.%m.%d")
es = Elasticsearch()
if not es.indices.exists(logstashIndex):
    # Setting mappings for index
    mapping = '''
        {
            "mappings": {
                  "_default_": {
                    "_all": {
                      "enabled": true,
                      "norms": false
                    },
                    "dynamic_templates": [
                      {
                        "message_field": {
                          "path_match": "message",
                          "match_mapping_type": "string",
                          "mapping": {
                            "norms": false,
                            "type": "text"
                          }
                        }
                      },
                      {
                        "string_fields": {
                          "match": "*",
                          "match_mapping_type": "string",
                          "mapping": {
                            "fields": {
                              "keyword": {
                                "type": "keyword"
                              }
                            },
                            "norms": false,
                            "type": "text"
                          }
                        }
                      }
                    ],
                    "properties": {
                      "@timestamp": {
                        "type": "date",
                        "include_in_all": true
                      },
                      "@version": {
                        "type": "keyword",
                        "include_in_all": true
                      }
                    }
                  }
            }
        }
    '''
    es.indices.create(logstashIndex, ignore=400, body=mapping)

es.index(index=logstashIndex, doc_type='system', timestamp=timestamp, body=data)

data is a dict structure with a valid @timestamp defined like this data['@timestamp'] = datetime.datetime.now()

The problem is, even if there is a timestamp value in my data, Kibana doesn't show the entry in «discovery» field. :(

Here is an example of a full entry in ElasicSearch:

{ "_index": "logstash-2017.06.25", "_type": "system", "_id": "AVzf3QX3iazKBndbIkg4", "_score": 1, "_source": { "priority": 6, "uid": 0, "gid": 0, "systemd_slice": "system.slice", "cap_effective": "1fffffffff", "exe": "/usr/bin/bash", "hostname": "ns3003395", "syslog_facility": 9, "comm": "crond", "systemd_cgroup": "/system.slice/cronie.service", "systemd_unit": "cronie.service", "syslog_identifier": "CROND", "message": "(root) CMD (/usr/local/rtm/bin/rtm 14 > /dev/null 2> /dev/null)", "systemd_invocation_id": "9228b6c72e6a4624a1806e4c59af8d04", "syslog_pid": 26652, "pid": 26652, "@timestamp": "2017-06-25T17:27:01.734453" } }

As you can see, there IS a @timestamp field but it doesn't seems to be what Kibana expects.

And don't know what to do to make my entries visible in Kibana.

Any idea ?

Answer 1

Elasticsearch is not recognizing @timestamp as a date, but as a string. If your data['@timestamp'] is a datetime object, you can try to convert it to a ISO string, which is automatically recognized, try:

timestamp = data.get('@timestamp').isoformat()

timestamp should now be a string, but in ISO format