Warm tip: This article is reproduced from serverfault.com, please click
elasticsearch logstash

logstash

发布于 2020-11-28 10:12:15

I have the following json file, each line is a diferent json:

{"s":"some address","c":"some city"}
{"s":"some address1","c":"some city1"}
{"s":"some address2","c":"some city2"}

I have the following job:

input {
    file {
        start_position => "beginning"
        path => "/sources/someFile.txt"
    }
}

filter {
    json {
        source => "a"
        target => "addresses[0].street"
    }
    mutate {
        remove_field => ["message", "@timestamp", "host", "path", "@version"]
    }
}

output {
    elasticsearch {
        hosts => "http://elasticsearch:9200"
        index => "store"
    }
}

I want to write to to the index as the following (each address go to a different doc as the forst element in an array):

{
   "addresses": [{"street" : "some address", "city" : "some city"}]
}

{
   "addresses": [{"street" : "some address2", "city" : "some city1"}]
}

{
   "addresses": [{"street" : "some address3", "city" : "some city2"}]
}

The attached job is not working. no error and not doing anything.
Thanks

Questioner
SexyMF
Viewed
0
分享
Badger 2020-11-28 22:35:07

You cannot use that field reference in the target option of the json filter. In any version of logstash from the last couple of years I would expect that to result in a _jsonparsefailure tag and the error

Exception caught in json filter {:exception=>"Invalid FieldReference: `addresses[0].street`"

If you change the reference to be [addresses][0] then it will run without error, but the reference will be interpreted as the "0" entry in the "addresses" hash, not the first entry in the addresses array.

Your incoming JSON has the wrong field names, so you will have to rename the fields. I think it is easiest to do it in a ruby filter

json { source => "message" target => "[@metadata][json]" }
ruby {
    code => '
        json = event.get("[@metadata][json]")
        event.set("addresses", [ { "street" => json["s"], "city" => json["c"] } ] )
    '
}

which produces

 "addresses" => [
    [0] {
          "city" => "some city",
        "street" => "some address"
    }
],

The original JSON is placed inside the [@metadata] field so that is available but not indexed by the output.

SexyMF 2020-11-28 17:25:00

Thanks. can you try and put it togehter with my original script. I can't make it work. thanks

Badger 2020-11-28 18:16:36

The json and ruby filter I provided should just replace the json filter you have. A file input will put the text of the line of the file in the field [message], not [a]. If you had previously run logstash it will have recorded how much of the input file it read in the sincedb, so you will need to append new lines to the file to get logstash to read them.

热门帖子

1
求推荐硬路由（1000 内）
2
学英语的朋友们可以来看看 https://langplayground.top/
3
华中科大的计算机技术的在职硕士，职场认可度如何？
4
十兆宽带 100 块钱一个月我是活在 80 年代吗？
5
关于高质量 屏幕共享的问题
6
我是真的受不了打小报告的三八了！
7
怎么管理自己的文件体系
8
[开发者自荐] AirBattery: 在 Mac 端获取所有设备的电量并显示在 Dock 或状态栏上
9
南京移动 IPV6 被停
10
WireGuard 跨国组网失败后,一个新工具的诞生

热门github

1
Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information
2
A Windows and Office activator using HWID / Ohook / KMS38 / Online KMS activation methods, with a focus on open-source code and fewer antivirus detections.
3
Get up and running with Llama 2, Mistral, Gemma, and other large language models.
4
该项目可以让你通过订阅的方式使用Cloudflare WARP+，自动获取流量。This project enables you to use Cloudflare WARP+ through subscription, automatically acquiring traffic.
5
Multi functional app to find duplicates, empty folders, similar images etc.
6
Xray panel supporting multi-protocol multi-user expire day & traffic & ip limit (Vmess & Vless & Trojan & ShadowSocks & Wireguard)
7
The Free Software Media System
8
lightweight, standalone C++ inference engine for Google's Gemma models.
9
📚 Freely available programming books
10
A collective list of free APIs
11
1️⃣🐝🏎️ The One Billion Row Challenge -- A fun exploration of how quickly 1B rows from a text file can be aggregated with Java
12
🎓 Path to a free self-taught education in Computer Science!
13
Curso para aprender el lenguaje de programación Python desde cero y para principiantes. 75 clases, 37 horas en vídeo, código, proyectos y grupo de chat. Fundamentos, frontend, backend, testing, IA...
14
This repository contains System Design resources which are useful while preparing for interviews and learning Distributed Systems
15
Mamba is a new state space model architecture showing promising performance on information-dense data such as language modeling, where previous subquadratic models fall short of Transformers. It is based on the line of progress on structured state space models, with an efficient hardware-aware design and implementation in the spirit of FlashAttention.