Using Lua filter to find if log matches a pattern, and remove last n characters.

Original log: [1646213018.112301921, {"log"=>"testtest1-"}

Changed log : [1646213018.112301921, {"log"=>"testtest1"}

---

    [FILTER]
        Name    lua
        Match   test.*
        script  checkend.lua
        call    check_end

If you use lua filter, you can find "If a log/word matches a specific pattern", and do some works. 

ex) if the condition suits, remove last n characters.

I wrote on those codes on fluentbit configmap, so there is tab&space in front area. 

  checkend.lua: |
    function check_end(tag, timestamp, record)
        if string.match(record["col"], '(.*)-$') then
            record["container_name"] = string.sub(record["col"], 1, -2)
            return 2, timestamp, record
        end
    end

Wrote a function that use string.match(words, pattern). so that you can check if log matches specific patterns.

In my case, I checked if log ends with -. 

if that is(If log matches with the pattern, the value is true. 

so I can go on with if statement.

Next, I used string.sub(word, start index, end index) to substract last n words.

* Lua's index starts with 1, not 0.

In my case, I wanted to remove last 1 words so I used -2(so that I can get results except last 1 character.)

return 2, timestamp, record means "the original timestamp is not modified and the record has been modified so it must be replaced by the returned values from record (third return value). The code 2 is supported from v1.4.3."

https://docs.fluentbit.io/manual/pipeline/filters/lua