Typing Logs to ElasticSearch

Are y’all typing your log messages that have to go to structured logging outputs? (i.e. NOT CloudWatch)

I just found out we’re required to push our logs to ElasticSearch. I was using LambdaLog which I was just adding a timestamp. I found it has a toJSON() method, so just shoved that in my elasticsearch.index method call, and voila, logs going to be CloudWatch and (I guess most of the time, lelz) to ElasticSearch.

… however, I’d occasionally get errors. Turns out, ElasticCache kind of builds a schema from your index, and when you create the index pattern, it locks in place. So if I start shoving random JSON data, it fails to log it because it’s like “You don’t match the schema, bruh”.

So I can either keep my log messages as whatever<'a>, or think about more structured logging and just use that same type each time so ReScript doesn’t have to use generics, and ElasticCache is happy. Is typing your log JSON structure messages a thing, or…???

I think it can definitely be useful, maybe take some inspiration from this ocaml lib:

2 Likes

The fun part is, I think hcarty was working at Elastic when he wrote this.

1 Like