Deprecated tables in Logtail

Where do these come from?
You may have noticed that new sources appeared in your account, with their names having a (deprecated) suffix. Don't worry - that's expected and completely fine!
We recently rolled out a new version of our internal storage for Logtail sources, and the deprecated sources are the ones holding your previous data. Your new data will be ingested into the regular sources. Live Tail, Views, Alerts as well as S3 Backups are migrated automatically and work with the two source types out of box.

What changed in Logtail

To make your Live Tail and Grafana queries faster, we updated the internal structure of our ClickHouse tables. Most functions remain the same, but there are a couple changes that you may come across with the new sources:

Columns are no longer suffixed with their types

To simplify accessing your data, we decided to drop the suffixes altogether. This means that when you previously queried for message_string:"Hello, world!" in Live Tail, you will query for message:"Hello, world!" now.

Copying to JSON maintains structure more closely to the input data

One of the frequent issues with Logtail was that we scrambled the JSON structure you send us. With the changes introduced here, we updated the Copy to JSON functionality to be more truthful to the source data.

Logs are no longer out of order

We updated our handling of logs with the same timestamps, and we will maintain the order of the logs you send us. No more shuffled stack traces!

Optional speed-ups using materialized columns

We strive for our base query performance to be blazing fast, but if you query for a specific attribute very often, we added a Materialized Columns section to the Advanced settings of your sources.
When you add a materialized column, we automatically optimize queries filtering by this column, leading to massive speed gains. Materialized columns pair very well with our Views, too!
Note that due to performance constraints, we have a limit on the number of materialized columns you can add to an individual source.
Find materialized columns in the Advanced settings of your sources!

Grafana SQL queries changed a little

To access your columns in your custom Grafana dashboards, you need to tweak the SELECT queries a little. Here's an example, and I'll share more about the underlying structure below:
Previous Grafana query:
SELECT level_string
FROM db_123.test_source
WHERE level_string IN ('ERROR', 'WARN')
New Grafana query:
SELECT JSONExtractString(json, 'level') AS level
FROM db_123.test_source
WHERE level IN ('ERROR', 'WARN')
As you can see in the query, you need to use the JSONExtractString ClickHouse function to access individual columns. That's because instead of flattening and storing your payloads into individual columns, we centralize them in the json column now. The advantage is that we now don't have to limit the number of individual attributes you send to Logtail - addressing a pain point of many of you.
Note that if you have any materialized columns added to the source, you can query those directly, without the JSON extract function, for example:
SELECT level
FROM db_123.test_source
WHERE level IN ('ERROR', 'WARN')
As you can see, using materialized columns simplifies the raw SQL querying as well - make sure to use them whenever appropriate!

Any questions?

We understand this is a large change, but a necessary one - with the experience of using Logtail being our priority.
If you have any questions about these changes, want to dive deeper into the technical side of things, or want to discuss anything else, please don't hesitate to contact us at [email protected]