Add timestamp to exported metrics

[ahus1]

The graphite exporter keeps the metrics for up to 5 minutes. It even records the time in Timestamp. But it doesn't export the time it received the metrics to Prometheus.

I would like the graphite_exporter to export the time when it received a metrics like the push gateway.

This would help a setup where Prometheus scrapes the graphite_exporter every 10 seconds, but new data is pushed to graphite in a different timing (i.e. 30 or 60 seconds).

[brian-brazil]

This has intricate dependencies elsewhere.

I would like the graphite_exporter to export the time when it received a metrics like the push gateway.

The push_gateway does not do this.

[ahus1]

Hello @brian-brazil - thank you for the fast reply. Apparently I've been mistaken about the push_gateway.

But isn't the timestamp intended to be used for something like this? Can you elaborate on the intricate dependencies?

[ahus1]

... I've read the "about timestamps" for the push gateway.

The only way to do it without a timestamp would be to align the the pushing to graphite with the scraping of prometheus, and to set the expiry time in the graphite_exporter to i.e. 2-3x the push/pull interval?!

[matthiasr]

I think adding timestamps is not wrong in this case – we are bridging from graphite's understanding of time to prometheus'. This is different from the Pushgateway use case. We just need to be clear that you can't use this to backfill like you can in graphite.

Could you please be more specific about the "intricate dependencies"?

[brian-brazil]

That was a reference to staleness handling. Lack of timestamp support in client_golang would be the main blocker now.

[matthiasr]

Another thought – with timestamps, what use is the staleness delta? I mean, it is still nice to remove metrics from memory eventually but would it make a difference logically, if the last sample just stays at the last timestamp?

[brian-brazil]

I assume you meant the sample expiry. You still want to put a bound on memory growth, and having data that's too old won't work well on the Prometheus end.

[ahus1]

@matthiasr - I liked the fact that the data of this exporter removed data eventually from memory. This makes it absolutely maintenance-free. Otherwise getting rid of obsolete metrics would require a manual purge or restart.

Regarding timestamps - yes, I would still like that.

[matthiasr]

So the best case is the combination of both, but right now the client library doesn't let us set timestamps. I think in this case we can only wait …

…

On Thu, Jan 18, 2018, 20:49 Alexander Schwartz ***@***.***> wrote: @matthiasr <https://github.com/matthiasr> - I liked the fact that the data of this exporter removed data eventually from memory. This makes it absolutely maintenance-free. Otherwise getting rid of obsolete metrics would require a manual purge or restart. Regarding timestamps - yes, I would still like that. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#29 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAICBiaMXfW03eyep15vcufwazRhJigjks5tL5_MgaJpZM4NfkY3> .

[naveenyallane]

We are facing similar issue with Azure exporter (https://github.com/RobustPerception/azure_metrics_exporter).
We observed 5 minutes gap from the metric that is exported to prometheus and actual metric in the Azure monitor.

[brian-brazil]

Please keep on topic, this issue is about the Graphite exporter. The relevant support has been in client_golang for a while.