docs(observability): add actor application telemetry best practices

[Da Huang (git286)]

The observability guide covers telemetry emitted by Substrate's own control plane, but says nothing to actor authors whose application code emits OpenTelemetry metrics and spans. Those signals sit in the OTel SDK's in-memory buffers (PeriodicReader accumulators, BatchSpanProcessor queues) and are subject to actor lifecycle events in ways a normal long-lived process never is.

Add a section documenting the actual behavior:

• onPause: Full preserves the buffer; onPause: Data discards it, since process memory is excluded from the snapshot and the containers are deleted at the end of every checkpoint.
• The workload receives no SIGTERM and no pre-suspend callback, so a flush can only happen where the actor itself initiates the suspend.
• On resume, gVisor's CLOCK_MONOTONIC includes the suspended interval, so overdue export timers fire immediately.
• Metric data points are stamped at collection time and so are attributed to the resume; span timestamps are recorded when the span ran and so go stale across a long suspension.
• The OTLP connection does not survive a pod migration.

Along with the corresponding recommendations: flush before self-suspend, stay on onPause: Full, shorten the metric and span export timers separately, configure gRPC keepalives, and use a DurableDir volume under onPause: Data.

Known gaps are tracked in #450 (PreSuspend/PostResume hooks), #23 (graceful termination) and #503.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[Eitan Yarmush (EItanya)]

I think these notes will be important, but I think it's worth waiting until we have some specific direction on the self-suspend and graceful-suspend before putting this doc out there as it will be superseded by that

[Da Huang (git286)]

Thanks — I want to push back gently on the timing.

We discussed this with sfunkenhauser and there's currently no plan for #450. So "wait for direction" isn't a short hold, it's indefinite — and in the meantime anyone running a workload on Substrate is silently losing telemetry with no documented way to find out why. The purpose of this section isn't to describe the eventual design; it's to tell people what they can do today (stay on onPause: Full, shorten both export timers, flush before self-suspend) so their telemetry works at all in the interim.

I'd also note that only part of this gets superseded. The mitigations do. But the behavioral notes don't — the CLOCK_MONOTONIC jump across restore, metric points being stamped at collection time while span timestamps go stale, and the dead OTLP connection after migration are gVisor and OTel SDK properties that hold no matter what we land in #450/#23.

I've pushed 8773431, which adds an explicit "interim guidance" callout at the top of the section naming #450 and #23, so nobody mistakes these for a stable contract. I'll own the follow-up PR once those have direction. Does that address the concern?