A single stray byte on an H:4 serial link hangs power_on() forever (no parser resync, no reset timeout)

[TheArkadiuszGrzelka]

A single stray byte on a serial (H:4) link can hang Device.power_on() forever.

Three separate behaviours combine into a permanent hang. Each is defensible alone; together they mean one bad byte at the wrong moment blocks startup with no timeout and no recovery.

1. A bad type byte discards the rest of the read chunk, silently

PacketParser.feed_data() (transport/common.py:143) raises InvalidPacketError from the middle of its loop, so any bytes remaining in the current data buffer are never parsed. StreamPacketSource.data_received() (transport/common.py:303-307) catches it and only logs:

def data_received(self, data: bytes) -> None:
    try:
        self.parser.feed_data(data)
    except core.InvalidPacketError:
        logger.warning("invalid packet, ignoring data")

Deterministic reproduction, no hardware:

import asyncio, logging
from bumble.transport import common
logging.disable(logging.CRITICAL)

async def main():
    got = []
    class Sink:
        def on_packet(self, p): got.append(p.hex())

    valid = bytes([0x04, 0x0E, 0x04, 0x01, 0x03, 0x0C, 0x00])  # HCI Command Complete

    src = common.StreamPacketSource(); src.set_packet_sink(Sink())
    src.data_received(bytes([0xFF]) + valid)   # one stray byte, same read() chunk
    print(got)                                 # [] -- the valid packet is gone

    src.data_received(valid)
    print(got)                                 # recovers only on the NEXT chunk

asyncio.run(main())

The packet bundled with the garbage is lost. There is no attempt to resynchronise within the chunk (e.g. skip one byte and retry framing).

2. HCI_Reset is awaited with no timeout

Host.reset() (host.py:347) does await self.send_sync_command(hci.HCI_Reset_Command()), and _send_command() (host.py:690) awaits with response_timeout=None:

response = await asyncio.wait_for(self.pending_response, timeout=response_timeout)

So if the command-complete is lost, power_on() never returns and never raises.

3. The serial transport never drains the port before parsing

open_serial_transport() (transport/serial.py:60) opens the port and starts feeding the parser without calling reset_input_buffer(). Bytes queued in the tty RX buffer before open() — e.g. a previous process left the controller advertising, or the controller is mid-packet — are delivered to the parser first.

Putting it together

If a stray byte arrives in the same read() as HCI_Reset's command-complete, (3) supplies the byte, (1) throws away the reply along with it, and (2) waits for that reply forever. Symptom: invalid packet, ignoring data repeating, power_on() never completing.

Field measurements

On an nRF52833-DK running samples/bluetooth/hci_uart at 1 Mbaud over the on-board J-Link VCOM, driving bumble.apps.auracast:

• Opening a quiet port: ~7% of attempts hang (1/15).
• Opening right after a previous process was killed while scanning (so the controller is still streaming advertising reports into the UART): 3/12 hang (25%).

We also tested the obvious fix and it does not work, so it may save you time: adding serial.reset_input_buffer() right after the port opens gave 7/12 hangs (58%) against 3/12 without it — Fisher exact two-sided p = 0.21, i.e. not significantly worse, but certainly not better. Plausibly because tcflush mid-stream truncates a packet and leaves the parser awaiting body bytes that were just discarded. Flushing before the parser cannot help while the parser itself cannot resynchronise.

Suggested direction

The durable fix looks like (1): on an unrecognised type byte, skip that byte and re-attempt framing on the remainder of the chunk, rather than discarding it. That makes any start position recoverable, which is what a byte-stream transport needs.

Independently, giving Host.reset() a bounded response_timeout would turn "hangs forever" into a clear, retryable error — valuable even after (1), since a controller can also simply not answer.

We are working around this downstream by bounding and retrying the open, but the retry only succeeds because the garbage is transient.

Happy to send a PR for the parser resynchronisation if that direction is acceptable.

[zxzxwu]

Just as you mentioned, skip the bad byte might improve the stability, but still not solve the problem completely - the next byte might just be better, worse, or the UART transport break completely. Current UART transport design is just unreliable, but I am not sure whether there is a better software approach to solve this problem.

[TheArkadiuszGrzelka]

Thanks — and you are right that byte-skipping is a heuristic. H:4 has no framing and no CRC, so any resynchronisation inside the parser can in principle lock onto a false frame. I am not going to argue for that.

But I think there is a deterministic recovery available, outside the parser. Below is a correction to my own report, a negative result, and a measured proposal.

Correction: my stated trigger was wrong

I wrote that stale bytes sat in the tty RX buffer before open(). That was a guess and it was wrong.

The actual trigger, in our case, was an application that left the controller transmitting. The port is not dirty because of the UART — it is dirty because something is still sending into it when the next process attaches.

That matters here, because apps/auracast.py does it too:

• BroadcastScanner.stop() calls device.stop_scanning() only. The periodic advertising syncs it established stay alive. They are terminated in on_broadcast_loss (apps/auracast.py:479) — i.e. only when a broadcast disappears from the air, never when the scanner is shut down.
• Device.power_off() sends nothing to the controller: it cancels a host task, calls host.flush() (which just releases the command semaphore) and clears a flag.
• create_device() never calls power_off() anyway.

So a scanner that exits while its broadcasts are still on the air leaves the controller emitting LE Periodic Advertising Report events. We confirmed this by opening the port passively right after our app exited: it streamed continuously, subevent 0x0F, indefinitely. After we terminated the syncs on every exit path, the same capture reads 0 bytes (positive control, SIGKILLed app: 6944 and 6604 bytes).

I suspect a fair number of "the UART is unreliable" reports are really "the previous process left the controller talking".

Negative result: do not flush the input buffer

The obvious fix — serial.reset_input_buffer() right after opening — does not work, and looked slightly worse:

	stalls
no flush	3/12
flush at open	7/12

(Fisher exact two-sided p = 0.21 — so not significantly worse, but certainly not a fix.) Plausible reason: tcflush discards bytes underneath a parser that is already reading, mid-packet, so it moves where you lose framing rather than removing the loss. Recording it here so nobody spends the time.

Proposal: blind reset, then drain to idle, before the parser exists

The asymmetry that makes this deterministic: the host→controller direction never desynchronises. The host only ever writes whole HCI commands, so the controller's own H:4 receiver is always on a frame boundary — no matter how badly the host's receiver is lost. A blindly written HCI_Reset therefore always executes.

And HCI_Reset stops advertising, scanning and periodic syncs, so the inbound stream must stop. Silence is then a frame boundary — proven, not guessed.

# before constructing the parser / transport
ser.write(bytes([0x01, 0x03, 0x0C, 0x00]))   # H:4 HCI_Reset
# read and discard until the port has been idle for ~100 ms
# (cap the wait; a controller that never goes quiet is a different problem)

Then open normally. The stack's own power_on() → HCI_Reset now runs on a quiet port.

Measured on an nRF52833-DK running samples/bluetooth/hci_uart at 1 Mbaud (J-Link VCOM), against a deliberately dirtied port — a SIGKILLed app left the controller scanning and PA-synced before each attempt. 48 runs, arms strictly interleaved, run as two batches of 24 that started on opposite arms to rule out an order effect:

Arm	Stalls
stock open_serial_transport	8 / 24 (33 %)
blind reset + drain-to-idle	0 / 24 (0 %)

Fisher exact, two-sided: p = 0.0039. Both batches individually 4/12 vs 0/12. Every poisoned repetition was validated (the killed process must not have run its own cleanup).

In all 24 recovery runs the port went quiet immediately: 181–692 bytes discarded (mean 469), and the 100 ms idle threshold was reached at 120–121 ms every single time — i.e. the controller stopped within ~20 ms of the blind reset, always.

Independently of the above

Host.reset() awaits HCI_Reset's command-complete with no timeout (_send_command passes response_timeout=None, host.py:690). A bounded timeout there would turn "hangs forever" into a clear, retryable error even when recovery is impossible. That one needs no heuristics at all.

Offer

I can share the probe script and the raw run logs if useful. I am not able to open a PR — contributions here need a Google CLA and that is a decision above my pay grade — so please treat all of the above as freely usable.

[barbibulle]

You're right that it is pretty easy to leave the controller in an indeterminate state where it still emits data, without an easy way to get back to a clean state with an explicit hardware-level reset (some controllers with a serial port may have that ability, with the CTS/RTS lines, but I wouldn't count on it). Even if app app performs a "clean" exit (by doing an explicit reset before exiting, for example), that can't always be guaranteed. So I agree that having a way to be more robust upon initialization is a good idea. I'll spend some time going over the proposals here and follow up shortly with more comments.

[TheArkadiuszGrzelka]

Correction: my "host→controller never desynchronises" claim was wrong

In my earlier comment I argued the blind HCI_Reset is deterministic because "the host only ever writes whole HCI commands, so the controller's own H:4 receiver is always on a frame boundary." Further measurement disproved that. A controller can be left mid-RX-payload, and in that state it swallows a blindly written HCI_Reset as four more payload bytes — the reset never executes, and the recovery I proposed fails.

How we reproduced a mid-payload controller

Separately from this issue, we found that a single serial write() larger than 64 bytes is corrupted on our J-Link VCOM link (measured on an nRF52833-DK at 1 Mbaud running samples/bluetooth/hci_uart): a 64-byte HCI command is answered, a 65-byte one is not, and the same 65 bytes split across two back-to-back writes always is. 64 is the USB full-speed bulk max packet size, so the truncation is below HCI — independent of RTS/CTS and of baud. (This is a host/transport issue, not bumble's; we worked around it by chunking serial.Serial.write at 64 bytes. serial_asyncio hands the whole buffer to one serial.Serial.write, and pyserial with write_timeout=0 issues a single os.write of the full length — so LE_Set_Extended_Advertising_Data for a ~65-byte Broadcast Audio Announcement goes out as one oversize write.)

When that oversize command is corrupted, the controller's H:4 receiver (samples/bluetooth/hci_uart/src/main.c) drops the command after its 3-byte header and returns to scanning for a packet-type byte — it then locks onto a byte inside our payload and starts reading a phantom command whose param_len can be up to 255. From then on it is waiting for up to 255 body bytes, and anything the host writes — including a blind HCI_Reset — is consumed as that payload. We confirmed this with RTT logging of the sample's own LOG_WRN("Unknown header %d", type): the "unknown header" values were exactly our payload bytes, in order.

The blind-reset proposal, completed

The blind-reset-and-drain still works for the case I originally described (a controller left transmitting), but it needs one addition to also recover a controller left mid-RX: satisfy any pending payload first. param_len is a single byte, so the controller can be waiting for at most 255 payload bytes plus a 3-byte header. Writing 3 + 255 = 258 zero bytes covers the worst case, and 0x00 is not a valid H:4 packet type, so once the parser reaches its idle state the surplus zeros are discarded rather than misread. Only then does the blind HCI_Reset land on a frame boundary.

# before constructing the parser / transport
ser.write(bytes(3 + 255))                    # satisfy any pending RX payload
ser.write(bytes([0x01, 0x03, 0x0C, 0x00]))   # H:4 HCI_Reset, now on a boundary
# then read/discard until the port has been idle for ~100 ms (capped)

(If the link itself corrupts oversize writes, as ours did, the pad must also be written in ≤64-byte chunks, or it is the very corruption it exists to clear.)

Measured on the same nRF52833-DK, poisoning the controller with an oversize command before each attempt: blind reset alone does not recover it (the reset reply never comes); blind reset preceded by the 258-byte pad recovers first-try, 3/3 rounds, 6/6 subsequent reads clean.

None of this contradicts your point that H:4 is inherently unreliable — it is a best-effort initialization heuristic, not a framing fix. But an idempotent "arrive at a known state" step at open time does seem to eliminate the hang in practice, mid-TX and mid-RX alike.

[barbibulle]

Thanks for the detailed analysis. I'll work on a PR along the lines of what you're suggesting. I will be made optional, because not all UART-based controllers will be happy with a stream of 0 padding at the start, but it will definitely help for controllers that do.