[BUG] silently swallowed RPC calls

[colmaengus]

Bug Description

Summary: what look like invisible stale registrations are swallowing RPC calls

Expected Behavior:
We have a node.js agent that creates 10 sessions and registers the same handler on each using round robin. Our API server makes RPC calls using the registrations as part of its API handling. Normally this works just fine and all API calls are handled correctly.

Actual Behavior:
In one deployment we're seeing that API calls don't complete and are silently swallowed. What we think is happening, however unlikely is that the crossbar router has some "invisible" registrations that don't show up when we query using wamp.registration.list or wamp.session.list but that somehow prevent some API calls from completing. We've seen for example that when we change the number of agent sessions to 5 we see a 5/10 success/failure pattern of API calls. Another thing we tried was to change the agent from round robin to random and we get and "wamp.error.procedure_exists_with_different_invocation_policy" error even though there are no reported registrations.
During our next maintenance window we plan to restart the crossbar router and we're expecting the problem to go away at least for a while. I suspect the behavior might be caused by something in the connection between our lab network and Azure but I don't have any evidence of this yet. But even so I would expect that the ping/pong application level keep alive in the crossbar router would deal with any TCP related socket issues.

Unfortunately we've not found a way to reproduce the problem so I'm hoping that something in the above description rings a bell.

I also want to take a moment to thank you very much for your amazing software. We've been using crossbar and autobahn as the real-time cornerstone of our device farm software for many years and they've always been rock solid. Its very exiting to see them under active development again. Long may it continue.

Environment
We are using crossbar 25.12.1 with autobahn 25.12.2 with python 3.14.5 on Debian Bookworm. The clients are using node.js.

[colmaengus]

a small update: we did a test with our agent disconnected and now all our API calls are being swallowed. This would agree with the thinking that the crossbar router somehow has to zombie registrations.

[oberstet]

Hi @colmaengus 👋

First off — thank you for the very kind words, and for taking the time to write up such a clear and thoughtful report. Hearing that Crossbar.io and Autobahn have been the rock-solid real-time cornerstone of your device-farm software for years genuinely makes our day. And yes — to answer the implicit question — the project is very much under active development again, and we intend to keep it that way. Long may it continue indeed. 🚀

Reports like yours, from people running it hard in production, are exactly what make the codebase better, so this is very welcome.

A few questions, so we get this right

Before I say anything about the bug itself, I'd love to learn a bit more about you and how you're using Crossbar — both because it helps us prioritize and reproduce, and frankly because we're always curious who's out there building on WAMP:

• Who are you / what are you building? You mentioned a "device farm" — can you share a bit about the company/org and the use case? (Totally fine to keep it high-level if it's sensitive.)
• Topology: Single router process, or a cluster / router-to-router (rlink) setup? You mention a "lab network ↔ Azure" link — is the router on one side and the Node.js agent on the other, with the link in between? Where does the API server sit relative to the router?
• Transport: WebSocket (ws/wss) or RawSocket between the agent and the router? Any proxies, load balancers, or NAT/firewall idle-timeouts on that lab↔Azure path?
• Numbers: When you say "10 sessions," are those 10 separate WAMP sessions over 10 transports, or multiplexed? And roughly how long does a deployment run before you start seeing the silent swallowing?

On the bug itself — what I think, and what I don't want to do yet

Your instinct is a good one, and the symptoms hang together in a way that points at a real, plausible failure mode:

1. The N/2N success/failure pattern under round-robin is the strongest clue. Round-robin dispatch cycles through the observers (callees) registered for a procedure. If some of those observers are stale — i.e. the underlying transport is gone but the dealer still has the session registered as a callee — then every Nth call gets routed into a black hole and is silently swallowed, while the calls that land on live callees succeed. Reducing to 5 sessions giving you ~5/10 is exactly the shape you'd expect if roughly half the registered callees are dead-but-still-registered.

2. The wamp.error.procedure_exists_with_different_invocation_policy when switching to random is the second clue, and it's consistent with the first: the registration object still exists in the dealer (with its original roundrobin invocation policy), so a new random registration on the same URI is correctly rejected — even though the callee sessions backing it look gone to you.

3. The "invisible" part — nothing showing in wamp.registration.list / wamp.session.list — is the most interesting and the part I most want to pin down. The meta-API procedures read from the dealer's live registration map and session table, so in principle a registration that's still affecting dispatch should show up there. If it genuinely doesn't, that's a discrepancy worth chasing carefully.

So my honest take: yes, there may well be a metadata/registration-cleanup issue inside the router — most likely around what happens when a callee's transport drops ungracefully (e.g. a silently dead TCP connection across that lab↔Azure link) rather than via a clean GOODBYE/UNREGISTER. That's the classic gap: a clean detach runs the full cleanup path in the dealer (drops the observer, deletes the registration when the last callee leaves, fires wamp.registration.on_unregister/on_delete), but a connection that's wedged-open at the TCP layer may keep the session — and its registrations — alive in the router until something actually notices the transport is dead.

That's also why your point about keepalives is the right thread to pull: WebSocket ping/pong auto-ping/auto-timeout settings on the router's transport are what's supposed to detect and tear down these zombie connections. If they're not configured (or not aggressive enough relative to the network), a half-open connection can linger for a long time, and during that window its registrations are exactly the "invisible stale registrations swallowing calls" you describe.

But — and this is the important part — I'm not aware of a specific, confirmed bug here, and this codebase is complex enough that I don't want to start changing dealer/registration code based on a plausible-sounding theory. That's how regressions get born. So my strong preference is:

First, write a new integration test that proves there is an issue — by reproducing the silent-swallow under round-robin when a callee disconnects ungracefully — added alongside the integration tests we already have. Only once we have a red test do we touch the router.

We already have a functional-test harness that spins up a real Crossbar router and drives real WAMP sessions against it (test/functests/cbtests/, e.g. test_cb_rpc.py), which is the right place for this. The test I have in mind would:

1. Register the same procedure on N callee sessions with invoke: "roundrobin" (shared registration).
2. Confirm round-robin works and all N calls succeed.
3. Forcibly kill the transport of a subset of callees without a clean GOODBYE/UNREGISTER (simulating the dead lab↔Azure link).
4. Drive a batch of calls and assert that none are silently swallowed — i.e. every call either hits a live callee or fails loudly with a proper error, and assert that wamp.registration.list reflects reality after the transport drop.
5. Probe the random-policy path too, to reproduce the spurious procedure_exists_with_different_invocation_policy.

If that test goes red, we've localized a real bug and can fix it with confidence (and the test guards against regressions forever). If it stays green, that tells us the issue is more likely in transport keepalive configuration / the network path, and we redirect the investigation there.

What would help us most right now

To make that reproduction faithful, anything you can share:

• Your router config (the relevant transport + realm/role bits — redact secrets), especially any WebSocket auto_ping_interval / auto_ping_timeout / max_message_size settings on the transport the agent connects through. (If there are no auto-ping settings, that alone is very telling.)
• The agent's registration options — the exact invoke policy and how the round-robin is set up across the 10 sessions.
• Whether, when the problem is occurring, the affected callee sessions still show as connected on the OS/network level (e.g. ss/netstat on the router host showing ESTABLISHED sockets that the application thinks are gone).
• Crossbar/router logs at the time of failure (even debug-level around session detach / transport close would be gold).

For the record, the environment you're on — Crossbar 25.12.1 / Autobahn 25.12.2 / Python 3.14.5 / Debian Bookworm — is current and well within what we're actively testing, so this isn't a "please upgrade first" situation.

Thanks again for the report and the encouragement. I'll start sketching the integration test described above; if you can get me the config and logs, we can converge on this much faster.

— and welcome to the repo! 🙏

[oberstet]

@colmaengus — following up with the concrete plan we'd like to propose, and we'd genuinely like your input before we write any code.

As mentioned, given how complex the router internals are, we don't want to change anything based on a plausible-sounding theory. The first deliverable is a new integration test that proves the failure exists — added alongside our existing functional tests — that reproduces the silent-swallow under a round-robin shared registration when a callee's transport dies ungracefully. Only once that test is reliably red do we touch the dealer/registration code.

Here's how we're thinking about it.

Why "ungraceful" is the whole game

The key realization is that the obvious ways to drop a client don't reproduce your symptom:

• A clean GOODBYE/UNREGISTER, or even a kill -9 / killing the client container, all cause the kernel to send a FIN/RST. The router sees it, the transport's connectionLost fires, and the dealer runs its normal cleanup — the registration is removed correctly. No zombie.
• Your "invisible stale registrations" can only occur if the router never learns the peer is gone — i.e. packets are silently dropped in the path (a dead link, an expired NAT/firewall state, a withdrawn route). No FIN, no RST, no connectionLost. The session — and its registrations — linger until something else notices: TCP keepalive, or the WebSocket application-level ping/pong (auto_ping_interval / auto_ping_timeout).

That last point is exactly the thread you pulled on: if those keepalive timeouts aren't configured (or aren't aggressive enough for the network), a half-open connection can survive long enough to keep swallowing every Nth round-robin call — and to reject a random re-registration with procedure_exists_with_different_invocation_policy, because the old roundrobin registration is still alive in the dealer.

So the test must reproduce a network-layer blackhole, not a process death.

Failure taxonomy we want to cover

Scenario	What it exercises
Graceful GOODBYE/UNREGISTER	normal cleanup path (control case)
App crash (kill -9 / container kill)	kernel-sent FIN/RST → detach (control case, ≠ zombie)
Half-open / blackhole	the suspected bug: zombie registration swallowing calls
Stalled / "infinitely slow"	auto_ping_timeout and backpressure handling

How we'll inject the faults

We want to use standard, community-proven fault-injection tooling, not a bespoke ad-hoc proxy. The CI/CD already runs in Docker (Compose/Podman) both locally and on GitHub Actions, so the tooling has to work cleanly there.

Phase 1 — single router, many clients: Toxiproxy

This is the minimum, and it's enough to prove the bug:

• A simple Docker Compose topology: Crossbar router + Toxiproxy + a test driver.
• We interpose one Toxiproxy per callee session, so we can fault a subset of sessions independently. This directly reproduces your N/2N pattern — blackhole half of the callees and watch half the round-robin calls vanish.
• Faults are commanded from the test at runtime via Toxiproxy's API:
  • timeout toxic → hold the socket open but stop forwarding = a true half-open / zombie connection.
  • bandwidth≈0 / latency → "infinitely slow" connection.
  • reset_peer / slow_close → contrast cases.
• The test then asserts:
  1. no call is silently swallowed — every call either reaches a live callee or fails loudly with a proper error;
  2. after auto_ping_timeout fires, wamp.registration.list / wamp.session.list reflect reality (no invisible stale registrations);
  3. the random re-registration path no longer spuriously errors once stale registrations are gone.

This fits our existing functional-test harness (which already drives a real router with real WAMP sessions and real, short wall-clock waits), so it's not a new class of test fragility.

Phase 2 — multi-router / router-to-router (rlink), later

Toxiproxy can still fault a specific inter-router link (an rlink is just another WAMP transport we can route through a proxy). But once we want node-/path-level conditions across a router mesh — realistic WAN characteristics (latency + jitter + loss, à la a lab↔Azure link), or clean partitions of a group of nodes — a TCP-stream proxy is the wrong layer and doesn't scale (O(N²) proxies). For that we'd add tc/netem (likely via pumba for Docker ergonomics) for packet-level emulation and partitions, keeping Toxiproxy for targeted single-link faults.

We're flagging Phase 2 only so the direction is clear — the bug in this issue should be reproducible entirely within Phase 1.

What we'd love from you before we start coding

So the reproduction faithfully matches your deployment:

• Round-robin / registration setup: the exact invoke policy and how the 10 sessions share the registration. Anything non-default (concurrency limits, force_reregister, prefix/wildcard match)?
• Transport & keepalive config: the relevant router transport block (redact secrets) — especially whether auto_ping_interval / auto_ping_timeout are set, and to what. If they're unset, that alone is a strong lead.
• Topology: where the router, the Node.js agent (10 sessions), and the API server sit relative to the lab↔Azure link. WebSocket or RawSocket? Any proxies / load balancers / NAT with idle timeouts in the path?
• Observation: when the problem occurs, do the affected callee sockets still show ESTABLISHED on the router host (ss/netstat) even though the app thinks they're gone? That would confirm the half-open hypothesis on the spot.
• Anything in this plan that doesn't match how your system actually behaves — that's the most useful feedback of all.

If the plan looks right to you (or once you've poked holes in it), we'll write the Phase 1 Toxiproxy test and report back whether it goes red. Thanks again — really appreciate you engaging on this.

[colmaengus]

@oberstet thanks so much for spending time on this issue. Some more input from my side

we are using websocket transport with the router configured using the following

{
  "compression": {
    "deflate": {
      "request_no_context_takeover": false,
      "request_max_window_bits": 11,
      "no_context_takeover": false,
      "max_window_bits": 11,
      "memory_level": 4
    }
  },
  "enable_webstatus": false,
  "max_frame_size": 20971520,
  "max_message_size": 20971520,
  "auto_fragment_size": 65536,
  "fail_by_drop": true,
  "open_handshake_timeout": 2500,
  "close_handshake_timeout": 1000,
  "auto_ping_interval": 10000,
  "auto_ping_timeout": 5000,
  "auto_ping_size": 12
}

our node.js agent is connecting using 10 connections with the following settings

  {
    autoping_interval: 10,
    autoping_timeout: 5,
    max_retries: 0    
  }

when any of the connections close we restart out agent

registrations are done using { match: 'wildcard', invoke: 'roundrobin' }

I'll need to get back to you on what's between the router running on Azure k8s and the node.js agent running in the lab.

One thing to note is that we are using ws not native websockets on the node.js agent. So we are using autobahn-js 22.11.1 with the following modifications

• changed detection from ("WebSocket" in global) to (typeof window !== 'undefined') to allow us to continue to use ws
• add support for passing agent and auth options down to the websocket transport. This is to facilitate agent operation behind proxies and also to include our http basic authentication

I would like to move to using an unmodified autobahn-js but I need to figure out a way to handle the agent/authentication requirement. For switching from ws to native websockets I'm wondering is the application level autoping still in place when using websockets? If not what is recommended to detect sockets that don't properly close?

An additional piece of information on our problem is that once the registrations are stale they doesn't go away, until we assume when we reboot the router. So its not that its just a small window between the TCP connections having a problem and the autoping detected this.

[colmaengus]

We regard to our use case, we (Accenture) have built a device farm solution which allows automotive testbenches to be more efficiently shared between teams. The testbenches can be located anywhere in the world, only requiring an outbound HTTPS connection to the server. We then use WAMP to send commands from a browser application to the remote testbenches. Mostly these commands are used to establish websocket connections between the browser and the testbench over which we can then stream video, terminals, logs etc.

One of the main advantages of our solution is the ability to build a distributed device farm only requiring outbound HTTPS connections and the reason we are able to do this is because we are using WAMP.

Our solution is not open source yet, maybe this will change in time. In the meantime we've made it available as part of our Horizon partnership with Google https://github.com/googlecloudplatform/horizon-sdv

[oberstet]

Hi @colmaengus — thanks for the careful and detailed write-up, and for the kind words on stability. Reports like this, with the symptom pattern laid out so clearly, are genuinely useful even without a repro. Let me share what we think is going on, an immediate mitigation you can try, and a couple of questions to help us pin down whether there's also a residual bug on our side.

What this looks like

The symptom triad — silently swallowed calls, the clean N/2N success/failure ratio when you change the session count, and a procedure_exists_with_different_invocation_policy for a registration that doesn't show up in wamp.registration.list — all point at the same mechanism:

Undetected half-open TCP connections leaving stale callees inside a shared (round-robin) registration group.

The sequence:

1. A callee's TCP connection dies ungracefully — no FIN, no RST. This is the textbook behavior of a NAT or stateful firewall silently expiring an idle flow, which is exactly what you'd expect on a lab↔Azure path.
2. The router never learns the session is gone, so that session and its registration stay in the round-robin group.
3. Round-robin then dispatches an INVOCATION to the dead callee. It goes into the void — no YIELD, no ERROR comes back — so the caller's CALL never receives a RESULT or ERROR. That's your "silently swallowed" call, and it's why the failures track the rotation: with K dead callees out of N, roughly K/N of calls land on a corpse.
4. The procedure_exists_with_different_invocation_policy you get when switching round-robin → random is actually the most useful clue: it proves a round-robin registration group still exists in the router at that moment. So the registrations aren't really gone — which makes the "nothing in wamp.registration.list" part the interesting discrepancy (more on that below).

The crucial point: application-level / WAMP-level heartbeating between your client and its own logic won't break this, because the thing that needs to notice the dead callee is the router actively probing that callee's socket. A half-open socket looks perfectly healthy until someone writes to it and waits.

Immediate mitigation — you're already doing the right thing, just turn it up (on both ends)

You're already running WebSocket auto-ping heartbeating, which is great — that feature exists precisely to detect ungraceful TCP failures. The key thing is that both sides need it, and for different reasons — so it's worth running it aggressively on the router and on the clients:

• Router-initiated heartbeat lets the router detect a dead callee quickly and clean up its own state — the connection, the session metadata, and crucially the shared registration group. This is what stops round-robin from dispatching into a corpse. Without it, the stale callee lingers indefinitely.
• Client-initiated heartbeat lets the client detect the dead link quickly and restart from scratch — tear down, re-establish a fresh TCP connection, and re-register. And if the reconnect itself fails immediately, the client gets to escalate a real network problem right away instead of sitting on a silently-dead socket.

So configure auto-ping on the router's listening transport (the one your callees connect to) and on the clients, and make both timings tight, so on the router side:

"options": {
    "auto_ping_interval": 250,
    "auto_ping_timeout": 200
}

(In current Crossbar these are milliseconds.) The reasoning: even a full round-the-world RTT — say EU ↔ US West Coast — comfortably fits under 200 ms, so a 250 ms ping interval with a 200 ms pong deadline is safe under any normal network conditions, while letting each side detect a dead peer in well under a second instead of never.

One sequencing detail that matters when both heartbeats are aggressive: when a client tears down and reconnects after a drop, have it wait a reconnect delay that is larger than the router's heartbeat timeout before re-registering. That ordering guarantees the router has already detected the old (half-open) session and reaped its entry from the shared registration group before it ever sees the reconnecting client re-register. Otherwise you can briefly end up with both the stale and the fresh registration coexisting in the group — which is exactly the kind of window where calls get swallowed or a policy conflict shows up.

As a belt-and-suspenders safety net, also set a call timeout on the caller side (call(..., options={'timeout': ...})). Even if a stale registration ever does slip through, the caller then gets a clean timeout error instead of hanging forever.

Two version notes

• Crossbar 25.12.1 / Autobahn|Python 25.12.2 — good, those are current. 👍
• Node.js clients: make sure you're on the current Autobahn|JS (autobahn 26.3.1 on npm). That release dropped the bundled legacy ws dependency, since native WebSocket is now universally available in both browsers and Node.js. We've seen a demanding application in US defense upgrade straight to the current releases with zero impact — so this is a low-risk move and worth doing to rule out any client-side transport quirks.

What we'll do on our side

The "invisible registration" part deserves a real investigation, not just a config answer — if the registration map and the meta API can diverge (a stale callee that the policy check still sees but wamp.registration.list / wamp.session.list don't), that's a bug we want to find and fix, and it carries the needs-test label for a reason.

To do that properly, we're going to build an integration test that injects exactly this failure mode: a virtualized, fault-injectable network (Toxiproxy) sitting between callees and the router, so we can sever connections ungracefully — drop the link with no FIN/RST, add latency, partition — and assert that round-robin dispatch and the registration meta API both behave correctly as sessions die undetected and then get reaped. That's the only way to reproduce a half-open socket deterministically in CI, and it'll let us nail down whether there's a residual cleanup bug beyond the heartbeat tuning.

---

One last thing — it's great to hear the stack has been rock-solid for you. If you're able to say anything more about the deployment and what you're building on WAMP, and your Google and Horizon collaboration, we'd genuinely love to hear it; real-world use cases directly shape where we take the project.

And a thought for where you might take the architecture: you're already reaping one of WAMP's core advantages — outbound-connection-only, seamless application messaging between locations through a single cloud router, with shared registrations giving you built-in, app-transparent load balancing (a WAMP Advanced Profile feature). Crossbar.io goes well beyond a single node here: with router-to-router meshes you can run the cloud node plus an edge router at each location, with clients connecting to their nearest edge router and continuing to do the exact same seamless messaging across the whole mesh — with zero application code changes. For multi-site setups that often improves both resilience (local survivability) and latency, while keeping the simple programming model you already have.

Thanks again — we'll follow up here as the Toxiproxy-based test takes shape.