Replies: 11 comments 2 replies
|
Yes, it outlines the same problems, and also suggests flatbuffers. My question is mostly, is there anything Im overlooking when using websockets as a transport? |
|
Moving halui into task and bypassing NML would be a good thing. The real problem currently in NML is the fact that error messages are not available on all readers and errors can therefore not be correlated correctly over all connections. This needs to be rethought, how errors are generated, propagated and correlated. I'm not to thrilled about using boost derived packages. It is very heavy and most of it, if not all, has been made redundant in modern C++. Actually, I'd rather have the boost dependency removed completely. Weren't you a fan of pybind11? ;-) With the proper interface, a "view" of HAL can be made visible remotely. As long as it stays within the confines of "slow as hell will do", then it should work. However, using a different interface structure would probably be better and leave HAL completely on the server-/machine-side. Using websockets, just like many other network communication layers, is how to handle errors. Errors are very much more recurring on network connections than any local interface. How to handle state-full information and sync/recovery is the art of the beast, so to say. |
|
talking of which, I have a branch that removes boost-python dependency, replacing it with pybind11. currently UIs are a hal userspace component, and that is what killed remote UIs. (it was possible before hal) agree on the error handling. |
|
yes, the error thing is a major pain with halui. |
|
A problem with TCP can be that it is not easy to notice if and when the other end of a connection goes away. Another problem can be connections that only transmit in one direction, without anything coming back, TCP delayed ACK can slow down connections to a crawl. E.g. Windows uses AFAIK 200ms per default, and you really are not supposed to change that. 200ms means 5 packets per second. It seems websocket does some message framing, that should take care of that. |
Network connections are notoriously difficult when you want guarantees. You'd have to start by examining the L1/L2 configuration. With Ethernet, it can take "ages" before you detect a disconnect. At least Tokenring had more real consistency. But then looking at L3/L4 with TCP/IP does not do much better because of routing and connection protocols. FWIW, TCP is just not good at timing, at all, and SCTP or UDP are a pain and no help when the underlying layers give you no guarantees. You will need to have a very well controlled local network to have some decorum. That leaves you with with everything (very) non-critical that can be done safely on a network connection. Anything that has intermediaries will be problematic with respect to timing and consistency. You only need to look at the efforts to get a point-to-point Ethernet connected I/O card stable (TCP/IP or Ethercat). And these are the well controlled environments. |
|
Remote HAL interface should enable to use something like halshow to run remotely. Remote HAL components with uncritical timing could also make sense to run remotely, things like a VFD interface. Anything "realtime" I would say doesn't need supporting and should be discouraged, at least over websockets. OTOH something like mesa hostmot network protocol or remora could possibly be extended to talk "hal" directly. |
|
Even a VFD interface can be time critical. Any operation that can "race" other operations must be properly and guaranteed sequence-able. |
|
chrome has a feature specifically for this: https://developer.chrome.com/docs/devtools/settings/throttling |
|
Some questions from reading the code, since a few assumptions in this thread deserve testing: On "zero copy matters locally": did you know the local SHMEM path already XDR-serializes every message? linuxcnc.nml says On the error problem: @BsAtHome, you said errors cannot be correlated across readers. Looking at the mechanism, it is worse: emcError is a queue with head/tail stored in the shared buffer itself, and read() destructively pops (cms_in.cc:817-828). No per-reader cursor, no retention. With two UIs attached, each error goes to exactly one of them, chosen by timing. Queue full and the writer drops silently (emctaskmain.cc:173-186). And motion/io faults mostly bypass the channel altogether; several emcOperatorError call sites are commented out (emctaskmain.cc:2801-2823). So the question is not "how do we serialize errors" but: what should the error model be? My sketch: task keeps a monotonic sequence-numbered ring, entries carry source (task/interp/motion/io), severity, and the originating command's serial; new entries are pushed to all subscribers, and a client sends its last-seen seq on connect to get replayed what it missed. Broadcast, correlation, and reconnect recovery from one mechanism, maybe 100 lines in task. What would it break? On scope: after halui moves into task, is the seam not just three files? shcom.cc (~60 functions; linuxcncrsh, schedrmt, emcsh all ride on it), emcmodule.cc (~40 command methods, ~88 stat attributes; every Python GUI), and emctaskmain.cc itself. The tool channel is already mmap (tooldata_mmap.cc), iocontrol already merged. Is anything else left that I missed? On the command handshake: today completion is a poll race on echo_serial_number with exponential backoff (shcom.cc:263-325; Python spins with esleep under a 5 s timeout, emcmodule.cc:922-960). If the transport is request/reply, the ack becomes explicit and carries an error reference. Any reason to preserve the echo mechanism at all? On transport: the rates involved are tiny (UIs poll status at 20 ms, errors at 200 ms, axis.py:789,834; nobody blocks on NML anywhere in-tree). Given that, is there an argument against running the same websocket server on loopback for local clients too, at least for v1? One code path, no parallel shm protocol; if profiling shows the status publish hurts, add a shm ring behind the same client API later, schema unchanged. On versioning: today the only skew check is the EMC_STAT_TYPE peek at connect (shcom.cc:90); emcmodule does a raw memcpy(sizeof(EMC_STAT)) plus an offsetof-based member table (emcmodule.cc:1022, :1124-1125). UI and task from different builds are silently undetected. FlatBuffers field evolution plus a handshake version fixes this for free. Worth stating as a hard requirement? On prior art: Machinekit already did this migration (machinetalk: protobuf over ZeroMQ, command/status/error/file services). Two lessons seem relevant: they kept NML alive beside it for years and the dual-stack dragged, which Rene's phasing avoids by shrinking NML to one seam first; and ZeroMQ never worked in browsers, so web UIs still needed a bridge, which websocket removes. Anyone here have firsthand machinetalk experience to confirm or correct that? And a starting point: rmu75's rs/zmq-experiments branch already has emc_cmd.fbs (583 lines), emc_stat.fbs, emc_error.fbs. The error schema there is a verbatim copy of the three NML operator messages, so it needs the seq/source/correlator treatment above, but the cmd/stat IDL looks like real groundwork. Rene, does it match what you had in mind for phase 3? |
|
Yes, machinekit never fully replaced nml, they did the classical Linuxcnc thing: putting things on top of other things, not removing the old one. I now have a branch with the first steps done, but will wait until bertos hal changes are done, as they do touch the same things in some places. |
|
On Sa, 2026-09-19 at 14:39 -0700, BsAtHome wrote:
> A problem with TCP can be that it is not easy to notice if and when
> the other end of a connection goes away.
Network connections are notoriously difficult when you want
guarantees.
We should not overthink this. Of course with TCP/IP you are on shaky
grounds in principle and there are no real guarantees in an IP network.
Token ring also had problems, it was slow, and, the token could get
lost, then somebody would have to generate a new one, after some
timeouts.
We just need to be prepared that a websocket-client can go away and
that it is not straight forward to detect that.
|
Uh oh!
There was an error while loading. Please reload this page.
During the Stuttgart meeting, I gave a NML replacement some thought.
The only positive thing to say about NML is that it has served us well over the years.
I think we all agree that it is unmaintainable, and nobody wants to touch it.
In the past, and probably even more likely in the future, people always wanted to have/create remote UIs, especially web based UIs.
While this is in theory possible with nml, there are very few examples showing how to do it, and web UIs require a bridge.
there have been experiments with zmq and flatbuffers, e.g. https://github.com/rmu75/linuxcnc/tree/rs/zmq-experiments
and flatbuffers actually seem like a good idea. they are zero copy, efficient, and bindings can be generated for almost any language. Leaves the question of transport and framing. websockets do both, and work native in a browser, and almost any language supports websockets. It is mostly just a tcp socket that has been set up in a special way.
so here is my plan:
remove nml where it is used only for helpers
this are mostly timers, some types, and other little helpers. I am currently working on this, it can go straight into master, as it doesn't really change much.
move halui into task
halui talks to task via nml, and mostly exposes its state machine to hal. there is no reason task cant do this directly.
I have done the same with iocontrol, and it saved over 2000 lines of code, and exposed a lot of race conditions.
#2497
this can also go in master, as it doesnt affect anything user facing.
after this, nml is only used in communication between task and the UI.
new c++ api
create new c++ api, that talks via flatbuffers over a websocket to task, with the same messages as nml.
this wil be used by the c++ clients, used mostly for testing. this is what shcom.cc currently does, but extended so all UIs can use it.
create python bindings for the c++ interface
this is what emcmodule.cc currently does. In theory, no change to UI code will be required. Practice will probably tell that there are quirks of nml addressed in the UI.
as a stack, I suggest using beast, which is part of boost, which we already depend on.
Preview
currently the preview interpreter instance lives in the UI, which means every UI has to load it, it needs bindings, and every UI needs to manage it. I had the idea to move it to task, so that the UI can request parsed chunks of gcode, and only needs to render it. it has the advantage that limits are applied in headless systems, and it massively simplifies UI code.this is a big change for the future, and needs changes to the canon interface. just dropping the idea here.
HAL
currently a remote UI doesn't have access to hal. I am still thinking about how to solve this in a way that the same c++/python api can be used via SHM or network transport.
All reactions