Fix for Issue #205 (Overlapping Signals / Multiplexors) by friessssss · Pull Request #825 · cantools/cantools

friessssss · 2026-08-19T21:28:18Z

Summary

Fixes #205.

cantools still treated overlapping signals as invalid at decode/encode time, even when a database was loaded with strict=False. That shows up as ValueError: Short data (or a DecodeError wrapping it).

This is a real pattern in automotive DBs: bit flags plus a wider status/fault field covering the same bits. The original report used eight 1-bit flags overlapping an 8-bit Status on a 1-byte Intel message. The same failure also happened for Motorola signals and for multiplexed SYM frames where a status word shares bits with a flag.

strict=True is unchanged: overlapping signals still raise at load time. With strict=False, decode and encode now return consistent values for both the wide field and the overlapping bits.

Approach

bitstruct cannot represent overlapping fields in a single format string. Overlapping signals are now split into internally non-overlapping layers, each compiled as its own codec. Decode unpacks every layer and merges the results. Encode packs each layer and ORs them together.

Non-overlapping messages still use a single layer, so the common path is unchanged.

Tests

The original Overlapping signals causes ValueError: Short data. #205 DBC (little-endian flags + status byte)
Overlapping big-endian signals
A reduced multiplexed SYM frame (tests/files/sym/issue_205.sym)

Notes

Load overlapping databases with strict=False, same as before. Encoding also needs strict=False so the overlap check in Message.encode() does not reject the payload.

andlaus

I did not go into this too deeply, but overall it looks good. I'm wondering if this works with multiplexers, though...

andlaus · 2026-08-20T06:56:30Z

+        | int.from_bytes(formats.little_endian.pack(raw_signal_values), "little")
+    )
+
+    for big_endian, little_endian in formats.extra_layers:


please add a comment about what extra layers are

andlaus · 2026-08-20T06:59:29Z

+    try:
+        return bitstruct.c.compile(fmt, names)
+    except Exception:
+        return bitstruct.compile(fmt, names)


without importing bitstruct, this will crash as well. please use the previous approach and simply call bitstruct.compile()

I moved this to a function for ease of re-use. Returned the initial import to the original approach, so this should be much more valid now.

since bitstruct.c is imported as bitstruct (if available), this function is redundant: simply call bitstruct.compile().

I don’t think calling bitstruct.compile() directly works here. import bitstruct.c doesn’t rebind bitstruct to the C module. Like any import a.b, it binds the name bitstruct to the top-level package, which is the pure-Python one. I checked: after import bitstruct.c, bitstruct.compile is a Python function, while bitstruct.c.compile is the built-in. So bitstruct.compile() would quietly drop the C speedup for every message.

The fallback is also still needed. bitstruct.c may not be installed, and when it is, it doesn’t support every format. For example, bitstruct.c.compile('u65', ...) raises NotImplementedError, which can happen with long diagnostic DIDs.

This is the same try/fallback master already had (once for big-endian, once for little-endian). I moved it into a helper because it’s now called once per layer. I added a docstring explaining this.

what I meant was to import bitstruct like this:

try: import bitstruct.c as bitstruct except ImportError: import bitstruct

if done like this, bitstruct.compile() should automatically select the correct module. (assuming that no features that are unsupported by bitstruct.c are used, I think this is the case in cantools.)

I tried import bitstruct.c as bitstruct, but cantools does compile formats that bitstruct.c doesn’t support. With that import, 38 tests fail:

Signals wider than 64 bits (e.g. tests/files/kcd/message_layout.kcd has a 128-bit signal) raise NotImplementedError. Whether the C version works depends on the format string, so the fallback has to happen when each format is compiled, not at import.

The little-endian padding mask uses bitstruct.pack(f'u{length}', ...) over the whole message, so anything longer than 8 bytes fails to load.

bitstruct.c has no Error, so the except bitstruct.Error in decode_data would raise AttributeError.

I’ve kept the per-format try/fallback from master, let me know if that’s okay with you ! I'm not totally against the switch but it might need some more ironing out.

friessssss · 2026-08-20T16:39:08Z

@andlaus I added an explicit multiplex test to the test suite to make sure that this works as expected. Also the issue_205.sym is stripped down version of the symbol file that caused me to stumble upon this issue in the first place, and it is now used in tests. Let me know if there are any other tests / improvements that you would like to see !

ckagerer · 2026-10-15T09:47:44Z

We would like to see this merged. Our DBCs describe status bitfields twice: once as the whole value and once as single-bit signals, so that tools like CANoe show both the raw word and the named flags. CANoe handles this fine, but cantools cannot decode these messages today, so we cannot use the same DBCs in our Python tooling.

I tested this branch, merged onto current master, with two of our DBCs (68 and 6 overlapping pairs, all inside a multiplexer branch). On master both fail to decode with strict=False; with this PR all values decode correctly. Overlap within one mux branch, between a branch and a plain signal, and across branches all work. Tests, ruff and mypy pass after the merge.

Two points:

Encoding inconsistent values (for example Whole=0 with Bit0..7=1) silently ORs them (ff00…). Raising an EncodeError there would avoid the "undefined behaviour" concern from Setting 'strict' flag in 'False' does not work for overlapped signals #63.
A load/dump round-trip test for an overlapping DBC would cover what was asked for in Fix overlapping signals #763.

ckagerer · 2026-10-15T11:17:56Z

Following up on encoding: I put a proposal in a draft PR in my fork, on top of this branch: ckagerer#4. Its diff shows only the change on top of this PR.

The rule: overlapping signals within the root or one multiplexer branch must have the same raw bits where they overlap, else encode() raises EncodeError. An overlapping signal can be left out if the given signals set all its bits, so the whole value alone or the bits alone both encode. Messages without overlaps keep the current code path, and I measured no change in load, encode, or decode time.

One cost: decode, change one view, and encode again raises until the other views are changed or removed. With the current OR, clearing a bit that way gives a wrong frame without an error. If this direction is fine, I can open it as a follow-up PR after this one is merged.

andlaus · 2026-10-21T08:57:41Z

-    little_fmt, little_padding_mask, little_names = create_little()
+    def compile_layer(
+            be_signals: Sequence[Union["Data", "Signal"]],
+            le_signals: Sequence[Union["Data", "Signal"]],


this should be two functions: compile_be_layer(), and compile_le_layer(). This would avoid the shenanigans below.

Done. It is now compile_be_layer() and compile_le_layer(), which return (codec, padding_mask).

andlaus · 2026-10-21T09:00:26Z

+        for index in range(layer_count)
+    ]
+
+    padding = compiled_layers[0].padding_mask


a comment what is done here and why would be good.

The padding mask marks bits that no signal uses. Message.encode(padding=True) fills exactly those bits with unused_bit_pattern, so the mask must not include any bit that belongs to a signal.

With overlapping signals, each layer is compiled separately and only knows about its own signals. Its mask therefore treats every other layer’s bits as padding. A bit is really unused only if it is padding in all layers, so the masks are combined with AND.

Here’s what goes wrong with only the first layer’s mask (the old primary-codec behaviour). Take a 1-byte message with Low on bits 0–3 and Wide on bits 2–5, with unused_bit_pattern=0xff. They overlap, so they land in different layers:

Layer 1 (Low) has mask 0b11110000. Bits 4–5 look like padding, but Wide uses them.
Layer 2 (Wide) has mask 0b11000011.
ANDed, the mask is 0b11000000. Only bits 6–7 are really unused.

Encoding {'Low': 0, 'Wide': 0} with padding=True gives 0xc0 with the combined mask. With layer 1’s mask alone it gives 0xf0, which decodes as Wide == 12. The value is silently corrupted.

I expanded the code comment to explain this and added test_overlapping_signals_padding for this case.

andlaus · 2026-10-21T09:03:30Z

    padding_mask: int
+    # Additional (big, little) codecs for overlapping signals. Each
+    # layer is internally non-overlapping so bitstruct can unpack it.
+    extra_layers: tuple[tuple[CompiledFormatDict, CompiledFormatDict], ...] = ()


why not simply use lists for everything? i.e., something like

big_endian_layers: list[CompiledFormatDict] little_endian_layers: list[CompiledFormatDict]

if there are no overlapping signals, these lists exhibit length 1.

Formats now has big_endian_layers and little_endian_layers lists, each with one entry when nothing overlaps

andlaus · 2026-10-23T07:37:37Z

@friessssss: are you gonna push this or do you prefer if I take it over? This is a long-standing major missing feature which I'd like to get merged in the not too distant future...

friessssss · 2026-10-23T17:41:01Z

@andlaus I will knock out these changes today and test them out ! I'd love to get this in ASAP !

- Formats now holds big_endian_layers / little_endian_layers lists instead of a primary codec pair plus extra_layers - split compile_layer() into compile_be_layer() / compile_le_layer() - document _compile_bitstruct() and the padding mask computation - add a dump/load round-trip test for an overlapping DBC

friessssss · 2026-10-23T17:53:59Z

@ckagerer I added the dump/load round-trip test for an overlapping DBC (#763). Would you prefer the EncodeError fix in this PR or do you want to handle that?

andlaus · 2026-10-26T20:24:18Z

almost there.

I think that we should consider adding support for negative padding in bitstruct/bitstruct.c as it would make implementing overlapping signals in cantools significantly simpler and probably more performant. (I have a patch for that, but I did not yet find the time to submit it.)

andlaus · 2026-11-03T11:41:26Z

ok, negative padding is not going to fly with bitstruct, so I think we should go the route which you proposed. IMO the PR is ready, once the try...except construct in _compile_codecs is replaced doing this at the import-level. (see my previous comment. alternatively, explaining why the import trickery cannot be used works as well.)

andlaus · 2026-11-04T10:08:05Z

@friessssss: are you going to try the import-level bitstruct switching or do you prefer me to finish this job?

friessssss · 2026-11-04T23:31:14Z

@andlaus I left you a quick comment back on that ! Let me know which direction you want to go on this, happy to make the change wit we decide thats the best way to go !

andlaus · 2026-11-05T22:55:03Z

you mean this one?

Note that import bitstruct.c binds the name bitstruct to
the pure Python package, so bitstruct.compile() is the slow
implementation. bitstruct.c is unavailable on some platforms
and does not support every format (e.g. integers wider than 64
bits), hence the fallback.

The unavailability of bitstruct.c on some platforms is also handled by the import-level solution. For bitstruct.c's limitations, I am pretty confident that real-world use-cases of cantools do not get restricted by those (the most relevant one seems to be <= 64 bit integers to me). On the other hand, creating a list potentially containing a mix of bitstruct.c and bitstruct codec objects seems like a very bad idea to me, in particular because this case never gets tested. If there are no better arguments for doing this at runtime, please change to using an import-level switch; If there are better reasons, please add a well-commented unit test based on a valid database file (DBC, ...).

friessssss added 2 commits August 19, 2026 15:25

fixes for Issue cantools#205

2430474

fix linter error

00d1fb7

friessssss changed the title ~~Fix for Issue #205~~ Fix for Issue #205 (Overlapping Signals / Multiplexors) Aug 19, 2026

andlaus reviewed Aug 20, 2026

View reviewed changes

Addressed PR comments

028ef50

andlaus reviewed Oct 21, 2026

View reviewed changes

andlaus mentioned this pull request Oct 21, 2026

Fix overlapping signals #763

Closed

Explain padding mask for overlapping signals, add padding test

a1dd38c

andlaus mentioned this pull request Nov 3, 2026

implement support for negative padding eerimoq/bitstruct#59

Closed

Conversation

friessssss commented Aug 19, 2026

Summary

Approach

Tests

Notes

Uh oh!

andlaus left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andlaus Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

friessssss commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ckagerer commented Oct 15, 2026

Uh oh!

ckagerer commented Oct 15, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andlaus Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andlaus commented Oct 23, 2026

Uh oh!

friessssss commented Oct 23, 2026

Uh oh!

friessssss commented Oct 23, 2026

Uh oh!

andlaus commented Oct 26, 2026

Uh oh!

andlaus commented Nov 3, 2026

Uh oh!

andlaus commented Nov 4, 2026

Uh oh!

friessssss commented Nov 4, 2026

Uh oh!

andlaus commented Nov 5, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andlaus Oct 21, 2026 •

edited

Loading

friessssss commented Aug 20, 2026 •

edited

Loading

andlaus Oct 21, 2026 •

edited

Loading