[jasperpotts]

Closes #1018. Stacked on #1016 (1014-unboxed-oneofs-part1).

Summary

Adds an opt-in, generated, zero-allocation write API. With pbj.writers, each message gets a nested
Xxx.Writer: a reusable flyweight that encodes fields straight into a reusable ProtoWriteBuffer as the caller gives
them, with no model objects. Output is byte-identical to Xxx.PROTOBUF.toBytes(object) for the same values. It is the
write-side counterpart of the views, and one option on a root message (for example Block) covers the whole tree.

The whole issue is in this PR: the runtime, the option with its transitive closure, writers for every field kind (maps,
wrapper types, raw and object embeds, and cross-module fields included), writeFrom, tests, fuzzing, the benchmark and
the docs.

Results on EPYC, in short: writing block items from field values allocates 0 B/item (building objects then
encoding them allocates 1,604 B/item) and is 15% faster than build-then-toBytes, but 10% slower than
build-then-writeReverse. writeFrom on cached objects is 33% faster than writeReverse, but 2.3× slower than
write(T, byte[]). Two of the issue's targets are not met; details and options for closing the gap are below.

After the rest of the stack (#1029, the final run on the synthetic HAPI block): the writers encode a transaction in 1,122 ns with 0 B, 1.5× the hand-written reference codec, and 21% faster than building objects and writing them back to front (1,426 ns, 8,066 B) or 40% faster than building them and calling toBytes (1,884 ns, 9,347 B). Full table: #1023 (comment)

The option

Comment (file or message)	Gradle (pbj { })	Values	Default
// <<<pbj.writers = true>>>	generateWriters	true / false	false

• Resolves like the other options (message, then file, then project), but it is not part of performance = "max": writers are additive API with a real code-size cost, as the issue suggested. PerformanceOptions gains a
  writers component and resolveExplicit, so a performance setting at one level doesn't hide writers set at a
  less specific level.
• Transitive closure (LookupHelper.writers()): a worklist over a visited set, starting from every message that
  resolves to true, through singular, repeated, oneof and map-value message fields, across files. Cycles are reached
  once, with no provisional values. A reachable message with an explicit false (in its own or its file's comments)
  fails the build, naming the path, for example example.Root.middle -> example.Middle.leaf -> example.Leaf.
• Cross-module types (not generated in this build) get no child writer; their fields have only x(Obj), written
  with Obj.PROTOBUF, and xRaw(Bytes).
• Errors: a message declaring its own nested Writer type, a field whose type is named Writer, a repeated
  google.protobuf wrapper field (no HAPI message has one), more than 64 oneofs, or two fields generating the same
  method signature.
• Field names that clash with the writer's own methods (start, end, offset, length, isOpen, writeFrom, the
  Object methods, the protected checks) get the suffix Field, for example lengthField(long).

Runtime (com.hedera.pbj.runtime.io.buffer)

• ProtoWriteBuffer owns its array and doubles it when full. It offers reset(), length(), array(), bytes(),
  bytes(offset, length), toBytes(), copyTo() and shiftCount(), with a LengthReservation of ADAPTIVE
  (default), PER_TYPE or ONE_BYTE. The API for generated writers is one call per field (tagVarint, tagFixed32,
  tagFixed64, tagString, tagBytes, tagMessage, tagRaw and packedXxx(tag, values, offset, count)). Each has
  its common case (a one-byte tag and value, room available) as a few inline stores and everything else out of line,
  so it fits C2's inlining budget (see Performance work).
• MessageWriter is the base of every generated writer and holds the state machine:
  • Order checks, always on: one compare per field against guard, which is the last field number written, or
    Integer.MAX_VALUE while a child is open or the writer isn't. The slow path ends the child (auto-close) or throws.
    It also covers a repeated singular field, repeated elements that aren't consecutive, a second oneof alternative
    (one bit per oneof), use after end(), use before start(), and writeFrom after any field.
  • Nesting: the parent writes the tag and reserves 1 or 2 bytes, and the child starts after them. On close, the
    length is patched in place (one or two stores when the guess was right), otherwise the body moves by the
    difference. Output is always canonical. A closing child restores its parent's fast path directly, so an explicit
    end() costs one call.
  • Auto-close (chosen, documented and tested): any call on a writer first ends whatever is open inside it, and
    ending a parent ends its children.
  • Item ranges: offset()/length() after end(). Direct children of a top-level message never move, so
    hashing block items as they end is exact. Deeper messages can move when an ancestor's length prefix needs a
    different size; this is documented.
• PackedIntWriter, PackedLongWriter, PackedFloatWriter, PackedDoubleWriter and PackedBooleanWriter stream
  packed elements; an empty field is dropped entirely, as the model writes it. MapEntryWriter holds the entry of a
  message-valued map.

Generated writer

WriterGenerator, called from ModelGenerator next to the view. Per field kind:

Field kind	Methods
int/uint/sint/bool/enum	x(value); enums also x(Enum) and xProtoOrdinal(int); defaults skipped outside oneofs
fixed/sfixed/float/double	x(value); -0.0 counts as the default and NaN doesn't, exactly as the model
string	x(String), x(byte[] utf8, off, len), validated as well-formed UTF-8 (throws before writing anything)
bytes	x(Bytes), x(byte[], off, len)
message	Child.Writer x(), x(Child obj) (static codec writeTo with cached size), xRaw(Bytes)
repeated message	addX(), addX(obj), addXRaw(Bytes)
repeated scalar	x(long[]/int[]/…, off, count) packed in one call, or PackedXxxWriter x()
repeated string/bytes	addX(...)
map	putX(key, value); Value.Writer putX(key) for message values; ascending keys checked
oneof	each alternative's methods, always written; a null message alternative is written empty, as the model does
wrappers	x(value) writes the wrapper even for the default value; leaving the field out means null

writeFrom(object) writes every field, and the object's unknown fields, through the writer's own methods and child
writers. It sizes each nested child exactly from the object's cached protobufSize(), so it never moves bytes, and it
allocates nothing. In messages with more than MAX_INLINE_FIELDS fields, each message alternative's code goes in a
small helper method (TransactionBody.Writer.writeFrom was 7,181 bytes before, which GeneratedMethodSizeTest caught).

Generated code for one field kind of each sort (from the issue's Order example)

public Writer id(final long value) {
    checkField(1);
    if (value != 0) {
        out.tagVarint(8, value);
    }
    return this;
}

public Price.Writer price() {
    checkField(3);
    if (priceWriter == null) {
        priceWriter = new Price.Writer();
    }
    openChild(priceWriter, 1, 0x1A);
    return priceWriter;
}

public Writer putLimits(@NonNull final String key, final long value) {
    final boolean first = isFirstElement(6);
    checkRepeatedField(6);
    if (!first && key.compareTo(limitsLastKey) <= 0) {
        throw new IllegalStateException("The keys of map limits must be written in ascending order, each once,"
                + " as the model writes them: " + key + " after " + limitsLastKey);
    }
    limitsLastKey = key;
    final int keyLength = ProtoWriterTools.sizeOfStringNoTag(key);
    final int keySize = 1 + ProtoWriterTools.sizeOfUnsignedVarInt32(keyLength) + keyLength;
    final int valueSize = 1 + ProtoWriterTools.sizeOfUnsignedVarInt64(value);
    out.tagVarint(0x32, keySize + valueSize);
    out.tagString(0x0A, key, keyLength);
    out.tagVarint(0x10, value);
    return this;
}

Tests

All green in both integration configurations, which now run 327,265 tests each (default and
-PpbjPerformance=max); max also turns generateWriters on for every message. pbj-core qualityGate passes.

1. Byte equivalence, generated per message (TestGenerator): every test object of every message with writers,
   random objects included, goes through writeFrom into a 1-byte buffer (so it grows mid-message), once per
   reservation strategy and again after a reset(). The bytes must equal toBytes, and parse back equal. In max
   that is all 608 messages, the full HAPI corpus included: about 162k cases, 0 failures.
2. WriterTest (hand-written, both configurations) covers every row of the field-kind table:
   • defaults skipped; negative int32 as ten bytes; zigzag; NaN; -0.0; five-byte tags;
   • UTF-8 arrays, including invalid ones, which throw before writing;
   • packed fields in one call and streamed, and empty packed fields;
   • maps in Java key order, and out-of-order or duplicate keys;
   • oneof alternatives with default values, null message alternatives, and two oneofs;
   • all nine wrappers, with defaults and with absent values;
   • reserved names, raw embeds and object embeds;
   • fields from another module (writerShared.proto, used from the custom source set);
   • self, 2- and 3-cycles (in custom, since the generated test-argument code can't build recursive objects);
   • misuse (order, repeated singular, second oneof, use after end, unstarted, writeFrom rules) and auto-close;
   • nested lengths of 0, 1, 120-128, 16,383, 16,384 and 2.1 MB under each strategy;
   • 100-deep nesting, beyond any stack;
   • item ranges staying correct while later siblings are written and the buffer grows.
3. Fuzz (fuzzCallSequences, seeded): 2,000 random valid call sequences from a walk of WriterRoot's schema.
   Each randomly picks child writers, objects or raw bytes; arrays or streamed packed fields; explicit or automatic
   ends. All must equal the model's bytes. Then 2,000 shuffled sequences, with duplicates, must throw at the first
   invalid call without writing a byte, and only sorted ones may succeed.
4. ProtoWriteBufferTest (runtime): every varint and tag size; length boundaries 0 to 2.5 MB under every strategy
   and per-type guess; adaptive learning; 200-deep nesting; growth mid-message; closed children's ranges; packed
   elision; auto-close; misuse; reset abandoning open writers.
5. WriterGenerationTest (compiler): closure across files; self, 2- and 3-cycles; explicit false as an error with
   the path (message and file level); the project option, and max not enabling writers; cross-module fields getting
   only embed methods; the Writer name guard; reserved-name suffixes; repeated wrappers rejected; the 1-or-2-byte
   reservation. PerformanceOptionsTest covers parsing and resolveExplicit.
6. Allocation: AllocationTest.writersDoNotAllocate covers a message tree from values and through writeFrom
   (0 B). BlockWriterTest (src/testMax) writes the sample block's items from extracted values, from objects, and as
   a whole block with SHA-384 hashing of each item as it ends: 0 B for all three. It warms up over every item first,
   since uncompiled branches deoptimize and that allocates. It also checks the item hashes equal hashes of
   toBytes(item) and the whole block equals Block.PROTOBUF.toBytes, under each strategy.

Benchmark: BlockItemWriterBench on EPYC

AMD EPYC 9275F (Zen 5), JDK 25.0.4, -Xms16g -Xmn12g, pinned to one CCD of an otherwise idle machine, 3 forks × 5
iterations of 2 s, -prof gc, per block item of the sample block …030221.

"From values": BlockItemValues (in src/testMax, also used by the tests) extracts, once, the field values of the 4
item kinds that make up almost all of the block: signed transactions, transaction results, event headers, and state
changes that are map updates (accounts, token relations, NFTs, topics, storage slots) or map deletes. That is 66,166
of 67,869 items. It is hand-written. Both paths read the same flat arrays in the same order; every extracted item is
checked to rebuild and rewrite byte for byte.

Benchmark	ns/item	B/item	gc.time
objectsThenToBytes: build objects with builders, toBytes	371.0 ± 9.7	1,604	234 ms
objectsThenWriteByteArray: build, write(T, byte[])	360.1 ± 17.2	1,291	123 ms
objectsThenWriteReverse: build, writeReverse	289.0 ± 6.7	1,291	250 ms
writerFromValues (ADAPTIVE)	317.1 ± 5.3	0	0
writerFromValues (PER_TYPE / ONE_BYTE)	327.7 ± 1.8 / 316.4 ± 5.1	0	0
objectsWriteByteArray: cached objects, write(T, byte[])	160.9 ± 3.6	0
objectsWriteReverse: cached objects, writeReverse	555.8 ± 22.2	0
writerFromObjects (ADAPTIVE): cached objects, writeFrom	372.4 ± 8.9	0
writerFromObjects (PER_TYPE / ONE_BYTE)	370.3 ± 5.5 / 364.1 ± 10.1	0
objectsWholeBlock: each item write + SHA-384, then write(Block)	755.9 ± 13.9	0
writerWholeBlock (ADAPTIVE): Block.Writer, SHA-384 of each item's range	815.2 ± 14.1	0

Against the acceptance targets

Target	Result
writerFromValues 0 B/op	Met: 0 B against 1,604 B, no GC time against 234 ms
writerFromValues clearly faster than objectsThenToBytes	Partly: 15% faster than toBytes and 12% faster than write(byte[]), but 10% slower than build + writeReverse
writerFromObjects no slower than write(T, byte[]) with cached sizes	Not met: 2.3× slower (372 against 161 ns)
writerFromObjects within 20% of writeReverse	Met: 33% faster (372 against 556 ns)
Report length-reservation strategies	Below
Code size	Below

Where the gap is. Per item kind (local per-kind loops), from values the writer is much faster than building objects
for transaction results and signed transactions (about 98 against 370 ns, and 16 against 61). State changes carry it
all: about 45 nested messages per item (StateChange → MapUpdateChange → key/value → Account → AccountID, Key, TokenID,
NftID …). With cached sizes a codec writes a nested message's tag and length and calls its static writeTo. A writer
opens a child, checks each field, and patches the length on close.

Profiling (JFR, -XX:+LogCompilation) shows that bookkeeping is what's left. Inlining and length moves are not:

• Cutting moves from 26.5 to 8.1 per state-changes item changed the time by about 7%.
• Exact sizing in writeFrom (zero moves) improved it by about 5%.
• Running under ParallelGC (no G1 barriers) changed nothing.

Options to close it, for discussion; none is in this PR:

1. writeFrom embeds nested in-build objects with their static codec, as x(Child obj) already does. That gives
   codec speed from objects, but then the generated per-message tests would only check top-level methods; child
   writers would remain covered by WriterTest, the fuzz test and BlockWriterTest's values path.
2. Unchecked writers, an option or a second generated class without the order and oneof checks. The issue asks for
   always-on checks.
3. Deferred length fix-up per top-level child: always reserve a fixed size and compact each block item once when
   it ends, instead of a patch (and possibly a move) per nested message.

Length reservation

Moves per item from values (over all 66,166 items; state changes alone in brackets):

	ADAPTIVE (default)	PER_TYPE	ONE_BYTE
Moves per item	3.4 (10.1)	11.8 (26.5)	2.7 (8.1)
writerFromValues ns/item	317.1	327.7	316.4

Mispredictions come mostly from writers whose message alternates around 128 bytes; MapChangeValue, for example, holds
accounts of about 150 bytes and token relations of about 50. ADAPTIVE and ONE_BYTE are within noise, PER_TYPE is 3%
slower. ADAPTIVE stays the default because it doesn't move every message of 128 bytes or more, every time, as ONE_BYTE
does. writeFrom sizes children from the objects, so it never moves bytes.

Varint stores

The issue asked to measure the single 8-byte store, which the buffer's slack makes safe without a guard, against the
exact-size stores. With the store, writerFromValues took 355.7 ± 14.8 ns and writerFromObjects 395.8 ± 24.4 ns,
against 317.1 and 372.4 ns. Exact-size stores stay, and the variant is removed.

Code size

Full HAPI build, writers on for every message (the max integration configuration): +6.36 MB of class files, +9.9%
of HAPI's generated main classes (70.82 MB against 64.46 MB), in 1,008 Writer classes. With writers on only for a
root such as Block, it is the reachable subset.

Performance work in this PR

• The inlining budget. First versions ran 415 ns from values. With several small buffer calls per field, a writer
  method for a 40-field message such as Account used about 150 bytes of inlined bytecode per field. C2 stops at
  8,000 (DesiredMethodLimit), so half of Account's fields became real calls. The fix was one buffer call per
  field, with a tiny inline common case.
• The auto-close path became one call, with close inlined into it, instead of four.

Found along the way (not changed here)

• The model's toBytes corrupts three wrapper values. For FloatValue(-0.0) and DoubleValue(-0.0), the size
  says 0 but 5 or 9 bytes are written. For UInt32Value of 2³¹ or more, the size uses the signed varint length (10
  bytes) but writes 5. Writers write these as protoc does; WriterTest.wrapperValuesTheModelMeasuresWrongAreWrittenAsProtocDoes
  checks against protoc. The generated test data never hits these values.
• PBJ's JSON codec indents a map inside a nested message differently from protoc, which the generated JSON test
  reports. That's why WriterMaps has writers of its own, rather than being a field of WriterRoot.

Docs

• docs/usage-guide.md: when to use writers against objects, a state-changes example, order, auto-close, aliasing,
  embeds, item byte ranges with a hashing example, allocation, reservation.
• docs/codec-protobuf.md: the option, its transitive closure, the per-kind API, output rules.
• docs/fast-codegen-plan.md: step 6, PR 12, and the lessons (per-nested-message cost, the single-store varint again).

🤖 Generated with Claude Code

[github-actions]

JUnit Test Report

   559 files  + 2     559 suites  +2   1m 12s ⏱️ +5s
 2 111 tests +27   2 107 ✅ +27   4 💤 ±0  0 ❌ ±0 
11 643 runs  +27  11 615 ✅ +27  28 💤 ±0  0 ❌ ±0 

Results for commit b42070f. ± Comparison against base commit c791de5.

♻️ This comment has been updated with latest results.

[github-actions]

Integration Test Report

    657 files  +     11      657 suites  +11   59m 6s ⏱️ + 12m 52s
164 362 tests +    179  164 362 ✅ +    179  0 💤 ±0  0 ❌ ±0 
652 325 runs  +163 078  652 325 ✅ +163 078  0 💤 ±0  0 ❌ ±0 

Results for commit b42070f. ± Comparison against base commit c791de5.

♻️ This comment has been updated with latest results.

[jasperpotts]

Synthetic HAPI workload (#1028), EPYC, ns per transaction, every option on:

	PBJ	Hand-written reference	Ratio	External ratio
Produce from values: writers	1,369 (0 B)	1,073 (0 B)	1.28×	1.7×
Produce: objects + writeReverse	1,410 (8,066 B)		writers 0.97×	0.9×
Block decode	1,442 (8,565 B)	1,072 (4,155 B)	1.34×	1.3×
Read every field, views	2,428 (22 B)	785 (0 B)	3.09×	2.7×
Transaction read every field, views	245	70	3.47×	2.4×

Three of the four validation ratios are within ±20%. The writers' ratio isn't, because of the reference encoder rather than the data; details are in #1028.

Also in this PR: with #1024 (#1026), wrapper writers now write FloatValue/DoubleValue of -0.0 as the model does (an empty wrapper), and UInt32Value of 2³¹ or more as protoc does. So writers stay byte-identical to toBytes.

🤖 Generated with Claude Code

[jasperpotts]

#1028 updated: the reference codec now writes back to front and reads in order, so it's a harder target. EPYC, synthetic HAPI block, ns per transaction, every option on:

	PBJ (#1028 head)	Reference	Ratio	External ratio
Produce from values: writers	1,359 (0 B)	758 (0 B)	1.79×	1.7×
Block decode	1,457 (8,565 B)	997 (3,750 B)	1.46×	1.3×
Read every field, views	2,400 (22 B)	682 (0 B)	3.52×	2.7×
Transaction read every field, views	244	56	4.36×	2.4×

Writers now validate against the external ratio; the views ratio is now 30% over it, as HAPI has more small nested messages per byte.

#1033 (writers, #1029): one-call setters for small messages (ids, timestamps; 439 in HAPI) and a leaner writer state. produceWriters 1,381 → 1,121 ns (−18.9%), still 0 B; 1.52× the reference in the same run, down from 1.79×.

🤖 Generated with Claude Code

[jasperpotts]

EPYC, synthetic HAPI block, ns per transaction, every option on:

#1034 (views, #1030): small oneofs in the in-order fast path, a shared slot per oneof, smaller resets, child views wrapped from the parent's checked cursor, unboxed wrapper accessors, and bytes fields read in place (xxxByteAt, xxxAs(view)). scanViews 2,415 → 2,140 (−11%), 22 → 0 B; txScanViews 244 → 215 (−12%). That's about 3.2× the reference for the block, from 3.5×.

#1035 (array parser, #1031): small oneofs in order, short lists sized once, and a varint read the JIT inlines. decode −3.5% and −414 B; scanObjects −11.5%; scanViews −5.9%; transactions −96 B. Decode is 1.38× the reference, from 1.46×.

The final numbers for the whole stack (0.16.4 vs defaults vs performance=max vs every option vs the reference) are running now; I'll post them here.

🤖 Generated with Claude Code

[jasperpotts]

Final numbers for the whole stack

These figures are for the synthetic HAPI block (#1028), in ns per transaction. Allocation per transaction is in brackets, and the ratio compares each figure with the hand-written reference codec in the same benchmark jar.

Setup: AMD EPYC 9275F, JDK 25.0.4.1, one CCD (taskset -c 0-2,24-26) of an idle machine, 3 forks × 5 iterations of 2 s, -prof gc.

Configurations:

• 0.16.4: the current release.
• Defaults: the stack (head of perf: 1031: cheaper nested decoding: oneofs in order, lists sized once, inlined varints #1035) with no options set.
• max, model only: performance = "max", model API only, without writers or views.
• Every option: max with writers and views.

ns/tx (alloc), × reference	0.16.4	Defaults	max, model only	Every option	Reference
Decode block	3,002 ± 121 (10,029 B) 3.02×	2,826 ± 90 (10,029 B) 2.86×	1,368 ± 9 (8,151 B) 1.39×	1,353 ± 7 (8,151 B) 1.37×	985-994 (3,750 B)
Decode, read every field	4,025 ± 186 (10,029 B) 5.98×	3,834 ± 122 (10,029 B) 5.66×	1,841 ± 50 (8,151 B) 2.90×	1,836 ± 40 (8,151 B) 2.73×	634-677 (0 B)
Encode block (model)	3,389 ± 161 (9,295 B) 4.37×	1,538 ± 15 (9,295 B) 2.05×	1,401 ± 16 (8,105 B) 1.89×	1,410 ± 25 (8,105 B) 1.85×	740-775 (0 B)
Encode block, back to front	–	–	–	989 ± 21 (8,105 B) 1.30×	740-775 (0 B)
Write block (writers)	–	–	–	1,122 ± 9 (0 B) 1.47×	740-775 (0 B)
Read every field (views)	–	–	–	2,014 ± 61 (0 B) 3.00×	634-677 (0 B)
Decode transactions	1,202 ± 105 (1,323 B) 11.88×	278 ± 7 (1,323 B) 2.60×	150 ± 2 (1,073 B) 1.33×	148 ± 2 (1,073 B) 1.27×	101-116 (701 B)
Decode tx, read every field	1,265 ± 103 (1,323 B) 25.24×	296 ± 10 (1,323 B) 5.99×	168 ± 1 (1,073 B) 3.37×	168 ± 3 (1,073 B) 3.23×	50-52 (0 B)
Read tx (views)	–	–	–	197 ± 1 (0 B) 3.77×	50-52 (0 B)

A second, identical machine agreed within about 3%. On it, writers were 1.54× and views 3.19× the reference.

In short:

• Defaults alone make block encoding 2.2× faster and transaction decoding 4.3× faster.
• performance = "max" makes block decoding 2.2× faster (1.4× the reference) and transaction decoding 8× faster (1.3×). It also cuts allocation by about a fifth.
• The writers encode with no allocation at 1.5× the reference.
• The views read with no allocation at about 3× the reference. Most of what remains is per-message bookkeeping.

🤖 Generated with Claude Code