feat(api): bound transaction field occurrences before parsing

[waynercheung]

What does this PR do?

Adds an admission check that counts field occurrences in a submitted Transaction before it is parsed, and rejects it once it exceeds 50,000. It applies to the API entry points that accept a binary Transaction:

• gRPC: BroadcastTransaction, GetTransactionSignWeight, GetTransactionApprovedList, GetShieldTransactionHash, CreateCommonTransaction (the Wallet methods whose request type is Transaction)
• HTTP: /wallet/broadcasthex

JSON transaction endpoints and gRPC methods with other request types are unchanged.

Changes:

• TransactionAdmissionGuard scans the wire bytes with CodedInputStream (the decoder the generated parser uses) and counts every tag, known or unknown. It recurses into sub-messages by descriptor; bytes and string fields are leaves, so large bytecode counts once. For each Contract it first resolves the effective type, type_url and value under protobuf merge rules (last occurrence wins, repeated parameter occurrences merge), then recurses into that value with the payload type TransactionCapsule.getOwner would unpack. A value that is never unpacked (overwritten, mismatched type_url, unknown type) stays a leaf. Group encoding and nesting deeper than 32 are rejected. The guard checks structure only; it does not validate field contents such as UTF-8 in strings.
• GuardedTransactionMarshaller replaces the request marshaller of those gRPC methods. It reads the request with the same bound as maxInboundMessageSize, runs the guard and passes the same bytes to the original marshaller, so parsing is unchanged for accepted requests.
• A rejection is returned as a per-reason sentinel instead of an exception, because grpc-java closes a call whose marshaller throws with UNKNOWN and logs a SEVERE stack trace each time. AdmissionCheckedHandler recognizes the sentinel by identity and closes the call with INVALID_ARGUMENT; the service method is never invoked.
• RpcService.guardTransactionMethods rebuilds each service definition with the guarded marshaller and handler for methods whose proto input type is Transaction. All services in RpcApiService, RpcApiServiceOnSolidity and RpcApiServiceOnPBFT are registered through it, so a Transaction method added later is covered too. Only the five FullNode Wallet methods match today.
• BroadcastHexServlet runs the guard before Transaction.parseFrom.
• Admission rejections use one client message, transaction rejected by admission check, and a sampled DEBUG log (AdmissionRejectLog: at most one line per 10 seconds per transport, with the count since the last line, no stack trace).

Block, database and P2P parsing do not use the guard: a valid block must never fail a local, non-consensus limit.

Why are these changes required?

protobuf-java materializes objects for repeated sub-message and unknown-field occurrences. A submitted Transaction containing many empty ret entries (2 bytes on the wire each) can therefore expand to tens of times its wire size on the heap during parseFrom (a 4 MiB request retained about 265 MiB in a local measurement). Empty raw.contract and raw.auths entries and unknown fields with distinct numbers behave the same way. Existing transaction-level checks, such as the transaction size check and signature verification, run after this allocation.

The outer parse keeps the contract payload (Any.value) as bytes, but getOwner unpacks it before signature verification, exposing its nested messages to the same amplification. Checking only the outer message is not enough.

Counting occurrences limits the structural complexity of the scanned schema, and with it the related object allocation, without field-specific rules. It is not an exact object count or heap size; the existing byte limits on requests stay in place.

Behavior changes (release note)

Transport and admission behavior was compared on the same inputs using a local differential harness with stubbed business handlers (gRPC) and a copy of the base servlet with a mocked Wallet (HTTP). Production business-response mappings were additionally checked against source.

• Requests exceeding 50,000 occurrences, using group encoding, nesting deeper than 32, or containing wire-malformed effective contract payloads detected by the guard (including in a second contract) are rejected before generated protobuf materialization:
  • gRPC: INVALID_ARGUMENT with description transaction rejected by admission check. Previously, inputs that failed to decode returned UNKNOWN ("Application error processing RPC") with a SEVERE log per request; inputs that decoded reached the service method and returned its result (OK with an error code in the body for most methods, mostly INTERNAL for GetShieldTransactionHash).
  • /wallet/broadcasthex: {"Error":"transaction rejected by admission check"} with HTTP 200. Previously a decodable request returned result/code/txid, and a malformed one returned internal server error.
• For requests newly rejected by the guard that would already have failed downstream broadcast validation, only the error response changes; broadcasthex no longer includes a txid.
• A contract deployment whose ABI exceeds the budget is rejected at these API entry points. With ordinary entries (a name, two inputs, one output) this starts at about 4,170 entries; 3,500 entries (42,019 occurrences, 191 KB) pass the admission guard. These are synthetic encoding and guard compatibility samples, not end-to-end validated deployments. The occurrence limit applies only to these API entry points; P2P admission and consensus validation rules are unchanged.
• Logging: the per-request SEVERE (gRPC) and the per-request DEBUG with stack trace (HTTP request failed, broadcasthex) are replaced by the sampled DEBUG line for these rejections. Exceptions raised inside service methods are unchanged.
• Metrics: tron:grpc_service_latency_seconds has only an endpoint label. Rejected calls are now closed through the intercepted call and are counted in this histogram; previously decode failures were closed by grpc directly and were not.
• Unchanged:
  • Content errors that the guard does not check, such as invalid UTF-8 in a string field of the contract payload, still fail later at unpack as before.
  • On broadcasthex, invalid hex and invalid UTF-8 in string fields of the outer Transaction (for example Any.type_url) still return internal server error.
  • gzip decompression over maxInboundMessageSize still returns UNKNOWN. A corrupt gzip header still fails inside grpc before the marshaller runs.
  • Uncompressed oversize requests still return RESOURCE_EXHAUSTED.
  • A mismatched type_url does not itself cause an admission rejection; normal business validation is unchanged.

Budget

• Valid traffic: one week of mainnet blocks (80.2 million transactions) has a maximum of 922 occurrences (a contract deployment), p99.99 of 20, and no transaction that the guard would reject.
• Allocation: across the tested payload shapes (20 shapes on x86_64/JDK 8 CMS under Rosetta and ARM64/JDK 17 G1/ZGC), the maximum observed cumulative allocation per request was approximately 41 MiB for guarded gRPC marshalling and 133 MiB for broadcasthex processing. These are not peak-live-heap bounds or whole-node availability guarantees, and the JDK 8 timings are not native x86_64 latency evidence.
• HTTP body reading, JSON parsing and hex decoding occur before the guard and still incur allocation for rejected requests (about 70 MiB for the largest rejected broadcasthex request in the same measurements).

This PR has been tested by:

• Unit Tests
• Manual Testing

New tests (61):

• TransactionAdmissionGuardTest (28): counts and exact budget boundaries for each amplification shape; merge semantics (last type, type_url and value win, explicit empty overrides); payload recursion including shielded and multiple contracts; first-pass budget; malformed input; depth; concurrency; a corpus of valid transactions; the pinned budget value; no packed repeated scalar reachable from Transaction or any contract type.
• AdmissionSupportTest (12): reason mapping, sentinel identity, bounded read edge cases, log sampling.
• RpcServiceAdmissionTest (12): real loopback Netty server. Valid requests pass (plain and gzip); over-budget, malformed, invalid UTF-8 and corrupt gzip bodies that reach the marshaller get INVALID_ARGUMENT without reaching the service method and without a grpc SEVERE log; interceptor permits are released; cancellation; disabledApi still applies; message size limits unchanged.
• RpcServiceRegistrationAdmissionTest (4): the definitions actually registered by each RpcService subclass; exactly five methods guarded on FullNode, none elsewhere; each guarded marshaller uses the production budget and maxMessageSize.
• BroadcastHexServletAdmissionTest (5): fixed response without calling Wallet, one sampled DEBUG line without a stack trace.

Selected existing suites, run locally on JDK 17 (aarch64) and JDK 8 (x86_64), 284 tests each, all passing:

./gradlew :framework:test --tests 'org.tron.core.admission.*' \
  --tests 'org.tron.common.application.*' \
  --tests 'org.tron.core.services.RpcServiceRegistrationAdmissionTest' \
  --tests 'org.tron.core.services.http.BroadcastHexServletAdmissionTest' \
  --tests 'org.tron.core.services.RpcApiServicesTest' \
  --tests 'org.tron.core.net.*'

In addition, all of org.tron.core.services.* (125 classes) was run together with the new tests on JDK 17 and passed. checkstyleMain and checkstyleTest pass.

Follow up

• P2P inbound messages are not covered and need a separate design: an over-budget transaction there must not disconnect the peer, since honest nodes may relay large transactions that exceed this budget.
• gzip decompression over the size limit still logs a SEVERE stack trace per request on every gRPC method. This predates this PR and will be handled together with request byte limits, mapping it to RESOURCE_EXHAUSTED.
• Performance: hand the original marshaller a KnownLength stream to restore grpc's array fast path, and simplify the bounded read; to be measured separately.

Extra details

Design choices that reviewers may ask about:

• Not configurable: 50,000 is about 54 times the largest value seen on mainnet, and allocation was measured at this value. Changing it should come with new measurements.
• TransactionAdmissionGuard.countTransaction is public so that sampling tools use exactly the production counting logic.
• TOO_LARGE is a fallback. grpc enforces the same inbound limit first, but the bound keeps the marshaller's allocation independent of transport settings.
• The original marshaller still parses the request (instead of Transaction.parser()) so that its configuration and behavior are kept.
• The outer isDebugEnabled() check in the handler avoids evaluating the method name, the remote address and the argument array when DEBUG is off.