Conversation
Implement the notified communication interface (MPI Standard section 12.6) in the osc/rdma component, which covers the libfabric/ofi BTL path: - MPI_PUT_NOTIFY / MPI_GET_NOTIFY, incrementing the target's notification counter with a btl atomic after the data movement has completed there - MPI_WIN_GET_NOTIFY_VALUE / MPI_WIN_RESET_NOTIFY_VALUE, using a btl compare-and-swap when it is not safe to mix CPU and btl atomics - MPI_WIN_SET_NUM_NOTIFY / MPI_WIN_GET_NUM_NOTIFY, publishing each rank's attached counter count to the group with an allgather A fixed region of OMPI_OSC_RDMA_NOTIFY_MAX counters is allocated in the window state so origins can update them with btl atomics; only the first notify_counts[rank] are considered attached. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add a btl interface for counters that the network adapter increments at the target once a remote operation completes there, and use it in osc/rdma so a notified put or get costs a single network operation instead of a data transfer followed by an atomic to announce it. opal/mca/btl: - MCA_BTL_FLAGS_NOTIFIED_RMA, set by a btl that provides btl_register_notification, btl_deregister_notification, btl_notification_read and btl_notification_wait - registering a region this way yields an ordinary registration handle alongside the counter, so an origin targets that handle in a put or get and the target's adapter counts the operation as a side effect - counters are monotonic and never reset by the btl; a consumer that needs a reset records a base value and subtracts it opal/mca/btl/ofi: - a notification is an fi_cntr bound to a memory region with FI_REMOTE_WRITE | FI_REMOTE_READ, so it requires the provider to have been opened with FI_RMA_EVENT - FI_RMA_EVENT is requested as an optional capability: fi_getinfo treats requested caps as mandatory, so the query is retried without it before any other fallback, since counters are an optimization and should be the first thing given up - these registrations bypass the rcache, which keys on the address range and would hand back one registration where several distinct counters over the same buffer are wanted - btl_ofi_enable_rma_event MCA parameter to suppress the request ompi/mca/osc/rdma: - each rank registers its window base once per attached notification index and publishes the resulting handles to the group; an origin picks a counter by choosing which handle to target - the accelerated path is taken only when the transfer maps to exactly one btl operation, since the counter counts operations and the standard requires exactly one increment per notified call; local peers, non-contiguous datatypes, size mismatches, transfers over the put/get limit and unaligned gets fall back to the atomic path - MPI_WIN_RESET_NOTIFY_VALUE records the raw counter value as a base rather than resetting it, which avoids racing increments in flight Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Request FI_RMA_EVENT only from the selected provider, register with FI_RMA_EVENT, and bind FI_REMOTE_WRITE only (reads are not counted). Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Agree on errors before set_num_notify changes state, report bounds, make reset safe against btl atomics, and use atomics for get_notify. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
Also unlink the grown counter segment when an attach fails. Signed-off-by: Joseph Antony <jajoseph.antony18@gmail.com>
joe-explr
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.