This project provides an interactive library designed to automatically synthesize parsable binary files that describe a Dynamic Neural Network (DyNN) and an Auxiliary DNN, employed in an inference process described in details in the paper to be appeared in MobiHoc 2025: "FlexRFML: Dynamic Neural Networks on FPGAs for Next-Generation Radio Spectrum Perception."
The .joblib files contain all the necessary information for the library to instantiate application-specific sub-units for each layer. The system converts the original PyTorch model into an HLS-compatible intermediate format, which is then transformed into synthesizable RTL code.
The library leverages High-Level Synthesis (HLS) to fully automate the generation, packaging, and integration of the resulting IP cores into the FPGA toolchain's IP catalog.
-
cnn_lib: This directory contains a set of libraries offering parametric descriptions of layer sub-units. These descriptions are automatically mapped to the layer hyperparameters (e.g., kernel size, stride, padding) during parsing. The DynamicDDrUnpiped library, in particular, is extensively used in the paper to evaluate various prototypes. It models both static and dynamic sub-units, designed to fetch weights from DDR memory via an AXI-Full interface. The latter resorts to gating vectors, parsed by the Auxiliary DNN through a Program Buffer, to dynamically compute memory offsets and access the scattered input feature maps and weights. The library includes two .tcl scripts, which are automatically selected and executed by the system to synthesize the IP cores and add them to the IP catalog.
-
input: This directory contains the binary files with the parsable description of the algorithm defined in PyTorch (e.g., deep.joblib). An example demonstrating how to parse a simple computer vision model (LeNet-5 trained on MNIST) is provided in ./Training/TrainingMax.py. This directory also includes input/output binary files (such as aM.joblib) that contain both the input tensor and the expected output of the inference process. These files can be used to validate both the behavioral and the synthesized implementation of the accelerator. If the user selects either the Unpiped or PIPOpipe libraries during synthesis, these are the only required binary files. However, in the context of Dynamic DNNs, the library also requires the description of the Auxiliary DNN (Aux.joblib) and the ROM content file (rom.csv). These additional files are used by the library to automatically generate the ROM sub-unit within the auxiliary accelerator.
-
PetalinuxDrivers: This directory contains the PetaLinux applications intended to be executed from user space. Two drivers are provided, depending on whether the user plans to deploy a standard static DNN or a Dynamic Neural Network (DyNN). In both cases, the user must manually update the physical (PHY) addresses corresponding to the components instantiated in the Programmable Logic (PL) within the Vivado design. These applications are provided as skeleton code, intended to be adapted and customized for specific use cases. For this reason, we strongly encourage users to review and understand the code thoroughly before integrating it into their PetaLinux development workflow.
-
tools: This directory contains several compilers used by the library according to the selected synthesis flow, which is configured through flags. The tool requires a pre-installed version of Vitis HLS and an active terminal session with the tool available. The Python scripts fetch the .joblib files and generate an intermediate HLS representation of the system in the output directory. Additionally, all weights are consolidated into a single file, which can be loaded into either BRAM for high-speed execution or DDR memory for accelerators with hardware resource constraints.
-
output: contains the synthetized IPs. Please include this directory in the Vivado IP catalog to include the accelerators in the block diagram and generate the bitstream. The IPs are intended to be deplyed in the ZCU102, pleaase modify the .tcl files in ./cnn_lib/DynamicDDrUnpiped to target a different SoC. We provide an example of the content of the output dir, please monitor it and remove the output directory before running the library code (rm -rf output)
-
Please Download Vitis_Libraries: We work with the 2023.2 version so checkout in the corresponding branch. For simplicity the libraries have been included through absolute paths. Therefore, as a first step please update all header files in ./cnn_lib/DynamicDDrUnpiped with the absolute path of the Vitis_Libraries in your local matchine. For example update line 4 of ./cnn_lib/DynamicDDrUnpiped/conv.h Modify the path to absolute in the Vitis_Libraries: open in sudo mode the following file: Vitis_Libraries/vision/L1/include/common/xf_params.hpp and modify line 31 to: #include "/home/fpessia/Documents/Vitis_Libraries/vision/L1/include/common/xf_params.hpp" with the prope user name
git clone git@github.com:Xilinx/Vitis_Libraries.git
-
Install python modules:
pip3 install joblib pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
-
Run the synthesis flow:
source /tools/Xilinx/Vitis_HLS/2023.2/settings64.sh python3 ./tools/genPyTorch.py --file ./input/deep.joblib --input ./input/aM.joblib --pipe 3 --aux ./input/Aux.joblib --rom ./input/rom.csv #pipe 3 selects the cnn_lib/DynamicDDrUnpiped library
-
To speed up code verification: The default configuration accelerates a VGG10 model trained for modulation classification. The training scripts can also be provided upon request if required by the committee. Due to the complexity of the accelerator, we strongly recommend that users avoid running both synthesis and co-simulation during initial code verification, as these steps are computationally intensive and may take considerable time on a local machine. To skip these stages, please comment out lines 24 and 26 in ./cnn_lib/DynamicDDrUnpiped/SynthesisAux.tcl and lines 43 and 46 in ./cnn_lib/DynamicDDrUnpiped/SynthesisDeepVGG.tcl.
-
Monitor output : Two behavioral simulations are performed. The first simulates the Auxiliary DNN, which produces the program that will be executed by the DyNN. The second simulates the complete inference process, assuming that the program has already been loaded into the program buffer by the Auxiliary DNN (see the paper for details). To verify the correctness of the behavioral simulation, compare the golden values provided in ./output/goldenvalues.h with the C simulation results. These results can be viewed either directly in the terminal during execution or in the vitis_hls.log file located in the output directory. If the two vectors match, the behavior of the IP is considered verified.
