Running a neural network on a robot is rarely a question of accuracy. It is a question of where the arithmetic happens, how much power that costs, and whether the answer arrives in time. Cormorant is a free, open-source project that puts that arithmetic on an FPGA — and, unusually for this corner of engineering, it does so without asking you to write a single line of hardware description language.
The repository is at github.com/GradeBuilderSL/cormorant, released under the Apache 2.0 licence.
What problem an FPGA accelerator actually solves
A robot perceiving the world runs the same small set of operations over and over: convolutions, matrix multiplications, pooling, activation functions. On a CPU these are executed by a general-purpose machine that must fetch and decode every instruction. On a GPU they run fast but draw power a battery-driven platform may not have to spare.
An FPGA sits between the two. You lay out arithmetic units in silicon that match your specific network, and data flows through them without instruction fetch, without cache misses and with latency you can calculate rather than measure. The cost of that determinism has always been the effort: describing hardware is a different discipline from writing Python.
Cormorant’s premise is that for the common case you should not have to. You hand it an ONNX model; it hands you a C project that drives ready-made hardware kernels.
What is inside
The project has two halves that meet in the middle.
The first half is a set of four hardware kernels written in Vitis HLS — C++ that the Xilinx toolchain synthesises into logic:
- VectorOPKernel — element-wise
Add,Sub,Mul,Div,ReluandClip(0,6), with the operation selected at runtime rather than at synthesis time, at one result per clock cycle. - MatmulKernel — tiled, batched general matrix multiply, with batch strides that let a strided view be broadcast without copying data.
- ConvKernel — 2-D convolution in NCHW layout with stride, dilation, padding and optional bias.
- PoolingKernel — max, average and Lp pooling, including the global variants.
All four default to ap_fixed<16,8>, a 16-bit fixed-point type with 8 integer bits, and share the same saturating cast so that overflow clamps instead of wrapping. The type is a CMake parameter, so it can be changed for a model that needs different range.
The second half is the inference scheduler: a Python program that reads an ONNX file, works out which kernel each node maps to, and emits a self-contained C project that runs the model end to end. Two model constructs cost nothing at all — Reshape becomes a pointer alias rather than a copy, and Gemm is decomposed into MatMul plus an optional Add when the model is loaded.
The target board
Cormorant targets the AMD Kria KV260 — a compact development board with a Zynq UltraScale+ device, popular precisely because it sits at the size and power budget of a real robot rather than a rack.
Adding another board does not require touching the kernels. Platforms are described by a JSON file: the FPGA part, the clock, and the compile-time bounds each kernel is synthesised against. Both the C++ build and the Python scheduler read that same file, and a missing field stops the build rather than producing something that silently misbehaves later.
What you can run without any hardware
This is the part worth knowing if you are learning rather than shipping. The kernels come with simulation tests that need no FPGA at all:
mkdir build && cd build
cmake ../
make TestSimulation # VectorOPKernel
make TestConvRef # ConvKernel
make TestMatmulRef # MatmulKernel
make TestPoolingSim # PoolingKernel
ctest # all fourThe Python side ships a test suite of some 1300 tests that generates its own models and checks the compiler against them. You can read the kernels, change a tiling parameter, and see the effect in simulation before ever touching a board.
The three demos
Each demo follows the same download, generate, deploy sequence behind a single command:
- MNIST — a small convolutional network and LeNet run over the 10 000 test images, reporting top-1 accuracy and per-image latency.
- Image classification — MobileNetV1 at 224×224 over static images, reporting top-5 ImageNet predictions.
- Camera — the same network against a live RealSense stream, with annotated frames sent back over SSH.
The camera demo is the honest one. Accuracy figures on a fixed dataset tell you the model works; a live stream tells you whether the whole path — capture, transfer, inference, return — holds together at a usable frame rate.
Where this fits in a robotics curriculum
Most people meet neural network deployment as a software problem: quantise the model, pick a runtime, measure. Cormorant makes the hardware layer legible without demanding a hardware background. You can see exactly which operations became silicon, what the memory interfaces look like, and where the data crosses between the processor and the fabric.
That maps onto several things worth understanding in their own right — fixed-point arithmetic and why saturation matters, memory bandwidth as the real constraint in accelerator design, the difference between throughput and latency, and why an operation that costs nothing on paper can dominate a design once data movement is counted.
If you want the surrounding context, our knowledge base covers edge compute, sensors and deployment in more depth, and the course catalogue takes the model side further.
Getting started
Clone the repository, initialise the submodule, and run the simulation tests — that path needs nothing but a compiler and CMake. Synthesis requires an installed Vitis toolchain, and running on the board requires a KV260, but neither is needed to read the code or to understand what it does.
The project is Apache 2.0 licensed, which means you can use it commercially, modify it and redistribute it, with the usual requirement to preserve the notices.
