			Installation Notes
		Broadcom FreeBSD User Space RoCE Library

		 	Broadcom
		 5300 California Avenue
	    	Irvine, California  92617

            Copyright (c) 2026 Broadcom Inc.
		    All Rights Reserved


Table of Contents
=================

  Introduction
  Driver Dependencies
  Building User space library
  Unloading and Removing RoCE lib
  Setup Verification

Introduction
============

This file describes the libbnxtre FreeBSD RoCE user space
driver for the Broadcom NetXtreme-C and NetXtreme-E BCM575xx/BCM576xx
10/20/25/40/50/100/200/400 Gbps Ethernet Network Controllers.

Driver Dependencies
===================

The user space RoCE driver depends on following kernel modules:

1. Ethernet driver for NetXtreme devices (if_bnxt.ko)
2. RoCE driver for NetXtreme devices (bnxt_re.ko)
3. uVerbs device shared lib, i.e. /lib/libibverbs.so.1

Building user space library
============================
1. Download and unpack the driver source files.
2. Navigate to the libbnxtre directory.
   ## cd freebsd-bnxt-227.0.24.X/libbnxtre
3. Compile the driver.
   ## make clean
   ## make
4. Copy the generated shared object file into /lib
   ## cp libbnxtre.so.1 /lib/

Note: On FreeBSD-15 and above OS's, please comment below line from Makefile
before executing the make command.
"VERSION_MAP=    ${_spath}/bnxt_re.map"


Unloading and Removing RoCE LIB
===============================
To uninstall libbnxtre, delete the shared object file from /lib
   ## rm -rf /lib/libbnxtre.so.1 

Setup Verification
==================

This section list the basic commands to verify the user space RoCE driver
configuration on the target host

List RoCE devices
-----------------

#ibv_devices
    device                 node GUID
    ------              ----------------
    bnxt_re1            be97e1fffedc99f1
    bnxt_re0            be97e1fffedc99f0


# ibv_devinfo

hca_id: bnxt_re1
        transport:                      InfiniBand (0)
        fw_ver:                         226.0.51.0
        node_guid:                      be97:e1ff:fedc:99f1
        sys_image_guid:                 be97:e1ff:fedc:99f1
        vendor_id:                      0x14e4
        vendor_part_id:                 5969
        hw_ver:                         0x4250
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

hca_id: bnxt_re0
        transport:                      InfiniBand (0)
        fw_ver:                         226.0.51.0
        node_guid:                      be97:e1ff:fedc:99f0
        sys_image_guid:                 be97:e1ff:fedc:99f0
        vendor_id:                      0x14e4
        vendor_part_id:                 5969
        hw_ver:                         0x4250
        phys_port_cnt:                  1
                port:   1
                        state:                  PORT_DOWN (1)
                        max_mtu:                4096 (5)
                        active_mtu:             1024 (3)
                        sm_lid:                 0
                        port_lid:               0
                        port_lmc:               0x00
                        link_layer:             Ethernet

Rebuild all IB tools
--------------------

Currently none of the InfiniBand tools (i.e. ibv_devices, rping, ibv_rc_pingpong, etc..)
are statically bound to libbnxtre.so.1 by default. To enable static binding of this library,
user need to rebuild all the InfiniBand tools. This means that user must go through the
process of rebuilding each IB tool to include static binding for libbnxtre.so.1

1. Rebuild libbnxtre:
Copy Sources :
# mkdir /usr/src/contrib/ofed/libbnxtre/
# cd /usr/src/contrib/ofed/libbnxtre/
# cp SIT_SRC/libbnxtre/*.[ch] .
# cp SIT_SRC/libbnxtre/bnxt_re.map .

Copy Makefiles:
# mkdir /usr/src/lib/ofed/libbnxtre
# cd /usr/src/lib/ofed/libbnxtre
# cp SIT_SRC/libbnxtre/Makefile .

Update “_spath” on the Makefile:
- update "_spath" to ${SRCTOP}/contrib/ofed/libbnxtre

Include LIBBNXTRE entry in /usr/src/share/mk/bsd.libnames.mk
---
LIBMLX5?=       ${LIBDESTDIR}${LIBDIR_BASE}/libmlx5.a
+ LIBBNXTRE?=       ${LIBDESTDIR}${LIBDIR_BASE}/libbnxtre.a
LIBMP?=         ${LIBDESTDIR}${LIBDIR_BASE}/libmp.a
---

Compile:
# cd /usr/src/lib/ofed/libbnxtre
# make
# make install
install  -C -o root -g wheel -m 444   libbnxtre.a /usr/lib/
install  -s -o root -g wheel -m 444   -S  libbnxtre.so.1 /lib/
install  -o root -g wheel -m 444    libbnxtre.so.1.debug /usr/lib/debug/lib/
install -l rs -o root -g wheel -m 755 -S   /lib/libbnxtre.so.1 /usr/lib/libbnxtre.so


2. Rebuild librdmacm & libibverbs :

2.1 List dynamic object dependencies of rping:

Currently, both ibv_devices & rping by default are not statically bound
to libbnxtre.so.1. It can be verified using 'ldd /usr/bin/rping' command output.
if this command output doesn't list 'libbnxtre.so.1' entry then it means that it is
not statically bounded to libbnxtre.so.1.

2.2 Navigate to the librdmacm directory:
#cd /usr/src/usr.bin/ofed/librdmacm

2.3 Include bnxtre in the Makefine.inc:
---
- LIBADD+=        ibverbs rdmacm irdma mlx4 mlx5 cxgb4 pthread
+ LIBADD+=        ibverbs rdmacm irdma bnxtre mlx4 mlx5 cxgb4 pthread
---

2.4 Include bnxtre entries on /usr/src/share/mk/src.libnames.mk
---
                 mlx5 \
+                bnxtre \
                 rdmacm \
---
  .if ${MK_OFED} != "no"
  _DP_pcap=       ibverbs mlx5
+ _DP_pcap=       ibverbs bnxtre
  .endif
---
  _DP_mlx5=       ibverbs pthread
+ _DP_bnxtre=     ibverbs pthread
  _DP_rdmacm=     ibverbs
---
  LIBMLX5DIR=     ${_LIB_OBJTOP}/lib/ofed/libmlx5
+ LIBBNXTREDIR=   ${_LIB_OBJTOP}/lib/ofed/libbnxtre
  LIBRDMACMDIR=   ${_LIB_OBJTOP}/lib/ofed/librdmacm
---

2.5 Compile librdmacm
# make
# make install
===> ucmatose (install)
install  -s -o root -g wheel -m 555   ucmatose /usr/bin/ucmatose
install  -o root -g wheel -m 444  ucmatose.debug /usr/lib/debug/usr/bin/ucmatose.debug
install  -o root -g wheel -m 444 ucmatose.1.gz  /usr/share/man/man1/
===> mckey (install)
install  -s -o root -g wheel -m 555   mckey /usr/bin/mckey
install  -o root -g wheel -m 444  mckey.debug /usr/lib/debug/usr/bin/mckey.debug
install  -o root -g wheel -m 444 mckey.1.gz  /usr/share/man/man1/
===> rping (install)
install  -s -o root -g wheel -m 555   rping /usr/bin/rping
install  -o root -g wheel -m 444  rping.debug /usr/lib/debug/usr/bin/rping.debug
install  -o root -g wheel -m 444 rping.1.gz  /usr/share/man/man1/
===> udaddy (install)
install  -s -o root -g wheel -m 555   udaddy /usr/bin/udaddy
install  -o root -g wheel -m 444  udaddy.debug /usr/lib/debug/usr/bin/udaddy.debug
install  -o root -g wheel -m 444 udaddy.1.gz  /usr/share/man/man1/

now 'ldd /usr/bin/rping'command output should list libbnxtre.so.1 entry.

2.6 Navigate to the libibverbs directory
#cd /usr/src/usr.bin/ofed/libibverbs

2.7 Navigate to the libibverbs directory
Follow steps 2.3 to 2.5 to compile libibverbs


3. Build Perftest tools to statically bound to libbnxtre.so.1:

3.1 Install perftest
# cd /usr/ports/benchmarks/perftest
# make

3.2 Edit Makefile and include libbnxtre (for the static binding) as applicable
# vim work/perftest-4.5-0.17/Makefile
<< Include libbnxtre as applicable >>

User should see below lines when he greps for bnxtre string as shown below,
# grep bnxtre -i  work/perftest-4.5-0.17/Makefile
LIBBNXTRE = -lbnxtre
ib_send_bw_LDADD = libperftest.a $(LIBUMAD) $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_send_lat_LDADD = libperftest.a $(LIBUMAD) $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_write_lat_LDADD = libperftest.a $(LIBMATH)  $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_write_bw_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_read_lat_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_read_bw_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_atomic_lat_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
ib_atomic_bw_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
raw_ethernet_bw_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
raw_ethernet_lat_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
raw_ethernet_burst_lat_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)
raw_ethernet_fs_rate_LDADD = libperftest.a $(LIBMATH) $(LIBMLX4) $(LIBMLX5) $(LIBBNXTRE) $(LIBEFA)

3.3. execute make and make install commands.

Now 'ldd /usr/local/bin/ib_send_bw' command output should list the 'libbnxtre.so.1' entry.


Traffic Test
------------

Server: rping -s -d -v -a <ip of server bnxt interface>
For example
rping -s -a 192.172.1.1 -Vv -C 3
server ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
server ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
server ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

Client: rping -c -d -v -a <ip of server bnxt interface>
For example
rping -c -a 192.172.1.1 -C 3 -vV
ping data: rdma-ping-0: ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqr
ping data: rdma-ping-1: BCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrs
ping data: rdma-ping-2: CDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrst

