				       README Notes
				Broadcom bnxt FreeBSD Driver

	                             Broadcom Inc.
				15101 Alton Parkway,
				   Irvine, CA 92618

			  Copyright (c) 2026 Broadcom Inc.
				   All rights reserved


Table of Contents
=================
  Introduction
  Driver Dependencies
  Driver Download and Compilation
  Unloading and Loading Driver
  Driver Settings
  Port Speeds
  Interface Driver Statistics
  Error Recovery
  KTLS Support
  SR-IOV Support
  enable_agg_rings sysctl option
  Link Down Reason Support
  Known Issues

Introduction
============
This file describes the bnxt FreeBSD driver for the Broadcom NetXtreme-C
and NetXtreme-E BCM573xx/BCM574xx/BCM575xx/BCM576xx 10/20/25/40/50/100/200/400 Gbps
Ethernet Network Controllers.


Driver Dependencies
===================
The driver has no dependencies on user-space packages and necessary
firmware must be programmed in NVRAM.


Driver Download and Compilation
===============================
1. Download and unpack the driver source files.
2. Navigate to the bnxt_en directory.
   ## cd freebsd-bnxt-227.0.24.X/bnxt_en
3. Compile the driver.
   ## make clean
   ## make
      Note: The Makefile's .PATH variable points to /usr/src/sys/dev/bnxt/bnxt_en/
            for the source files, but if there are any source files in the current
	    working directory (pwd), they will take precedence over the ones
	    specified in .PATH variable.
4. The driver binary will be generated in the current directory.
   ## ls ./if_bnxt.ko

Unloading and Loading Driver
=============================
1. To unload:
   ## kldunload if_bnxt

2. To load:
   A. Navigate to the directory containing the driver binary
      ## cd /usr/obj/usr/src/amd64.amd64/sys/modules/bnxt_en/
      (or)
      ## cd freebsd-bnxt-227.0.24.X/bnxt_en
   B. Load the driver
      ## kldload ./if_bnxt.ko

Driver Settings
===============
Commands to assign IP, MTU, down/up, speed change, etc..
   ## ifconfig
   ## ifconfig -a
   ## ifconfig bnxt0 12.0.0.10/24
   ## ifconfig bnxt0 mtu 9000
   ## ifconfig -m
   ## ifconfig -m bnxt0
   ## ifconfig bnxt0 media 25GBase-CR

Port Speeds
===========
On dual-port devices, the port speed of each port must be compatible with
the port speed of the other port.  10Gbps and 40Gbps are one set of
compatible speeds.  25Gbps and 50Gbps are the other set of compatible
speeds.  The 2 sets are not compatible with each other and will result
in link loss if one port uses a speed that is incompatible with that of
the other port.


Interface Driver Statistics
============================
$sysctl dev.bnxt.0

Error Recovery
==============
Firmware error recovery is supported for Thor and onwards adapters.
Hardware and Firmware can run into two types of FAULT/error-
1. Firmware can detect the error and signal driver to take further action.
2. Error results into firmware in unresponsive(dead/hung) state.

Driver supports only type 1 type of errors/faults recovery on a best-effort basis.
This type of error recovery can be called as “Firmware initiated error recovery”.
e.g. Mbuf leak, losing pci credits, re_flush, firmware unhandled exception etc.

To trigger error recovery, the user can use the nicclid tool as shown below:

nicclid -i <x> fw --cli "crash null_ptr"    (where <x> = 1, 2, ... controller index)

Note: The tool may be named either nicclid or nicclid.freebsd depending on the environment.

Limitations:
1. For firmware error recovery to function correctly, all the interfaces of its
   PFs must be in Up state (ifconfig <interface> up). Failing to do so may lead
   to unpredictable behavior, necessitating a reboot to restore normal operation.
2. Driver attempts error recovery on best-effort basis but few a times driver may abort firmware recovery.
3. In intial driver releases, few debug prints are added in firmware error recovery part for debug purpose.
   These prints can be tuned once this feature is hardened.

KTLS Support:
============
Here is the quick overview of the kTLS (Kernel TLS) support for Broadcom
BCM576xx based NICs within the FreeBSD driver, including details on hardware
offload for Tx kTLS. Further specifications on this feature can be found in
the BCM576xx Product Specification. This information focuses specifically on
the FreeBSD driver testing of this feature.

FreeBSD kernel has undergone changes to support kTLS HW offload for
Thor2 and onwards NICs. Below are the commit IDs of core changes in
the kernel to enable HW kTLS offload in if_bnxt driver:

4d692068f685 ktls: Capture initial tls seqno at time offload is initiated
43d7ee540efe iflib: support for transmit side nic KTLS offload

If the kernel has above two commit IDs included, then skip the next
section- 'Prepare kernel with HW kTLS support' and jump directly to
'kTLS testing' section.

Prepare kernel with HW kTLS support:
------------------------------------

This section is meant for the kernels which do NOT have the above
mentioned commit IDs included. In other words, the kernel which lacks
the HW ktls support in iflib has to be patched and recompiled with the
ktls support, below are the steps to do so:

1. Apply the kernel support patches (included in the driver source) to the FreeBSD development kernel.
Run the following command from the kernel source directory (typically `/usr/src`):

patch -p1 < $IF_BNXT_SRC/patches/ktls_iflib_support_fbsd<OS_VERSION>.x.patch
patch -p1 < $IF_BNXT_SRC/patches/ktls_record_sequence_support_fbsd<OS_VERSION>.x.patch


2. The kernel should be compiled and installed:

make buildkernel KERNCONF=GENERIC -j128
make installkernel KERNCONF=GENERIC -j128

3. Reboot the system to boot to the patched kernel.

4. Edit driver's Makefile and add below lines:

KERNBUILDDIR=/usr/obj/<installed_kernel_path>/amd64.amd64/sys/GENERIC/
CFLAGS+= -DKTLS_IFLIB_SUPPORT -DKTLS_REC_SEQ_NUMBER


kTLS testing:
-------------

1. Compiled and load the driver and run below commands to configure kTLS Tx offload:

sysctl kern.ipc.tls.enable=1
sysctl kern.ipc.mb_use_ext_pgs=1
sysctl kern.ipc.tls.ifnet.permitted=1
ifconfig bnxt<N> txtls    where 'N' is the network interface of Thor2

2. Verify kTLS is supported by the driver ifconfig. TXTLS4 and TXTLS6 flags indicate that kTLS is supported by the driver.

bnxt0: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=1ce507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,HWSTATS,MEXTPG,TXTLS4,TXTLS6>
        ether 14:23:f2:5f:df:40
        media: Ethernet autoselect (400G-AUI8-AC <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


3. Create certificate on DUT and PEER:
openssl req -newkey rsa:2048 -new -nodes -x509 -days 3650 -keyout key.pem -out newreq.pem

4. Initial basic testing can be performed using a special version of iperf that has added support for TLS.
   For advanced testing, nginx is being used.

   A. Testing with iperf:  
      * The source can be cloned from this git tree: https://github.com/Mellanox/iperf_ssl.git

      * After cloning the git tree, run ./configure and then edit src/Makefile and append “-lssl -lcrypto” to the following line:

		LIBS = -lrt

      * Run make
      * Use the --tls option on the iperf server and client to enable TLS.
		--------------
		Client side(DUT): ./iperf --tls=v1.2 -c <server_ip> -t <runtime> -i <interval>
		Server side(Peer): ./iperf --tls=v1.2 -s
		--------------

   B. Testing with nginx:
      It's beyond the scope of this document.


5. Few sysctls are added to traffic statistics can be monitored:

	sysctl -a | grep ktls_stats | grep bnxt

   Here is the sample output of stats:

        ----------
	dev.bnxt.4.ktls_stats.ktls_stats.tce_free: 0
	dev.bnxt.4.ktls_stats.ktls_stats.invalid_rec_sn: 0
	dev.bnxt.4.ktls_stats.ktls_stats.zero_payload: 0
	dev.bnxt.4.ktls_stats.ktls_stats.invalid_tag: 0
	dev.bnxt.4.ktls_stats.ktls_stats.unsupported_tls_hdr: 0
	dev.bnxt.4.ktls_stats.ktls_stats.tce_bad: 0
	dev.bnxt.4.ktls_stats.ktls_stats.tce_ready_delayed: 927
	dev.bnxt.4.ktls_stats.ktls_stats.tce_not_ready: 927
	dev.bnxt.4.ktls_stats.ktls_stats.mpc_failures: 0
	dev.bnxt.4.ktls_stats.ktls_stats.ctx_alloc_failures: 0
	dev.bnxt.4.ktls_stats.ktls_stats.ktls_tx_uninit: 0
	dev.bnxt.4.ktls_stats.ktls_stats.mpc_cmd_timeout: 0
	dev.bnxt.4.ktls_stats.ktls_stats.add_tx_session_failure: 0
	dev.bnxt.4.ktls_stats.ktls_stats.mbuf_errors: 0
	dev.bnxt.4.ktls_stats.ktls_stats.mbuf_frees: 4714826
	dev.bnxt.4.ktls_stats.ktls_stats.mbuf_allocs: 4714826
	dev.bnxt.4.ktls_stats.ktls_stats.seq_fwd_tx_replay_pkts: 16390
	dev.bnxt.4.ktls_stats.ktls_stats.seq_fwd_tx_silent_drops_pkts: 0
	dev.bnxt.4.ktls_stats.ktls_stats.seq_fwd_tx_tls_hdr_pkts: 16390
	dev.bnxt.4.ktls_stats.ktls_stats.seq_fwd_tx_pkts: 16390
	dev.bnxt.4.ktls_stats.ktls_stats.replay_tx_pkts: 4714826
	dev.bnxt.4.ktls_stats.ktls_stats.retrans_tx_pkts: 46784142
	dev.bnxt.4.ktls_stats.ktls_stats.ooo_tx_pkts: 46800532
	dev.bnxt.4.ktls_stats.ktls_stats.failed_tx_pkts: 0
	dev.bnxt.4.ktls_stats.ktls_stats.hw_tx_pkts: 1639611009
	dev.bnxt.4.ktls_stats.ktls_stats.delete_tx_session: 271640
	dev.bnxt.4.ktls_stats.ktls_stats.add_tx_session: 273288
	---------

SR-IOV Support:
===============

Overview:
---------

SR-IOV (Single Root I/O Virtualization) support allows a single PF (Physical Function)
network interface to present multiple VFs (Virtual Functions) to the OS and VMs.

The if_bnxt driver supports SR-IOV on Broadcom NetXtreme-E BCM575XX and BCM576XX series of
controllers.

The driver does not enforce a strict limit on VF creation, but supported configurations are:

- On FreeBSD 13.x and 14.x, only one VF per PF is supported. Creating more than one VF may
  appear to work, but behavior is undefined and not recommended.
- On FreeBSD 15.0, up to eight VFs per PF are supported. Creating more than eight VFs per PF
  is not supported, and results are undefined and not recommended.


- VF creation/deletion via `iovctl`
- VF passthrough to bhyve (hypervisor)

Prerequisites:
--------------

1. Kernel Configuration

   - Build kernel with options PCI_IOV in kernel config.

2. Kernel Patches

   - Apply the kernel patch located in the patches/ directory of the driver source:

     FreeBSD 14.x:
      # patch -p1 < sriov_kernel_support_fbsd14.x.patch (or git am <patch>)

     FreeBSD 13.x:
      # patch -p1 < sriov_kernel_support_fbsd13.x.patch (or git am <patch>)

3. Build & Install Patched Kernel

   # make buildkernel KERNCONF=GENERIC -j<core_count>
   # make installkernel KERNCONF=GENERIC -j<core_count>

4. Reboot into the patched kernel.

5. Driver Build Path (if kernel source is not in /usr/src)

   - Edit driver Makefile:
     KERNBUILDDIR=/usr/obj/<installed_kernel_path>/amd64.amd64/sys/GENERIC/

6. Install iovctl Utility

   # pkg install iovctl

7. The PF interface must be brought up before creating any VFs on it.

   # ifconfig <intf> up


Creating VFs Using iovctl:
--------------------------

1. Without Passthrough

Example iovctl config file:
       PF {
	  device : "bnxt0";       # PF name
	  num_vfs : 1;            # Number of VFs
       }

       DEFAULT {
	   passthrough : false;    # Disable passthrough
       }

2. With Passthrough

Example iovctl config file:
       PF {
	  device : "bnxt0";       # PF name
	  num_vfs : 1;            # Number of VFs
       }

       DEFAULT {
	   passthrough : true;  # Enable passthrough
       }

3. Create VFs:

       # iovctl -C -f /path/to/iovctl.conf

4. Verify VFs:
   - Without Passthrough

       # pciconf -lv | grep bnxt

   - With Passthrough

       # pciconf -lv | grep -i ppt

You should see newly created VFs with device IDs corresponding to VF (e.g., 0x1819 for Thor2 and 0x180x for Thor)


Deleting VFs Using iovctl:
--------------------------

    1. Use the same iovctl.conf file that was used to create the VFs.

       # iovctl -D -f /path/to/iovctl.conf


Notes:

1. passthrough = true, VF visible for passthrough (driver will not bind/load)
   passthrough = false, VF network interface available in host (driver will load).
2. VF limit per PF depends on hardware/firmware.
3. Broadcom devices place VFs on a different PCI bus from PF.
4. Apply config before driver load or reload driver after config.
5. Refer to man iovctl and man bhyve for more details.
6. VF MAC address can be assigned during VF creation using the iovctl schema
   (mac-addr option).
7. VF trust and spoof check (mac-anti-spoof) can be configured either:
   - during VF creation via iovctl schema, or
   - at runtime using sysctl:
         dev.bnxt.<pf>.vf<N>.trusted
         dev.bnxt.<pf>.vf<N>.spoofchk
8. When trust is disabled (trusted = 0), the VF is not allowed to change
   its MAC address. Any attempt to change the VF MAC may fail with errors such as:
         HWRM_CFA_L2_SET_RX_MASK command returned RESOURCE_ALLOC_ERROR
9. VF MAC address can be changed by the VF only when trust is enabled
   (trusted = 1).
10. When VF is trusted, spoof check restrictions are bypassed and traffic
    is allowed irrespective of the spoof check setting.

CQE mode:
=========
When CQE mode is enabled then the hardware resets the notification
timer each time a CQE entry arrives. This causes the host notification
to be delayed until there is a space in the arrival time between CQEs
that is greater than or equal to the timer expiration value.

When CQE mode is disabled then the hardware starts the notification
timer when a CQE arrives. The timer is not reset until it expires. When
it expires, the notification is sent to the host. This means that
the host will be notified by the time the timer expires.


- On Thor2 cards, users can enable/disable this CQE mode during driver
  load using the below sysctl option. This CQE mode is enabled by
  default. In case if the user wants to modify this mode at runtime then
  the user needs to perform an ifconfig down & up operation manually.

  dev.bnxt.X.rx_coalesce_mode

- On Thor cards, this CQE mode will be disabled. In future if firmware
  supports this mode then it can be enabled/disabled.

- On Wh+ cards, this CQE mode is enabled by default if it is enabled in
  firmware. So, user can enable/disable this feature at runtime using
  below sysctl option,
  dev.bnxt.X.rx_coalesce_mode

=======
enable_agg_rings sysctl option:
===============================
By default enable_agg_rings sysctl option is set to one, which means that
aggregator rings will be enabled only if MTU > 4056 or TPA is enabled.

Users can completely disable the aggregator rings by setting this sysctl
option to zero.

Note that this sysctl option needs to be configured only
before loading the driver either through the "kenv -qv" option or through
loader.conf file. Configuring this sysctl option after driver load doesn't
change the aggregator rings allocation behaviour.

When the user completely disables the aggregator rings by setting this
sysctl option to zero then the user can set MTU to maximum 4056.

dev.bnxt.X.enable_agg_rings

Link Down Reason support:
=========================

The bnxt driver includes support for reporting link-down reasons through the
SIOCGIFDOWNREASON ioctl, allowing ifconfig to display a descriptive reason when
the link is down.

This functionality depends on additional iflib support (IFDI_GET_DOWNREASON), which
is not present in stock FreeBSD kernels. The required iflib changes are provided as
a patch in the driver source tree:

patches/iflib_linkdownreason_support.patch

To avoid impacting normal driver builds, this feature is disabled by default and is
guarded by the IFLIB_LINK_DOWNREASON_SUPPORT build flag in the driver Makefile.

The link-down reason support is enabled only when:
- the iflib patch is applied and the kernel is rebuilt, and
- the IFLIB_LINK_DOWNREASON_SUPPORT flag is enabled in the driver Makefile before
  building the driver.

Known Issues:
=============
1. TCP / UDP performance may vary between 30% to 95% depending on the link speed and NIC.
2. There is a limit on Max ping packetsize that can work for various
   MTU's and this issue is seen with other vendor NICs as well.
   ========================================
	MTU	Max ping packetsize that
		works (-s option of ping)
   ========================================
	1500		25152
	2000		33584
	3000		50584
   ========================================
3. The current driver supports only MSI-x interrupts but not legacy interrupts (INT-x / INT-a).
   Disabling MSI-X may cause unpredictable behavior, including crash.
   #kenv -qv bnxt.0.iflib.disable_msix=1
4. The current driver doesn't support suspend/resume operations in FreeBSD,
   support will be added in future releases. Trying those operations may cause
   unpredictable behavior, including crashes.
5. Operations like 'driver unload' and 'vlan destroy' may be blocked when there is UDP Rx flood.
   The workaround is to stop the UDP flood so that those operations will continue.
6. It may be necessary to perform an ifconfig down/up on the interface after changing media
   speed with the 12.2-RELEASE kernel. This issue is fixed in the latest 12.2-STABLE kernel.
7. Asymmetric Tx and Rx queues configuration is not supported. configuring may result in
   undefined behavior.
8. devctl bus reset is not supported. post this command driver functionalities
   may not work until driver gets reloaded.
   # devctl reset bnxt<x> (where x=0,1..N)
9. devctl device reset is supported.
   # devctl reset -d bnxt<x> (where x=0,1..N)
10. On BCM574xx chips, when interface (bnxt0, bnxt1..) is in admin down state, 'ifconfig bnxt<x>' may
    not reflect correct link speed (may show unknown) upon link speed change operation from peer.
    To get correct link information before doing link speed change, please make sure to bring up
    the interface through command- 'ifconfig bnxt<x> up' (where x=0,1..N)
11. Driver compilation on FreeBSD12.4 OS throws a lot of set but not used warnings for variable 'e'
    And that is the kernel issue. It will be probably fixed in the upcoming freebsd OS releases.

    Below is the warning snippet:

    /usr/src/sys/dev/pci/pcivar.h:420:1: warning: variable 'e' set but not used [-Wunused-but-set-variable]
    PCIB_ACCESSOR(bus,              BUS,            uint32_t)
12. Force link speed change using ifconfig will be reflected within the angle bracket of media section in
    ifconfig only after link comes up. Below is how it looks in media section of ifconfig output:
    media: Ethernet autoselect (10GBase-CR1 <full-duplex>)
13. BCM574xx, BCM575xx and BCM576xx chips support LRO(Large Receive Offload) but it's disabled by default.
    Hardware LRO can be enabled through kenv/loader.conf during driver load.
    dev.bnxt.<x>.hw_lro.enable	(where x=0,1..N).
    Likewise, LRO can also be enabled with ifconfig bnxt<x> lro and
    disabled with ifconfig bnxt<x> -lro (where x=0,1..N).
14. ifconfig option to modify LRO settings is not effective.
15. The QSFP28 module information appears as unknown in FreeBSD 13.2, and the
    verbose information displayed by the ifconfig command may contain incorrect
    data for the module. When running the ifconfig command with the -v flag and
    specifying the interface, the output will show that the module is plugged in,
    but the type is identified as "QSFP28 Unknown".
16. The current driver does not support the 802.1ad VLAN protocol.
17. VLAN IPv6 will not support jumbo packets if jumbo MTU is inherited from base interface during VLAN creation.
    If user configures the jumbo MTU on VLAN interface after VLAN creation, then VLAN IPv6 jumbo works as expected.
    This is the OS limitation.
18. There is known disparity between Linux and FreeBSD ring counters exposed through sysfs/sysctl. This will be
    addressed on best-effort basis in future FreeBSD driver releases.
19. Ongoing multicast traffic continues without loss, when the multicast group is dynamically removed from the server.
20. In FreeBSD, driver does not set default DCBX parameters start of the day. User has to do desired DCBX programming
    through sysctl hooks. Also, it's not guaranteed DCBX settings done through systcl will be restored across system reboots/
    driver reloads.
21. FreeBSD currently lacks the macros to support 100G per lane for PAM4 112 types.
    For example, it should include support for 400G speeds over copper and fiber, such as IFM_400G_CP4/CR4 and IFM_400G_SR4.
    As a result, only PAM4 56 will be supported for now.
22. The Direct Reach (DR) fiber PHY type is not supported in the firmware. We used 400GBase-DR4 to represent 400G SR4 (Short Reach fiber)
    speed due to the lack of a 400G SR4 API in FreeBSD. This was the closest possible equivalent PHY type available for SR fiber.
23. Firmware host logging for SRT and CRT logs is not supported on THOR (BCM5750x) controllers. It is only supported on THOR2 (BCM5760x) controllers.
24. The driver fails to load on systems with a single CPU due to a limitation in the iflib layer. The error message "Cannot use iflib with only 1 MSI-X interrupt"
    will appear in the dmesg logs upon driver load. At least two CPUs (CPU > 1) are required for the driver to function. This limitation applies to both PF and
    VF interfaces.
27. Do not use devctl reset on VFs inside guests. devctl reset unloads and reloads the VF driver too quickly, before the PF completes
    reprovisioning, which leads to HWRM failures and VF attach failure. Instead, use devctl detach followed by devctl attach
    # sudo devctl detach <vf pci device>
    # sleep 3 seconds
    # sudo devctl attach <vf pci device>
28. VXLAN and GRE tunneling offloads are not supported.
29. Driver unload operation may take some time (in minutes) to complete if it is performed while kTLS TX traffic is active.
30. With 4093 or more VLANs, VLAN creation and deletion can take significantly longer on FreeBSD, and coredump collection via the bnxt_mgmt ioctl may
    be slow or appear stuck. These delays are due to OS kernel scheduling and resource contention at high VLAN count, not the driver.
