Proxmox VE에서 방 한구석에 쳐박혀 있는 Mellanox ConnectX-3 VPI Dual QSFP+ Port Adapter Card가 과연 잘 동작할까? 그래도 InfiniBand라는 건 한번 써보긴 해야하는거 아닌가? 중고가도 싼데… 이게 시작이었습니다.
일단 Proxmox VE 환경에서 펌웨어를 업데이트하고, 일단 이더넷 모드로 맞추는 것 부터 시작해야 합니다. 사용 경험이 거의 없어서 정확한 절차는 찾는게 일이 었던 것 같습니다.
항상 그렇듯 찾고 나니 쉬운 ^^
Proxmox VE에 SSH로 접속해서 진행한 내용들 정리해 보겠습니다.
파일들 준비
ConnectX-3 카드가 40G라고 써져 있어도 유형이 많고, OEM 버전이 많아서 모델에 맞는 펌웨어를 준비하는게 중요합니다.
첫번째 명령이 펌웨어 받는 것입니다. 이건 카드에 적혀 있는 모델명이나 파트넘버를 이용해서 검색하면 쉽게 구해질겁니다. 꼭 모델에 맞는거 사용하세요.
Shell
x
wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX353A-FCB_A2-A5-FlexBoot-3.4.752.bin.zipwget https://www.mellanox.com/downloads/firmware/mlxup/4.22.1/SFX/linux_x64/mlxupwget https://www.mellanox.com/downloads/MFT/mft-4.22.1-11-x86_64-deb.tgzwget https://github.com/Mellanox/mstflint/releases/download/v4.22.0-1/mstflint-4.22.0-1.tar.gz의존성 패키지 설치와 압축 풀고 펌웨어 업데이트
Shell
apt-get install gcc make dkms pve-headerstar -xvzf mft-4.22.1-11-x86_64-deb.tgzcd mft-4.22.1-11-x86_64-deb./install.shcd ..chmod +x mlxup./mlxupmlxup 실행 화면
Shell
root@pve-sky01:~/m# ./mlxupQuerying Mellanox devices firmware ...Device #1:---------- Device Type: ConnectX3 Part Number: MCX354A-FCB_A2-A4 Description: ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6 PSID: MT_1090120019 PCI Device Name: /dev/mst/mt4099_pci_cr0 Port1 GUID: f45214030033c321 Port2 GUID: f45214030033c322 Versions: Current Available FW 2.11.0500 2.42.5000 PXE 3.4.0146 3.4.0752 Status: Update required---------Found 1 device(s) requiring firmware update...Perform FW update? [y/N]:Restart needed for updates to take effect.Log File: /tmp/mlxup_workdir/mlxup-20250204_193956_5677.logroot@pve-sky01:~/m# 펌웨어 업데이트하고 리부팅해야 한다고 나옵니다.
리부팅하고 나서 mst 명령을 이용해서 하드웨어를 확인합니다.
Shell
mst startmst statusmst 명령 실행 화면
Shell
root@pve-sky01:~# mst startStarting MST (Mellanox Software Tools) driver setLoading MST PCI module - SuccessLoading MST PCI configuration module - SuccessCreate devicesroot@pve-sky01:~# mst statusMST modules:------------ MST PCI module loaded MST PCI configuration module loadedMST devices:------------/dev/mst/mt4099_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:af:00.0 addr.reg=88 data.reg=92 cr_bar.gw_offset=-1 Chip revision is: 01/dev/mst/mt4099_pci_cr0 - PCI direct access. domain:bus:dev.fn=0000:af:00.0 bar=0xee500000 size=0x100000 Chip revision is: 01root@pve-sky01:~# mst 명령어로 확인한 디바이스 이름으로 mlxconfig 명령으로 구성 정보를 조회
Shell
mlxconfig -d /dev/mst/mt4099_pciconf0 qmlxconfig 명령 실행 화면
Shell
root@pve-sky01:~# mlxconfig -d /dev/mst/mt4099_pciconf0 qDevice #1:----------Device type: ConnectX3Device: /dev/mst/mt4099_pciconf0Configurations: Next Boot SRIOV_EN False(0) NUM_OF_VFS 8 LINK_TYPE_P1 VPI(3) LINK_TYPE_P2 VPI(3) LOG_BAR_SIZE 3 BOOT_PKEY_P1 0 BOOT_PKEY_P2 0 BOOT_OPTION_ROM_EN_P1 True(1) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 True(1) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0) CQ_TIMESTAMP True(1)root@pve-sky01:~# SRIOV는 비활성화되어 있고, LINK_TYPE_P1, LINK_TYPE_P2가 VPI 모드로 설정되어 있는 것이 보입니다.
mlxconfig 명령을 이용해서 2개의 포트를 이더넷 모드로 설정합니다.
( 나중에 보니 한 줄에 해도 되더라고요. )
Shell
mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=2mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P2=2mlxconfig 실행 화면
Shell
root@pve-sky01:~# mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P1=2Device #1:----------Device type: ConnectX3Device: /dev/mst/mt4099_pciconf0Configurations: Next Boot New LINK_TYPE_P1 VPI(3) ETH(2) Apply new Configuration? (y/n) [n] : yApplying... Done!-I- Please reboot machine to load new configurations.root@pve-sky01:~# mlxconfig -d /dev/mst/mt4099_pciconf0 set LINK_TYPE_P2=2Device #1:----------Device type: ConnectX3Device: /dev/mst/mt4099_pciconf0Configurations: Next Boot New LINK_TYPE_P2 VPI(3) ETH(2) Apply new Configuration? (y/n) [n] : yApplying... Done!-I- Please reboot machine to load new configurations.root@pve-sky01:~#리부팅하라고 합니다. 전 나오는 대로 잘 따라하는 편입니다 ^^
리부팅하고 나서 다시 mst start를 하고 구성 정보를 조회합니다.
Shell
mst startmlxconfig -d /dev/mst/mt4099_pciconf0 qmst와 mlxconfig 실행 화면
Shell
root@pve-sky01:~# mst startStarting MST (Mellanox Software Tools) driver setLoading MST PCI module - SuccessLoading MST PCI configuration module - SuccessCreate devicesroot@pve-sky01:~# mlxconfig -d /dev/mst/mt4099_pciconf0 qDevice #1:----------Device type: ConnectX3Device: /dev/mst/mt4099_pciconf0Configurations: Next Boot SRIOV_EN False(0) NUM_OF_VFS 8 LINK_TYPE_P1 ETH(2) LINK_TYPE_P2 ETH(2) LOG_BAR_SIZE 3 BOOT_PKEY_P1 0 BOOT_PKEY_P2 0 BOOT_OPTION_ROM_EN_P1 True(1) BOOT_VLAN_EN_P1 False(0) BOOT_RETRY_CNT_P1 0 LEGACY_BOOT_PROTOCOL_P1 PXE(1) BOOT_VLAN_P1 1 BOOT_OPTION_ROM_EN_P2 True(1) BOOT_VLAN_EN_P2 False(0) BOOT_RETRY_CNT_P2 0 LEGACY_BOOT_PROTOCOL_P2 PXE(1) BOOT_VLAN_P2 1 IP_VER_P1 IPv4(0) IP_VER_P2 IPv4(0) CQ_TIMESTAMP True(1)root@pve-sky01:~#(당연한거겠지만…) 잘 변경 됐습니다.
lshw, ethtool 과 같은 명령으로 이더넷 네트워크 디바이스로 잘 인식됐는지 확인합니다.
( 디바이스 이름은 시스템 마다 다를 수 있습니다 )
Shell
lshw -c networkethtool enp175s0lshw, ethtool 실행 화면
Shell
root@pve-sky01:~# lshw -c network *-network:0 DISABLED description: Ethernet interface product: Ethernet Connection X722 for 10GBASE-T vendor: Intel Corporation physical id: 0 bus info: pci@0000:1a:00.0 logical name: eno1np0 version: 09 serial: 6c:92:bf:1a:c6:51 size: 1Gbit/s capacity: 10Gbit/s width: 64 bits clock: 33MHz capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical tp 1000bt-fd 10000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=i40e driverversion=6.8.12-8-pve duplex=full firmware=4.10 0x80001de3 1.2585.0 latency=0 link=no multicast=yes port=twisted pair speed=1Gbit/s resources: iomemory:381f0-381ef iomemory:381f0-381ef irq:29 memory:381ffe000000-381ffeffffff memory:381fff808000-381fff80ffff memory:aad80000-aadfffff memory:381fff400000-381fff7fffff memory:381fff890000-381fff90ffff *-network:1 DISABLED description: Ethernet interface product: Ethernet Connection X722 for 10GBASE-T vendor: Intel Corporation physical id: 0.1 bus info: pci@0000:1a:00.1 logical name: eno2np1 version: 09 serial: 6c:92:bf:1a:c6:52 size: 1Gbit/s capacity: 10Gbit/s width: 64 bits clock: 33MHz capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical tp 1000bt-fd 10000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=i40e driverversion=6.8.12-8-pve duplex=full firmware=4.10 0x80001de3 1.2585.0 latency=0 link=no multicast=yes port=twisted pair speed=1Gbit/s resources: iomemory:381f0-381ef iomemory:381f0-381ef irq:29 memory:381ffd000000-381ffdffffff memory:381fff800000-381fff807fff memory:aad00000-aad7ffff memory:381fff000000-381fff3fffff memory:381fff810000-381fff88ffff *-network description: Ethernet interface product: 82574L Gigabit Network Connection vendor: Intel Corporation physical id: 0 bus info: pci@0000:5e:00.0 logical name: enp94s0 version: 00 serial: 68:05:ca:0d:83:64 size: 1Gbit/s capacity: 1Gbit/s width: 32 bits clock: 33MHz capabilities: pm msi pciexpress msix bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=6.8.12-8-pve duplex=full firmware=2.1-0 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s resources: irq:42 memory:c5ec0000-c5edffff memory:c5e00000-c5e7ffff ioport:9000(size=32) memory:c5ee0000-c5ee3fff memory:c5e80000-c5ebffff *-network DISABLED description: Ethernet interface product: MT27500 Family [ConnectX-3] vendor: Mellanox Technologies physical id: 0 bus info: pci@0000:af:00.0 logical name: enp175s0 version: 00 serial: f4:52:14:33:c3:21 capacity: 56Gbit/s width: 64 bits clock: 33MHz capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=4.0-0 firmware=2.42.5000 latency=0 link=no multicast=yes port=fibre resources: iomemory:386f0-386ef irq:46 memory:ee500000-ee5fffff memory:386fff800000-386fffffffff memory:ee400000-ee4fffff *-network DISABLED description: Ethernet interface physical id: 5 bus info: pci@0000:af:00.0 logical name: enp175s0d1 serial: f4:52:14:33:c3:22 capacity: 56Gbit/s capabilities: ethernet physical fibre 10000bt-fd 40000bt-fd 56000bt-fd autonegotiation configuration: autonegotiation=off broadcast=yes driver=mlx4_en driverversion=4.0-0 firmware=2.42.5000 link=no multicast=yes port=fibreroot@pve-sky01:~# ethtool enp175s0Settings for enp175s0: Supported ports: [ FIBRE ] Supported link modes: 10000baseKX4/Full 40000baseCR4/Full 40000baseSR4/Full 56000baseCR4/Full 56000baseSR4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10000baseKX4/Full 40000baseCR4/Full 40000baseSR4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: Unknown! Duplex: Unknown! (255) Auto-negotiation: off Port: FIBRE PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000014 (20) link ifdown Link detected: noroot@pve-sky01:~# ethtool enp175s0d1Settings for enp175s0d1: Supported ports: [ FIBRE ] Supported link modes: 10000baseKX4/Full 40000baseCR4/Full 40000baseSR4/Full 56000baseCR4/Full 56000baseSR4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10000baseKX4/Full 40000baseCR4/Full 40000baseSR4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: Unknown! Duplex: Unknown! (255) Auto-negotiation: off Port: FIBRE PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000014 (20) link ifdown Link detected: noroot@pve-sky01:~#Speed가 Unknown으로 나왔습니다. 이건 나중에 2호기 설정 끝나면 40000Mb/s 나옵니다. 56000 도 지원은 된다고 하나, 직결해서 그런지 40000 까지만 되는 것 같습니다.
( 이건 나중에 스위치 연결해서 다시 확인하는거로~)
2호기는 HP OEM 제품이라 앞에서 받은 펌웨어를 사용 못합니다. 파트 넘버로 조회하니 금방 찾아졌습니다.
업데이트 과정은 거의 같아서 생략…
숫자놀이 시간이 돌아왔습니다
이더넷 디바이스에 ip 주소 주고, MTU는 9000으로 설정한 상태에서 진행했습니다.
Shell
root@pve-sky01:~# iperf3 -c 10.0.175.212Connecting to host 10.0.175.212, port 5201[ 5] local 10.0.175.211 port 37910 connected to 10.0.175.212 port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-1.00 sec 2.54 GBytes 21.8 Gbits/sec 0 821 KBytes[ 5] 1.00-2.00 sec 2.65 GBytes 22.8 Gbits/sec 0 821 KBytes[ 5] 2.00-3.00 sec 1.93 GBytes 16.6 Gbits/sec 0 1.59 MBytes[ 5] 3.00-4.00 sec 1.88 GBytes 16.2 Gbits/sec 0 1.59 MBytes[ 5] 4.00-5.00 sec 2.12 GBytes 18.2 Gbits/sec 0 2.28 MBytes[ 5] 5.00-6.00 sec 2.67 GBytes 22.9 Gbits/sec 0 2.28 MBytes[ 5] 6.00-7.00 sec 2.08 GBytes 17.8 Gbits/sec 0 2.51 MBytes[ 5] 7.00-8.00 sec 1.88 GBytes 16.1 Gbits/sec 0 2.64 MBytes[ 5] 8.00-9.00 sec 1.88 GBytes 16.2 Gbits/sec 0 2.64 MBytes[ 5] 9.00-10.00 sec 1.89 GBytes 16.2 Gbits/sec 0 2.64 MBytes- - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec 0 sender[ 5] 0.00-10.00 sec 21.5 GBytes 18.5 Gbits/sec receiveriperf Done.root@pve-sky01:~#검색하니 튜닝 정보 있는 곳이 보입니다.
- https://fasterdata.es.net/host-tuning/linux/test-measurement-host-tuning/
아래 값들을 /etc/sysctl.conf 에 추가하고, sysctl -p
Shell
# increase TCP max buffer size setable using setsockopt() # allow testing with 256MB buffers net.core.rmem_max = 268435456 net.core.wmem_max = 268435456 # increase Linux autotuning TCP buffer limits # min, default, and max number of bytes to use # allow auto-tuning up to 128MB buffers net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728 # don't cache ssthresh from previous connection net.ipv4.tcp_no_metrics_save = 1 # If you are using [Jumbo Frames](https://fasterdata.es.net/network-tuning/mtu-issues/), also set this net.ipv4.tcp_mtu_probing = 1 # recommended to enable 'fair queueing' (fq or fq_codel) net.core.default_qdisc = fq실행 할 때마다 조금 들쑥 날쑥합니다. 2번 하면 1번은 나오는 최대값은 이정도 입니다.
Shell
root@pve-sky01:~# iperf3 -c 10.0.175.212Connecting to host 10.0.175.212, port 5201[ 5] local 10.0.175.211 port 50326 connected to 10.0.175.212 port 5201[ ID] Interval Transfer Bitrate Retr Cwnd[ 5] 0.00-1.00 sec 2.84 GBytes 24.4 Gbits/sec 0 3.12 MBytes[ 5] 1.00-2.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 2.00-3.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 3.00-4.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 4.00-5.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 5.00-6.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 6.00-7.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 7.00-8.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 8.00-9.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes[ 5] 9.00-10.00 sec 2.91 GBytes 25.0 Gbits/sec 0 3.12 MBytes- - - - - - - - - - - - - - - - - - - - - - - - -[ ID] Interval Transfer Bitrate Retr[ 5] 0.00-10.00 sec 29.0 GBytes 24.9 Gbits/sec 0 sender[ 5] 0.00-10.00 sec 29.0 GBytes 24.9 Gbits/sec receiveriperf Done.root@pve-sky01:~#일단은 iperf3를 걍 실행시켰을 때도 20G 넘게 나오는 것에 만족합니다.
-P 4 또는 -P 8 옵션을 추가해서 멀티스레드로 하면 전체 전송 속도는 도리어 떨어졌습니다.
숫자 놀이는 좀 더 연구가 필요해 보입니다.
일단 판깔 준비는 했으니, 천천히 가지고 놀아보겠습니다 ㅋ
