Error while configuring Mirroring
mohamedibrahimabdoun opened this issue · comments
Hi,
I'm getting below error upon enabling mirroring , i keep getting this error at step 5 of" Changing Mirroring State "[iplex add] in (https://github.com/att/vfd/wiki/Debugging-Tricks):
iplex add 86d13cde-4a29-41e1-8956-4b62424431dd
{ "state": "ERROR", "msg": "unable to add vf: /var/lib/vfd/config/86d13cde-4a29-41e1-8956-4b62424431dd.json: mirror target is out of range or is the same as this VF (28): 29" }
- Also i tried to sudo iplex mirror ,but I can not see traffic in target VM [50.50.50.52_Mirroring_Guest]
problematic VF:
cat /var/lib/vfd/config/86d13cde-4a29-41e1-8956-4b62424431dd.json
{"strip_stag": true, "macs": [], "name": "MO_TEST_VM53/86d13cde-4a29-41e1-8956-4b62424431dd", "link_status": "auto", "pciid": "0000:83:00.0", "allow_bcast": true, "vfid": 28, "mac_anti_spoof": true, "vlan_anti_spoof": true, "allow_mcast": true, "insert_stag": false, "vlans": [1000], "allow_un_ucast": true,"mirror": { "target": 29, "direction": "all" }}
Target VF :
cat /var/lib/vfd/config/8e442a16-4e93-4162-8dac-83a8853a0cc5.json
{"strip_stag": true, "macs": [], "name": "MO_TEST_VM52_mirror/8e442a16-4e93-4162-8dac-83a8853a0cc5", "link_status": "auto", "pciid": "0000:83:00.0", "allow_bcast": true, "vfid": 29, "mac_anti_spoof": true, "vlan_anti_spoof": true, "allow_mcast": true, "insert_stag": false, "vlans": [1000], "allow_un_ucast": false}
===================Logs & Info========================
50.50.50.52_Mirroring_Guest:
iplex show all|grep 0000:83:17.2
vf 29 0000:83:17.2 UP 0 0 0 0 107 32846 0 0
#:~# nova interface-list c2fc40cd-d529-43aa-bebd-b8cc691c82f0
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| ACTIVE | 8e442a16-4e93-4162-8dac-83a8853a0cc5 | 4eb7e90b-b026-4857-b1f5-098ce144627e | 50.50.50.52 | 02:8e:44:2a:16:4e |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
cat /var/lib/vfd/config/8e442a16-4e93-4162-8dac-83a8853a0cc5.json
{"strip_stag": true, "macs": [], "name": "MO_TEST_VM52_mirror/8e442a16-4e93-4162-8dac-83a8853a0cc5", "link_status": "auto", "pciid": "0000:83:00.0", "allow_bcast": true, "vfid": 29, "mac_anti_spoof": true, "vlan_anti_spoof": true, "allow_mcast": true, "insert_stag": false, "vlans": [1000], "allow_un_ucast": false}
50.50.50.52_Mirroring_Guest:
iplex show all|grep 0000:83:17.2
vf 29 0000:83:17.2 UP 0 0 0 0 107 32846 0 0
#:~# nova interface-list c2fc40cd-d529-43aa-bebd-b8cc691c82f0
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| Port State | Port ID | Net ID | IP addresses | MAC Addr |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
| ACTIVE | 8e442a16-4e93-4162-8dac-83a8853a0cc5 | 4eb7e90b-b026-4857-b1f5-098ce144627e | 50.50.50.52 | 02:8e:44:2a:16:4e |
+------------+--------------------------------------+--------------------------------------+--------------+-------------------+
cat /var/lib/vfd/config/8e442a16-4e93-4162-8dac-83a8853a0cc5.json
{"strip_stag": true, "macs": [], "name": "MO_TEST_VM52_mirror/8e442a16-4e93-4162-8dac-83a8853a0cc5", "link_status": "auto", "pciid": "0000:83:00.0", "allow_bcast": true, "vfid": 29, "mac_anti_spoof": true, "vlan_anti_spoof": true, "allow_mcast": true, "insert_stag": false, "vlans": [1000], "allow_un_ucast": false}
I'll try to set up something to reproduce this. Can you provide the output from the following command:
vfd -?
That will give me a commit id to build from so I'm looking at things from the same perspective.
Also, how many VFs do you have configured on the PF? 30, or more? if you have 30, this could be an off by one bug.
I'm not able to reproduce this with the current code. I can add a mirror either by adding a VF config with the mirror definition in the json, or by using the 'iplex mirror' command.
I do note that there were two bug fixes related to setting up mirrors, one of which directly affected the check which generates the message you're seeing. This change addressed issue 242 and was committed on Jan 31, 2018. If you're using code that is older than this, there is a chance you've encountered the bug that was fixed. If you're using current code, then we'll need to look further. At this point I'll leave this open as review/discuss.
Thanks for you response ,it seems like an old version issue ,but not sure
please check the output of vfd -? command
_VFd v2 +++2105d85e9b7bd83fcba457c33f48aa8a0617f372-notag build: Nov 2 2017 11:36:53
iplex output:
iplex show all
{ "state": "OK", "msg": "
PF/VF ID PCIID Link Speed Duplex RX pkts RX bytes RX errors RX dropped TX pkts TX bytes TX errors Spoofed
pf 2 0000:83:00.0 UP 10000 1 43 6710 0 0 24 2040 0 0
vf 27 0000:83:16.6 UP 1 74 0 0 8 680 0 0
vf 28 0000:83:17.0 UP 15 1266 0 0 8 680 0 0
vf 29 0000:83:17.2 UP 15 1266 0 0 8 680 0 0
pf 3 0000:83:00.1 UP 10000 1 0 0 0 0 0 0 0 0
pf 0 0000:03:00.0 UP 10000 1 79 22284 0 0 85 20074 0 82
vf 20 0000:03:15.0 UP 35 10022 0 0 29 7946 0 0
vf 21 0000:03:15.2 UP 20 6648 0 0 35 10022 0 0
vf 22 0000:03:15.4 UP 0 0 0 0 0 0 0 0
vf 23 0000:03:15.6 UP 0 0 0 0 0 0 0 0
vf 24 0000:03:16.0 UP 0 0 0 0 0 0 0 0
vf 25 0000:03:16.2 UP 0 0 0 0 0 0 0 0
vf 26 0000:03:16.4 UP 0 0 0 0 3 342 0 0
vf 27 0000:03:16.6 UP 0 0 0 0 5 498 0 0
vf 28 0000:03:17.0 UP 0 0 0 0 3 270 0 0
vf 29 0000:03:17.2 UP 0 0 0 0 5 498 0 0
vf 30 0000:03:17.4 UP 0 0 0 0 5 498 0 0
pf 1 0000:03:00.1 UP 10000 1 14 4788 0 0 97 10922 0 95
vf 22 0000:03:15.5 UP 0 0 0 0 0 0 0 0
vf 23 0000:03:15.7 UP 0 0 0 0 0 0 0 0
vf 24 0000:03:16.1 UP 0 0 0 0 0 0 0 0
vf 25 0000:03:16.3 UP 0 0 0 0 0 0 0 0
vf 26 0000:03:16.5 UP 0 0 0 0 20 2172 0 0
vf 27 0000:03:16.7 UP 0 0 0 0 20 2244 0 0
vf 28 0000:03:17.1 UP 0 0 0 0 20 2244 0 0
vf 29 0000:03:17.3 UP 0 0 0 0 16 1848 0 0
vf 30 0000:03:17.5 UP 0 0 0 0 21 2358 0 0
" }
Right now it does seem like it's the bug that was fixed in Jan 2018. There look to be 6 obvious commits since the code point of your build that directly reference mirroring. If possible, I'd suggest pulling the current master branch and seeing if that doesn't solve the problem. Be advised that the current code does require building with DPDK 18.02.
A possible work round would be to start the mirror target as a low numbered VF. I realise that might not always be possible/practical, but it might be worth a shot if you can do it. I will also reconfigure my environment to create 31 VFs to mimic yours. I ran my tests with VFs 30 and 31; I doubt it's related to less than 32 VFs configured, but that's easy for me to rule out at least with the current code, so on the off chance that it is, I'll have a go with that at some point today.
I pulled the code using the commit id you posted and with that old build I do see the problem you've encountered. I also tried the work round I suggested above, and that doesn't work, so don't bother. It isn't related to the number of VFs which are configured as I've reproduced it with 32 VFs.
I verified that the bug you are seeing was corrected with the 31 Jan 2018 commit (c5f3e0e) described in issue #242.
There is a work round, but it may not be possible depending on your environment and/or setup. The bug was checking the VF number of the target against the number of VFs that are under VFd's control when it should have been checking against the number of VFs allocated on the NIC. So, if the target VF number is less than the total number of VFs under VFd's control the mirror will be successful. To make it a bit easier, if it's possible to assign VF 1 to a guest, and make that the target then the mirror add will work. Like I said, this is a bit of a hack, but it could get you past the bug until you are able to upgrade or rebuild VFd.
I'll leave this open pending any final comments you may have, but at this point I think the right thing to do is move to current code.
@ScottDaniels thanks for your support , i will try to arrange for upgrading vfd code to Jan 31, 2018 commit ..appreciate you help.
Great. If you can, it would be wise just to pull the latest master as there are about 80 commits after the 31 Jan 2018 commit. We also have a google group that you are most welcome to join. The activity there is very low, but it's a good place to ask questions if you're stuck. Certainly, post issues here should you encounter any!
https://groups.google.com/forum/#!forum/vfd-users
I'm closing this issue.