PTF does not support wildcard byte akin to STF
hesingh opened this issue · comments
STF supports wildcard byte in expected packet using * so that the byte can be ignored. PTF does not. I think, this code change to PTF should allow wildcard byte akin to STF.
diff --git a/src/ptf/mask.py b/src/ptf/mask.py
index 3b45d9a..3987e96 100644
--- a/src/ptf/mask.py
+++ b/src/ptf/mask.py
@@ -61,6 +61,8 @@ class Mask:
return False
exp_pkt = str(self.exp_pkt)
for i in xrange(self.size):
+ if exp_pkt[i] == "*":
+ continue;
if (ord(exp_pkt[i]) & self.mask[i]) != (ord(pkt[i]) & self.mask[i]):
return False
return True
I see PTF supports a mask to ignore certain bytes in expected packet. PTF has set_do_not_care() for this purpose. However, just using a * is very convenient vs. dealing with packet header type and field with mask.
In the README.md for PTF, I see this note regarding how to run the switch for use with PTF testing. Please tell me this is old and one should just use p4c's simple_switch?
First, you need to create the required veths:
cd $P4FACTORY/tools/
sudo ./veth_setup.sh
The next step is to compile the target switch and to run it:
cd $P4FACTORY/targets/switch/
make bm-switchsai
sudo ./behavioral-model
'*' is ASCII character '\x2a'. I can build a legitimate packet which contains '\x2a', I don't see why it should be interpreted as a wildcard.
antonin@ubuntu:~$ python
Python 2.7.12 (default, Nov 12 2018, 14:36:49)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print('\x2a')
*
>>>
A basic example: I can set the IPv4 TTL to 42 (0x2a). Maybe I want to verify that the TTL in the expected packet is actually 42, and not use a wildcard:
antonin@ubuntu:~$ scapy
INFO: Can't import python gnuplot wrapper . Won't be able to plot.
INFO: Can't import PyX. Won't be able to use psdump() or pdfdump().
INFO: Can't import python Crypto lib. Won't be able to decrypt WEP.
INFO: Can't import python Crypto lib. Disabled certificate manipulation tools
Welcome to Scapy (2.2.0)
>>> p = Ether()/IP(ttl=42)
>>> p.show2()
###[ Ethernet ]###
dst= ff:ff:ff:ff:ff:ff
src= 00:00:00:00:00:00
type= 0x800
###[ IP ]###
version= 4L
ihl= 5L
tos= 0x0
len= 20
id= 1
flags=
frag= 0L
ttl= 42
proto= hopopt
chksum= 0x92e7
src= 127.0.0.1
dst= 127.0.0.1
\options\
>>> str(p)[22]
'*'
>>> str(p)[22] == "*"
True
>>>
Thanks for getting back. STF is also written in Python and supports the * wildcard byte via this Python code.
https://github.com/p4lang/p4c/blob/master/backends/bmv2/bmv2stf.py#L698
I wonder if STF uses *, why can't PTF? Using mask for a specific header and a field offset in a header in PTF is more cumbersome.
I can close this issue for wildcard since I have already said PTF supports set_do_not_care for a field in the header. I can use that since Antonin brings up a valid point.
However, I would like to also know if the simple_switch from p4c can be used to test PTF instead of the proposal in the PTF README - thanks.
PTF can be used with anything. It's just a test framework which includes a library to send / receive / match packets. You usually need to combine it with the appropriate ptf.base_tests.BaseTest subclass for your control-plane interface of choice. For P4Runtime, these 2 implementations are available (they are basically the same):
I believe that the goals of both STF and PTF are very similar: test a P4-programmed data plane device, with optional establishing of table entries and/or other control plane setups before sending the data packets in, and check the output packets.
Differences:
-
STF has no programmability. Sent-to-the-device and expected packets are specified explicitly in hex. There are no data structures, loops, if statements, etc. The current p4c STF version supports adding some table entries, and creating multicast groups, but almost no other control plane functionality (another of version of STF checked into p4c, but not yet used to run automated tests, improves on that).
-
PTF is all about programmability. Not only can you send in packets that are the same every time you run a test, you could pseudo-randomly generate packet contents if you want. You can set up simulated networks of devices if you want. You can create fancy data structures to hold table entries you want to add, use those to help you generate sent-in and/or expected packets, do arbitrary arithmetic or other functions on the packets sent out by a device when checking whether they are good or bad, etc. It can do everything STF can (although with more lines of code required for simple things that STF is good for), and way more. The sky and your imagination is the limit.
Just because STF is implemented in Python, doesn't mean you get the power of Python when writing STF tests. Sure, you could add in-line Python code to STF tests if you made extensive changes to the STF test infrastructure, but why, when PTF exists?
STF has great documentation for its small functionality, but PTF has zero documentation. This is a sticking point for developers starting with PTF. Good summary (@jafingerhut) and thanks @antoninbas for the clarification.
However, seeing some examples of working PTF programs, I could learn PTF in a day to get my tests working. Right now, I am buried, but when I get the time, if needed, I will add documentation to PTF for how to add/delete table entry, use STF hex packets in PTF with Scapy code etc.
If you want some documentation and examples for using Scapy, including converting to/from Python strings of hex digits, you are welcome to link to this, and/or copy any parts of it that you find useful: https://github.com/jafingerhut/p4-guide/blob/master/README-scapy.md
@jafingerhut Yes, I have seen your website and using your scapy to STF bytes convertor Python code - thanks!
Also, the large Python code that PTF uses is Python's unittest and thus if unittest is added STF would be beefed up for Python. However, since PTF is available, why bother. I like STF as is because it is great for quickly testing new p4 code with packets.