conda env create -f environment.yml
conda activate warlock
conda install pytorch torchvision -c pytorch
pip install pyvmomi paramiko pyyamlThe primary purpose of a warlock tool is to discover and define optimal configurations or parameters tailored to specific execution environments for given tasks. These execution environments can vary, including elements such as virtualization platforms, container runtimes, and distinct functions like simulations designed to yield specific metrics. For instance, the tool might assess real-time latency for code executed within a Kubernetes environment or evaluate the packet-per-second rate achievable by a task. From an optimization perspective, the goal is to identify a value x that delivers the most favorable outcome for the specified task, enhancing performance and efficiency in the targeted environment, where x is some vector that captures a set of all parameters that the agent can mutate to run a simulation and collect a set of observations. For example, x might be the container set of values required to find an optimal TCP buffer and a TCP algorithm that can produce maximum throughput.
If use Spell caster lingo, Warlock is a spell-casting tool designed to combine a set of spells that mutate some environments. Consider IaaS as a world where Caas is an entity in a castle within that world, and worker nodes are minions. Each minion can perform some work and uses PODs. For example, a minion can move magic herbs from one spot to another. Minions can either walk (use a slow CNI) or teleport (use DPDK). The main play here is that the Warlock needs to learn a set of spells that will make the minions more efficient. Since the Warlock has the power to bend reality, it can mutate the entire world (IaaS).
Master spell.
Some definition.
A Spell Spec is a framework that encapsulates what is termed a "Master Spell." This Master Spell is essentially an aggregation of several spells which together create a comprehensive strategy.
These spells, when cast by a Spell Caster, are applied to various environments to achieve specific objectives, such as optimization tasks.
The objective is to learn a powerful Master Spell upon acting on environment producing maximizing desired behavior.
The framework is designed to be abstract, using terms like IaaS, CaaS, and pods metaphorically, rather than referring to actual infrastructure components.
The system categorizes spells into three primary types alongside additional, optional spells:
IaaS Spells: Aimed at transforming what is conceptually represented as IaaS. CaaS Spells: Focus on altering what is analogously understood as CaaS. Pods Spells: These are theoretical entities used to evaluate the effectiveness of a Master Spell, not direct references to Kubernetes pods or similar technologies.
The EsxiStateReader class is designed to encapsulate reading ESXi state information.
It provides methods for retrieving various metrics and data directly
from an ESXi host via SSH.
Creates an instance of the EsxiStateReader class.
ssh_operator: An instance of the SSHOperator class used for SSH communication.fqdn: The fully qualified domain name (FQDN) or IP address of the ESXi host.username: The username for authenticating with the ESXi host.password: The password for authenticating with the ESXi host.
Checks if there's an active SSH connection to the ESXi host.
Reads all adapter names from the ESXi host and returns them as a list.
Reads all adapter names that provide SR-IOV PF functionality.
Reads and returns a list of all virtual functions (VFs) for a particular parent network adapter (PF).
# Example usage of the EsxiStateReader class
with EsxiStateReader.from_optional_credentials(
esxi_fqdn="esxi.example.com", username="root", password="password") as esxi_reader:
adapter_names = esxi_reader.read_adapter_names()
print("Adapter Names:", adapter_names)
pf_adapter_names = esxi_reader.read_pf_adapter_names()
print("SR-IOV PF Adapter Names:", pf_adapter_names){
"vf-test-np1-h5mtj-9cf8fdcf6xcfln5-k6mdx": {
"esxiHost": "10.252.80.109",
"pnic_data": {
"50 36 69 7a d3 44 2d 5b-2d 9a 62 87 5e 09 35 18": {
"10.252.80.79": [
"vmnic5"
],
"10.252.80.107": [
"vmnic5"
],
"10.252.80.109": [
"vmnic5"
],
"10.252.80.96": [
"vmnic8"
]
},
"50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de": {
"10.252.80.79": [
"vmnic6"
],
"10.252.80.107": [
"vmnic6"
],
"10.252.80.109": [
"vmnic6"
],
"10.252.80.96": [
"vmnic9"
]
},
"50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7": {
"10.252.80.79": [
"vmnic7"
],
"10.252.80.107": [
"vmnic7"
],
"10.252.80.109": [
"vmnic7"
]
}
},
"sriov_adapters": [
{
"label": "SR-IOV network adapter 8",
"switchUuid": "50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de",
"portgroupKey": "dvportgroup-69",
"numaNode": 0,
"vf_mac": "00:50:56:b6:a0:5f",
"id": "0000:88:00.0",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:50",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic6"
},
{
"label": "SR-IOV network adapter 7",
"switchUuid": "50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de",
"portgroupKey": "dvportgroup-69",
"numaNode": 0,
"vf_mac": "00:50:56:b6:88:34",
"id": "0000:88:00.0",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:50",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic6"
},
{
"label": "SR-IOV network adapter 5",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:89:ed",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 4",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:d0:80",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 3",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:d0:d2",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 2",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:20:df",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 1",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:be:5f",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 9",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:9a:66",
"id": "0000:88:00.1",
"pf_host": "10.252.80.109",
"pf_mac": "40:a6:b7:3d:b2:51",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
}
],
"hardware_details": {
"numCPU": 32,
"numCoresPerSocket": 32,
"autoCoresPerSocket": false,
"memoryMB": 65536
}
},
"vf-test-np1-h5mtj-9cf8fdcf6xcfln5-k9jcm": {
"esxiHost": "10.252.80.107",
"pnic_data": {
"50 36 69 7a d3 44 2d 5b-2d 9a 62 87 5e 09 35 18": {
"10.252.80.79": [
"vmnic5"
],
"10.252.80.107": [
"vmnic5"
],
"10.252.80.109": [
"vmnic5"
],
"10.252.80.96": [
"vmnic8"
]
},
"50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de": {
"10.252.80.79": [
"vmnic6"
],
"10.252.80.107": [
"vmnic6"
],
"10.252.80.109": [
"vmnic6"
],
"10.252.80.96": [
"vmnic9"
]
},
"50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7": {
"10.252.80.79": [
"vmnic7"
],
"10.252.80.107": [
"vmnic7"
],
"10.252.80.109": [
"vmnic7"
]
}
},
"sriov_adapters": [
{
"label": "SR-IOV network adapter 9",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:4a:df",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 8",
"switchUuid": "50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de",
"portgroupKey": "dvportgroup-69",
"numaNode": 0,
"vf_mac": "00:50:56:b6:73:f1",
"id": "0000:88:00.0",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:30",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic6"
},
{
"label": "SR-IOV network adapter 7",
"switchUuid": "50 36 28 c6 57 2e 82 06-c0 28 b4 4a aa cd 33 de",
"portgroupKey": "dvportgroup-69",
"numaNode": 0,
"vf_mac": "00:50:56:b6:d7:3e",
"id": "0000:88:00.0",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:30",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic6"
},
{
"label": "SR-IOV network adapter 5",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:24:e5",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 4",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:81:16",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 3",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:2f:ff",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 2",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:07:a5",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
},
{
"label": "SR-IOV network adapter 1",
"switchUuid": "50 36 9d e2 45 9e 28 c0-81 8b 0b 75 e8 e1 5b d7",
"portgroupKey": "dvportgroup-2051",
"numaNode": 0,
"vf_mac": "00:50:56:b6:ba:c5",
"id": "0000:88:00.1",
"pf_host": "10.252.80.107",
"pf_mac": "40:a6:b7:35:6e:31",
"deviceName": "Ethernet Controller XXV710 for 25GbE SFP28",
"vendorName": "Intel(R)",
"pNIC": "vmnic7"
}
],
"hardware_details": {
"numCPU": 32,
"numCoresPerSocket": 32,
"autoCoresPerSocket": false,
"memoryMB": 65536
}
}
}