About increasing the number of bpf parameter transfers
ShawnLeung87 opened this issue · comments
Is it possible to adjust gk_bpf_cookie mem parameter, let bpf increase parameter passing. this "uint64_t mem[8];" What is the maximum limit?
struct gk_bpf_cookie {
uint64_t mem[8];
};
struct gk_bpf_cookie is used to hold the parameters and state of each flow. Having a larger struct for parameters only would affect the protocol between Grantor and Gatekeeper servers; this is not that challenging to do. Having a larger struct for state only would increase the demand for RAM to hold the flow table on Gatekeeper servers; this is not a cheap compromise.
Production Gatekeeper servers allocate 90+% of all their RAM to their flow tables, so increasing the size of struct gk_bpf_cookie would require even more RAM to track the same number of flows. More state per flow also increases the traffic in the buses that connect processors and memory banks, which can lead to bus saturation. Bus saturation limits the number of packets Gatekeeper servers can process each second. Increasing struct gk_bpf_cookie and tracking fewer flows is not attractive either because the efficacy and effectiveness of Gatekeeper come from its capacity to track billions of simultaneous flows.
Consider the following example. A production Gatekeeper server is typically dimensioned to track 1 billion (i.e., struct gk_bpf_cookie is a multiple of a processor cache line (i.e., 64 bytes). So to increase struct gk_bpf_cookie from one cache line (i.e. uint64_t mem[8];) to two cache lines (i.e. uint64_t mem[16];) would require 64 bytes * 1 billion
My arguments here are not meant to say that it cannot be done, but to point out that it has a high impact. As DDR5 RAM becomes more accessible, the expansion of struct gk_bpf_cookie may come into reach.
Could you describe the demand that brought your question up?
Ok, then I try to keep the BPF parameters in the mem[8] range.