Long delays to accept new ingress flows from new pods

[micahnoland]

What happened:

We are observing an issue in our EKS clusters where new client pod traffic is denied on the receiving side for a period (ranging from a few seconds to several minutes), until the flow is accepted. This is causing many spikes of tail latency across many production services. When we redeploy the client pod, we see Verdict DENY flow logs from the new pod for some time before finally switching to Verdict ACCEPT. Here is an example with nearly 3-minute delay, the new client pod is 10.5.223.87 and we see these messages in the network-policy-agent.log file on the receiving pod's host:

{"level":"info","ts":"2025-11-21T03:24:44.553Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:45.586Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:47.676Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:51.746Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56646 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:57.581Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:24:58.626Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:00.706Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:04.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 50600 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:13.645Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:14.706Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:16.636Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:16.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:17.666Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:19.746Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:20.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 46056 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:23.826Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 55760 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:35.697Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:36.708Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:38.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:41.724Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:42.786Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:42.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42802 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:44.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:48.946Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 42808 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:54.751Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:55.826Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:25:57.906Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:01.996Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 59622 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:13.806Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:14.866Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:16.946Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:21.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56272 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:42.890Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:43.906Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:45.986Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:50.076Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 56878 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:57.958Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:26:59.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:01.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:03.968Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:05.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:05.192Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41482 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:07.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:11.186Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41486 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:23.011Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:24.076Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:26.146Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:30.026Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:30.226Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35068 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"info","ts":"2025-11-21T03:27:31.106Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict DENY Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:31.931Z","caller":"ebpf/bpf_client.go:1074","msg":"No L4 specified. Add Catch all entry CIDR: 10.5.223.87/32"}
{"level":"info","ts":"2025-11-21T03:27:31.939Z","caller":"ebpf/bpf_client.go:1074","msg":"Updating Map with IP Key: 10.5.223.87/32"}
{"level":"debug","ts":"2025-11-21T03:27:33.186Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 35908 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:35.259Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 38164 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:35.310Z","caller":"ebpf/bpf_client.go:1074","msg":"No L4 specified. Add Catch all entry CIDR: 10.5.223.87/32"}
{"level":"info","ts":"2025-11-21T03:27:35.318Z","caller":"ebpf/bpf_client.go:1074","msg":"Updating Map with IP Key: 10.5.223.87/32"}
{"level":"debug","ts":"2025-11-21T03:27:44.470Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41508 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}
{"level":"debug","ts":"2025-11-21T03:27:46.533Z","caller":"runtime/asm_arm64.s:1223","msg":"Flow Info: Src IP: 10.5.223.87 Src Port: 41520 Dest IP: 10.5.72.157 Dest Port: 8080 Proto TCP Verdict ACCEPT Direction ingress"}

Attach logs

Logs sent to k8s-awscni-triage@amazon.com

What you expected to happen:

Short or negligible delays for a new flows from new client pods to be accepted on the ingress side when NetworkPolicy allows the traffic on both sides.

How to reproduce it (as minimally and precisely as possible):

We have been able to reproduce this at will in our large clusters (700 nodes, 11000 pods). We have two pods, a client and a server, in different namespaces running on different nodes with NetworkPolicy resources on both sides. Restarting the client pod nearly always triggers the issue.

Anything else we need to know?:

We are using the default NETWORK_POLICY_ENFORCING_MODE=standard

Environment:

• Kubernetes version (use kubectl version): Server Version: v1.33.5-eks-3cfe0ce
• CNI Version: v1.20.4-eksbuild.1
• Network Policy Agent Version: v1.2.7
• OS (e.g: cat /etc/os-release): Amazon Linux 2023.8.20250707
• Kernel (e.g. uname -a): 6.1.141-165.249.amzn2023.aarch64 #1 SMP Tue Jul 1 18:00:46 UTC 2025 aarch64 aarch64 aarch64 GNU/Linux

[yash97]

Hi can you also share network policy that was used by these components.

[micahnoland]

Hi can you also share network policy that was used by these components.

I emailed them to k8s-awscni-triage@amazon.com

[theadarshsaxena]

Are there any updates on this issue?