Ingress 旁路/镜像联动阻断方案（Nginx Ingress + ModSecurity/OWASP CRS + Redis）

kbsonlong

2025-12-26 约 2700 字预计阅读 6 分钟 - 次阅读

背景

旁路/镜像模式用于“先检测再阻断”。自身不在链路上，无法直接拦截，需要将阻断动作联动到真实入口（Ingress/Gateway/防火墙等）。本方案在 Kubernetes 中通过 Nginx Ingress Controller 启用 ModSecurity（仅检测），并用 Redis 维护动态黑名单，定期把黑名单同步到 Ingress 的阻断策略，实现“旁路检测→入口阻断”闭环。

架构

旁路检测：镜像流量给检测组件（可用开源 WAF 检测器或自研规则），命中后写入 Redis（含 TTL）
决策引擎：基于来源、命中次数和白名单等规则，写入/更新 Redis 黑名单集合
执行器：控制器轮询 Redis，将黑名单转为 Ingress 阻断（server-snippet/路径范围等），统一入口生效
观测：记录阻断率、误报率、来源分布，定期复盘规则

前置条件

已部署 Kubernetes 与 Nginx Ingress Controller
使用 Docker 运行第三方组件（如 Redis）
具备基础的 kubectl 与容器镜像仓库网络访问

部署步骤

1. 启用 ModSecurity（仅检测）

示例 Ingress（关键注解启用 ModSecurity 并设置为仅检测）：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine DetectionOnly
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: app-svc
                port:
                  number: 80

说明：先以 DetectionOnly 运行，避免误报直接拦截。待观测稳定后再切换到 On。

2. Docker 运行 Redis

1
2
3
4
docker run -d \
  --name waf-redis \
  -p 6379:6379 \
  redis:7

Redis 键设计示例：waf:ban:ip（Set），元素为 x.x.x.x，同时用 waf:ban:ttl:x.x.x.x（String）存储过期时间，控制器据此生成阻断名单。

3. 动态黑名单联动到 Ingress

在 Nginx Ingress 中用 server-snippet 注入阻断逻辑，控制器把 Redis 中的 IP 列表渲染为 map 并触发 Nginx reload（Ingress 变更会自动热加载）。

示例：在目标 Ingress 增加注解（由控制器渲染）：

1
2
3
4
5
6
7
8
metadata:
  annotations:
    nginx.ingress.kubernetes.io/server-snippet: |
      real_ip_header X-Forwarded-For;
      set $block_ip 0;
      # 由控制器渲染的 IP 列表
      if ($remote_addr ~* "^(1.2.3.4|5.6.7.8)$") { set $block_ip 1; }
      if ($block_ip) { return 403; }

控制器实现思路：

周期性查询 Redis waf:ban:ip，过滤过期项与白名单
将 IP 集合渲染为正则分组（或多条 deny）写入 server-snippet
对指定 Ingress 执行 kubectl patch 更新注解，使阻断立即生效

4. 旁路检测写入 Redis（示例流程）

镜像流量到检测组件（如使用开源 WAF 或自研 Lua/Go 规则）
命中后调用 Redis：SADD waf:ban:ip <ip> 并设置 SETEX waf:ban:ttl:<ip> <seconds> 1
达到阈值或多规则命中才写入，以降低误报

灰度与回滚

先仅检测：SecRuleEngine DetectionOnly
小流量灰度：只对小范围域名/路径启用阻断
TTL 自动解封：黑名单到期自动移除，降低长期误封
快速回滚：控制器支持清空或回退上一次注解版本

验证

正常请求与恶意请求各发一组，观察 403 返回与 Ingress 控制器日志
统计阻断率与误报率（结合后端 4xx/5xx 指标）

运维要点

白名单：监控/办公网、CDN 出口等固定来源
分层处置：边缘限速/挑战，入口阻断，内部服务细粒度策略
观测复盘：按周/月评估规则有效性，持续优化

何时切换为拦截

明显攻击流量（高频、特征明确）且经观测无业务误伤
采用短 TTL 与逐步扩域的方式启用拦截，随时可回退

引流到旁路WAF服务

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
apiVersion: v1
kind: Namespace
metadata:
  name: waf-poc
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: waf-poc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:7
        ports:
        - containerPort: 6379
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: waf-poc
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: waf-detector-config
  namespace: waf-poc
data:
  # Nginx 配置：启用 ModSecurity 并直接返回 200（作为旁路接收端）
  default.conf: |
    server {
        listen 80;
        server_name localhost;
        
        # 开启 ModSecurity
        modsecurity on;
        modsecurity_rules_file /etc/nginx/modsec/main.conf;

        location / {
            # 旁路模式只负责检测，检测完直接返回 OK
            return 200 "OK";
        }
    }
  
  # 日志处理脚本：读取 ModSecurity 审计日志 -> 提取特征 -> 写入 Redis
  log_processor.py: |
    import time, json, os, redis
    
    LOG_FILE = "/var/log/modsec/audit.log"
    REDIS_HOST = os.getenv('REDIS_HOST', 'redis')
    
    r = redis.Redis(host=REDIS_HOST, port=6379, decode_responses=True)
    
    def follow(thefile):
        thefile.seek(0, 2)
        while True:
            line = thefile.readline()
            if not line:
                time.sleep(0.1)
                continue
            yield line

    print("Starting Log Processor...")
    while not os.path.exists(LOG_FILE):
        print(f"Waiting for {LOG_FILE}...")
        time.sleep(1)

    with open(LOG_FILE, "r") as f:
        for line in follow(f):
            try:
                # ModSecurity JSON 格式日志
                entry = json.loads(line)
                transaction = entry.get('transaction', {})
                
                # 1. 提取 IP
                client_ip = transaction.get('client_ip')
                
                # 2. 提取 User-Agent
                headers = transaction.get('request', {}).get('headers', {})
                user_agent = headers.get('User-Agent')
                
                # 3. 提取攻击类型 (Rule ID / Tags)
                messages = [m['message'] for m in transaction.get('messages', [])]
                
                if client_ip:
                    print(f"Attack detected from {client_ip}: {messages[0] if messages else 'Unknown'}")
                    
                    # 写入 IP 黑名单
                    r.sadd('waf:ban:ip', client_ip)
                    r.setex(f'waf:ban:ttl:{client_ip}', 600, '1')
                    
                    # 写入 User-Agent 黑名单 (示例：如果包含 specific-bot 特征)
                    # 在实际场景中，建议结合规则 ID 判断是否封禁 UA
                    if user_agent and "BadBot" in user_agent:
                        r.sadd('waf:ban:ua', user_agent)
                        r.setex(f'waf:ban:ttl:ua:{user_agent}', 3600, '1')
                        
            except Exception as e:
                print(f"Error processing log: {e}")
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: waf-detector
  namespace: waf-poc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: waf-detector
  template:
    metadata:
      labels:
        app: waf-detector
    spec:
      containers:
      # 容器 1: ModSecurity (基于 Nginx)
      - name: modsec
        image: owasp/modsecurity-crs:nginx
        env:
        - name: PARANOIA
          value: "1"
        - name: ANOMALY_INBOUND
          value: "5"
        # 强制使用 JSON 日志格式
        - name: MODSEC_AUDIT_LOG_FORMAT
          value: "JSON"
        ports:
        - containerPort: 80
        volumeMounts:
        - name: config
          mountPath: /etc/nginx/conf.d/default.conf
          subPath: default.conf
        - name: logs
          mountPath: /var/log/modsec
          
      # 容器 2: 日志处理器 (Sidecar)
      - name: processor
        image: python:3.11-slim
        command: ["sh", "-c"]
        args: ["pip install redis && python -u /app/log_processor.py"]
        env:
        - name: REDIS_HOST
          value: redis
        volumeMounts:
        - name: script
          mountPath: /app/log_processor.py
          subPath: log_processor.py
        - name: logs
          mountPath: /var/log/modsec
          
      volumes:
      - name: config
        configMap:
          name: waf-detector-config
      - name: script
        configMap:
          name: waf-detector-config
      - name: logs
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: waf-detector
  namespace: waf-poc
spec:
  selector:
    app: waf-detector
  ports:
  - port: 80
    targetPort: 80

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/enable-modsecurity: "true"
    nginx.ingress.kubernetes.io/enable-owasp-core-rules: "true"
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine DetectionOnly
    nginx.ingress.kubernetes.io/configuration-snippet: |
      mirror /__waf_mirror;
      mirror_request_body on;
    nginx.ingress.kubernetes.io/server-snippet: |
      location = /__waf_mirror {
        proxy_pass http://waf-detector.waf-poc.svc.cluster.local:80$request_uri;
        proxy_read_timeout 1s;
      }
      real_ip_header X-Forwarded-For;
      set $block_ip 0;
      if ($remote_addr ~* "^(1.2.3.4)$") { set $block_ip 1; }
      if ($block_ip) { return 403; }
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: app-svc
            port:
              number: 80

可执行POC联动控制器

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
apiVersion: v1
kind: ConfigMap
metadata:
  name: waf-controller
  namespace: waf-poc
data:
  controller.py: |
    import os, redis, re
    from kubernetes import client, config
    config.load_incluster_config()
    k8s = client.NetworkingV1Api()
    r = redis.Redis(host=os.getenv('REDIS_HOST','redis'), port=int(os.getenv('REDIS_PORT','6379')))
    ingress_ns = os.getenv('INGRESS_NS','default')
    ingress_name = os.getenv('INGRESS_NAME','app-ingress')
    
    # 1. 处理 IP 黑名单
    ips = [i.decode() for i in r.smembers('waf:ban:ip')]
    valid_ips = []
    for ip in ips:
      if r.ttl(f'waf:ban:ttl:{ip}') > 0:
        valid_ips.append(ip)
    
    # 2. 处理 User-Agent 黑名单 (示例)
    uas = [u.decode() for u in r.smembers('waf:ban:ua')]
    valid_uas = []
    for ua in uas:
      if r.ttl(f'waf:ban:ttl:ua:{ua}') > 0:
        valid_uas.append(ua)

    # 3. 生成 Nginx 规则
    block_logic = "real_ip_header X-Forwarded-For;\nset $block_ip 0;\n"
    
    # IP 拦截规则
    if valid_ips:
        ip_pattern = "|".join([re.escape(i) for i in valid_ips])
        block_logic += f"if ($remote_addr ~* \"^({ip_pattern})$\") {{ set $block_ip 1; }}\n"
        
    # UA 拦截规则
    if valid_uas:
        ua_pattern = "|".join([re.escape(u) for u in valid_uas])
        block_logic += f"if ($http_user_agent ~* \"({ua_pattern})\") {{ set $block_ip 1; }}\n"
        
    block_logic += "if ($block_ip) { return 403; }\n"
    
    loc = "location = /__waf_mirror {\nproxy_pass http://waf-detector.waf-poc.svc.cluster.local:80$request_uri;\nproxy_read_timeout 1s;\n}\n"
    snippet = loc + block_logic
    
    body = {"metadata": {"annotations": {"nginx.ingress.kubernetes.io/server-snippet": snippet}}}
    k8s.patch_namespaced_ingress(name=ingress_name, namespace=ingress_ns, body=body)
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: waf-controller
  namespace: waf-poc
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: waf-controller
  namespace: default
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: waf-controller
  namespace: default
subjects:
- kind: ServiceAccount
  name: waf-controller
  namespace: waf-poc
roleRef:
  kind: Role
  name: waf-controller
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: waf-controller
  namespace: waf-poc
spec:
  schedule: "*/1 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: waf-controller
          restartPolicy: Never
          containers:
          - name: controller
            image: python:3.11-slim
            command: ["python","/opt/controller.py"]
            env:
            - name: REDIS_HOST
              value: redis
            - name: REDIS_PORT
              value: "6379"
            - name: INGRESS_NS
              value: default
            - name: INGRESS_NAME
              value: app-ingress
            volumeMounts:
            - name: script
              mountPath: /opt/controller.py
              subPath: controller.py
          volumes:
          - name: script
            configMap:
              name: waf-controller

WAF 规则管理

虽然 ModSecurity 主要使用官方的 CRS 规则集，但你仍然可以通过挂载配置文件的方式来添加自定义规则。

1. 自定义规则挂载

将自定义规则写入 ConfigMap：

1
2
3
4
5
6
7
8
9
apiVersion: v1
kind: ConfigMap
metadata:
  name: waf-custom-rules
  namespace: waf-poc
data:
  custom.conf: |
    # 示例：检测 Log4j Payload
    SecRule ARGS "\$\{jndi:" "id:1001,phase:2,deny,status:403,msg:'Log4j Attack'"

修改 Deployment 挂载此规则到 /etc/nginx/modsec/ 并在 main.conf 中 include 它（需调整 default.conf 或启动脚本，这里仅为示意）。

2. 查看攻击日志

可以通过查看 processor 容器的日志来观察实时的攻击检测情况：

1
kubectl logs -f -n waf-poc deployment/waf-detector -c processor

输出示例：

1
2
Attack detected from 1.2.3.4: SQL Injection Attack: Common Injection Testing Detected
Attack detected from 5.6.7.8: BadBot User-Agent Detected

执行步骤

应用命名空间与组件
部署 Ingress 并生效镜像引流
部署 CronJob 联动阻断
发送测试请求验证 403 与 Redis 命中