使用 docker 后出现的网络异常现象-白红宇

硬件:

[root@sh-storage-128204 ~]# dmidecode -t system dmidecode 2.12SMBIOS 2.7 present.Handle 0x0001, DMI type 1, 27 bytesSystem Information        Manufacturer: To be filled by O.E.M.        Product Name: Tecal RH2288H V2-24S        Version: V100R002        Serial Number: 2102310QPE10E9000146        UUID: 4A190814-D21D-B211-8DC0-000000821800        Wake-up Type: Power Switch        SKU Number: Type1Sku0        Family: Type1Family

环境:

[root@sh-storage-128204 ~]# docker infoContainers: 4Images: 153Storage Driver: devicemapper Pool Name: docker-8:17-1075308946-pool Pool Blocksize: 65.54 kB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 3.478 GB Data Space Total: 107.4 GB Data Space Available: 103.9 GB Metadata Space Used: 6.689 MB Metadata Space Total: 2.147 GB Metadata Space Available: 2.141 GB Udev Sync Supported: true Deferred Removal Enabled: false Data loop file: /var/lib/docker/devicemapper/devicemapper/data Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata Library Version: 1.02.107-RHEL7 (2015-10-14)Execution Driver: native-0.2Logging Driver: json-fileKernel Version: 3.10.0-229.el7.x86_64Operating System: CentOS Linux 7 (Core)CPUs: 24Total Memory: 125.7 GiBName: sh-storage-128204.sh.vclound.comID: QV4X:VHQE:EMOI:4TBJ:FZ6K:3N3C:A64Y:PRYR:X2QZ:HHUB:OTND:ZSFF

用法:

用于搭建 docker private registry

架构:

nginx    <- 本地 docker                    |    |---------------+-------------------|   registry       registry        registry  <- 本地 docker     |---------------+-------------------|                    |                ceph 集群 (rados) <- 另外一个集群

问题:

搭建了 docker, 启动成功后, 发现上传镜像时偶尔会出现连接卡住问题

检测与 mon 连接发现:

[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CSTsendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permittedOffending packet: TCP 10.198.128.204:43301 > 10.198.128.200:6789 S ttl=48 id=11619 iplen=44  seq=2271879518 win=1024 
     
      sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permittedOffending packet: TCP 10.198.128.204:43302 > 10.198.128.200:6789 S ttl=45 id=9421 iplen=44  seq=2271945055 win=1024 
      
       Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)Host is up (0.000091s latency).PORT     STATE    SERVICE6789/tcp filtered ibm-db2-adminMAC Address: 90:E2:BA:85:21:28 (Intel Corporate)Nmap done: 1 IP address (1 host up) scanned in 0.29 seconds[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CSTsendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permittedOffending packet: TCP 10.198.128.204:48544 > 10.198.128.200:6789 S ttl=54 id=17389 iplen=44  seq=1275261731 win=1024 
       
        sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 10.198.128.200, 16) => Operation not permittedOffending packet: TCP 10.198.128.204:48545 > 10.198.128.200:6789 S ttl=53 id=40168 iplen=44  seq=1275327266 win=1024 
        
         Nmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)Host is up (0.000092s latency).PORT     STATE    SERVICE6789/tcp filtered ibm-db2-adminMAC Address: 90:E2:BA:85:21:28 (Intel Corporate)Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds[root@sh-storage-128204 ~]# nmap -p 6789 10.198.128.200Starting Nmap 6.40 ( http://nmap.org ) at 2016-01-20 09:47 CSTNmap scan report for sh-storage-128200.sh.vclound.com (10.198.128.200)Host is up (0.00039s latency).PORT     STATE SERVICE6789/tcp open  ibm-db2-adminMAC Address: 90:E2:BA:85:21:28 (Intel Corporate)Nmap done: 1 IP address (1 host up) scanned in 0.21 seconds

与 mon 连接时, 会经常出现 6789/tcp filtered ibm-db2-admin 异常现象

另外, 参考系统日志

会出现下面异常 (过滤了大部分, 只显示重要部分信息

/var/log/messagesJan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packetJan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packetJan 20 09:49:16 sh-storage-128204 kernel: nf_conntrack: table full, dropping packet....Jan 20 09:49:44 sh-storage-128204 kernel: net_ratelimit: 137 callbacks suppressedJan 20 09:49:49 sh-storage-128204 kernel: net_ratelimit: 166 callbacks suppressedJan 20 09:49:54 sh-storage-128204 kernel: net_ratelimit: 64 callbacks suppressedJan 20 09:50:01 sh-storage-128204 kernel: net_ratelimit: 132 callbacks suppressed

ping 也会出现异常现场

[root@sh-storage-128204 ~]# ping localhostPING localhost (127.0.0.1) 56(84) bytes of data.ping: sendmsg: Operation not permitted

原因: 当前的网络连接数过大, 导致内核无法响应

解决方法:

修改 /et/sysctl.conf 进行内核参数优化

[root@sh-storage-128204 ~]# sysctl  -pvm.swappiness = 10net.ipv4.ip_forward = 1net.ipv4.tcp_max_syn_backlog = 8192net.core.netdev_max_backlog = 8192net.ipv4.tcp_fin_timeout = 15net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.netfilter.nf_conntrack_max = 1048576net.nf_conntrack_max = 1048576net.netfilter.nf_conntrack_tcp_timeout_established = 54000net.netfilter.nf_conntrack_generic_timeout = 120