# 部署

## cpuinfo
```bash
# 查看cpu个数
grep ^processor /proc/cpuinfo | wc -l
lshw -short -class cpu
cat /proc/cpuinfo | grep "model name"
```

## nofile ulimit
```bash
# cat /etc/security/limits.conf
# 查看系统资源限制
ulimit -a
ulimit -n
ulimit -Sn
ulimit -Hn
# ulimit -HSn 102400
或
# vim /etc/security/limits.conf  //加入以下配置，重启即可生效
* soft nofile 65535
* hard nofile 65535
```

systemd配置LimitNOFILE

```bash
# 全局配置
cat /etc/systemd/system.conf |grep -E 'NOFILE|NPROC'
## DefaultLimitNOFILE=1024000
## DefaultLimitNPROC=1024000
centos7里 openfile不能大于nr_open
cat /proc/sys/fs/nr_open
```
```bash
# systemd
[Service]
#LimitNOFILE=65536
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
LimitNOFILE=512000
LimitNPROC=512000
```
检测有没有生效

```bash
## 查看生效
cat /proc/[PID]/limits
```

## fs.file-max
```bash
sysctl -w net.ipv4.tcp_slow_start_after_idle=0
sysctl -w fs.file-max=6553560
# 查看分配状态，一共三个值
# 第一个代表全局已经分配的文件描述符数量
# 第二个代表自由的文件描述符（待重新分配的）
# 第三个代表总的文件描述符的数量
cat /proc/sys/fs/file-nr
```


## sysctl.conf
```ini
# fs.file-max一般为内存大小（KB）的10%来计算，如果使用shell，可以这样计算
# grep -r MemTotal /proc/meminfo | awk '{printf("%d",$2/10)}'

fs.file-max = 1024000
fs.inotify.max_user_instances = 8192
fs.inotify.max_user_watches = 89100

vm.max_map_count = 262144

#net.core.default_qdisc = fq
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 4096

net.ipv4.ip_forward = 1
net.ipv4.tcp_slow_start_after_idle = 0
#net.ipv4.tcp_congestion_control = bbr
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_max_tw_buckets = 6000
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_orphans = 65535
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 30


# TCP receive buffer
net.ipv4.tcp_rmem = 4096 87380 67108864
# TCP write buffer
net.ipv4.tcp_wmem = 4096 65536 67108864
# turn on path MTU discovery
net.ipv4.tcp_mtu_probing = 1
# max read buffer
net.core.rmem_max = 67108864
# max write buffer
net.core.wmem_max = 67108864
# default read buffer
net.core.rmem_default = 65536
# default write buffer
net.core.wmem_default = 65536


net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 30
net.ipv4.tcp_keepalive_probes = 10
```

## build
```bash
sed -i.bak 's/CFLAGS="$CFLAGS -g"/#CFLAGS="$CFLAGS -g"/g' auto/cc/gcc
## gcc编译参数
--with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
--with-cc-opt='-O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'
--with-ld-opt='-lrt -ljemalloc -Wl,-z,relro'
--with-ld-opt=-Wl,-E
--with-cc-opt='-m64 -mtune=native -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wno-sign-compare -Wno-string-plus-int -Wno-deprecated-declarations -Wno-unused-parameter -Wno-unused-const-variable -Wno-conditional-uninitialized -Wno-mismatched-tags -Wno-c++11-extensions -Wno-sometimes-uninitialized -Wno-parentheses-equality -Wno-tautological-compare -Wno-self-assign -Wno-deprecated-register -Wno-deprecated -Wno-invalid-source-encoding -Wno-pointer-sign -Wno-parentheses -Wno-enum-conversion'
## configure example
/bin # nginx -V
nginx version: nginx/1.15.2
built by gcc 6.4.0 (Alpine 6.4.0) 
built with OpenSSL 1.0.2o  27 Mar 2018
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_addition_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_random_index_module --with-http_secure_link_module --with-http_stub_status_module --with-http_auth_request_module --with-http_xslt_module=dynamic --with-http_image_filter_module=dynamic --with-http_geoip_module=dynamic --with-threads --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-stream_realip_module --with-stream_geoip_module=dynamic --with-http_slice_module --with-mail --with-mail_ssl_module --with-compat --with-file-aio --with-http_v2_module
```
## nginx.conf
```nginx
# One worker per CPU-core.
worker_processes  4;
events {
  worker_connections  8096;
  multi_accept        on;
  use                 epoll;
}
worker_rlimit_nofile 40000;
pcre_jit on;

# Gzip Settings
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 32 16k;
gzip_http_version 1.1;
gzip_min_length 250;
gzip_types image/jpeg image/bmp image/svg+xml text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon;
# Brotli Settings
brotli on;
brotli_comp_level 4;
brotli_buffers 32 8k;
brotli_min_length 100;
brotli_static on;
brotli_types image/jpeg image/bmp image/svg+xml text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript image/x-icon;


stream {
 server {
  listen 30094;
  proxy_connect_timeout 1s;
  proxy_timeout 5s;
  proxy_pass 192.168.33.27:30094;
 }
}

http {
  server_tokens off;
  sendfile           on;
  tcp_nopush         on;
  tcp_nodelay        on;
  keepalive_timeout  15;
  keepalive_requests 1000;
  open_file_cache max=200000 inactive=20s;
  open_file_cache_valid 30s;
  open_file_cache_min_uses 2;
  open_file_cache_errors on;
  server {
    # security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer-when-downgrade" always;
    add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
    #add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
    #add_header X-Frame-Options DENY;
    #add_header X-Content-Type-Options nosniff;
    #add_header X-XSS-Protection "1; mode=block";
    add_header Set-Cookie "HttpOnly";
    add_header Set-Cookie "Secure";
    send_timeout 10;
    #client_max_body_size 10m;
    #client_body_buffer_size 128k;
    client_body_timeout 12;
    client_header_timeout 12;		
    client_body_buffer_size 10K;
    client_header_buffer_size 1k;
    client_max_body_size 8m;
    large_client_header_buffers 2 1k;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_connect_timeout 10;
    proxy_send_timeout 10;
    proxy_read_timeout 10;		
    proxy_buffer_size 4k;
    proxy_buffers 4 32k;
    proxy_busy_buffers_size 64k;        
    proxy_temp_file_write_size 64k;           
    proxy_ignore_client_abort on;
    location /{
      proxy_pass http://backend;
    }
  }
  upstream backend{
    server   127.0.0.1:1313  weight=1 max_fails=2 fail_timeout=10s;
    keepalive 300;
  }
}
```

## ipv6
```nginx
# 同时监听IPV4和IPV6
server {
  listen [::]:80;
}
# 只监听ipv6
server {
  listen [::]:80 default ipv6only=on;
}
# ipv6 指定IP
server {
  listen [3608:f0f0:3002:31::1]:80;
}

```


## nginx.service
```toml
[Unit]
Description=The NGINX HTTP and reverse proxy server
#After=syslog.target network.target remote-fs.target nss-lookup.target
After=syslog.target network.target

[Service]
LimitNOFILE=30000
Type=forking
PIDFile=/opt/nginx/logs/nginx.pid
ExecStartPre=/opt/nginx/sbin/nginx -t
ExecStart=/opt/nginx/sbin/nginx
ExecReload=/opt/nginx/sbin/nginx -s reload
ExecStop=/bin/kill -s QUIT $MAINPID
PrivateTmp=true

[Install]
WantedBy=multi-user.target
```

## troubleshooting
```bash
netstat -n|awk '/^tcp/{++S[$NF]}END{for (key in S) print key,S[key]}'
# 返回结果如下：
LAST_ACK 14
SYN_RECV 348
ESTABLISHED 70
FIN_WAIT1 229
FIN_WAIT2 30
CLOSING 33
TIME_WAIT 18122
# 说明
# CLOSED：无连接是活动的或正在进行
# LISTEN：服务器在等待进入呼叫
# SYN_RECV：一个连接请求已经到达，等待确认
# SYN_SENT：应用已经开始，打开一个连接
# ESTABLISHED：正常数据传输状态
# FIN_WAIT1：应用说它已经完成
# FIN_WAIT2：另一边已同意释放
# ITMED_WAIT：等待所有分组死掉
# CLOSING：两边同时尝试关闭
# TIME_WAIT：另一边已初始化一个释放
# LAST_ACK：等待所有分组死掉

ss -tan state time-wait|wc -l
###
netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'
```
```nginx
location / {
  proxy_intercept_errors on;
  proxy_set_header Host $host;
  proxy_pass http://www_server3_plools;
  proxy_set_header X-Forwarded-For $remote_addr;
  proxy_next_upstream error timeout http_503 non_idempotent;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

error_page 500 502 503 504 =200 /error.json;

location = /error.json {
  default_type application/json;
  return 200 '{"retCode":"1001","retMsg":"invoke failed"}';
}

# 需要略小于upstream的对应配置
keepalive_timeout   30s;
keepalive_requests  10;
 
upstream servers_test {
    server 127.0.0.1:5001 max_fails=1 fail_timeout=10s weight=1;
    server 127.0.0.1:5002 max_fails=1 fail_timeout=10s weight=1;
 
    keepalive 10;
}
 
location = /test {
    proxy_connect_timeout 500ms;
    proxy_read_timeout 60s;
    proxy_send_timeout 10s;
    proxy_next_upstream error timeout non_idempoten;
    proxy_next_upstream_timeout 10s;
 
    proxy_pass http://servers_test;
}
```



# ref
* [Centos7一键优化](https://www.dgstack.cn/archives/2432.html)
* [linux ulimit 调优](https://www.cnblogs.com/sunsky303/p/8359592.html)
* [tcp参数设置](https://www.cnblogs.com/DengGao/p/tcp_parameter.html)
* [你所不知道的TIME_WAIT和CLOSE_WAIT](https://www.cnblogs.com/sunsky303/p/11642601.html)
* [tuning-nginx](https://www.nginx.com/blog/tuning-nginx/)
* [NGINX Tuning](http://www.tweaked.io/guide/nginx/)
* [Linux Kernel](http://www.tweaked.io/guide/kernel/)
* [NGINX Tuning For Best Performance](https://github.com/denji/nginx-tuning)
* [Getting Started with NGINX](https://www.linode.com/docs/web-servers/nginx/slightly-more-advanced-configurations-for-nginx/)
* [linode nginx doc](https://github.com/linode/docs/tree/master/docs/web-servers/nginx)
* [Core functionality](http://nginx.org/en/docs/ngx_core_module.html)
* [nginx 常用模块整理](http://blog.51cto.com/arm2012/1977090)
* [Performance Tuning – Tips & Tricks](https://www.nginx.com/blog/performance-tuning-tips-tricks/)
* [cors.nginxconf](https://gist.githubusercontent.com/pauloricardomg/7084524/raw/6d9c5b5e96e460a6b601f071ebb2db2cb438cea1/cors.nginxconf)
* [Strong SSL Security on nginx](https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html)
* [Best nginx configuration for improved security(and performance). ](https://gist.github.com/plentz/6737338)
* [Secure your Cookies (Secure and HttpOnly flags)](https://blog.dareboost.com/en/2019/03/secure-cookies-secure-httponly-flags/)
* [SSL and TLS Deployment Best Practices · ssllabs/research Wiki · GitHub](https://github.com/ssllabs/research/wiki/SSL-and-TLS-Deployment-Best-Practices)
* [详解压测中出现的 nginx 502 no live upstreams 错误](https://xnathan.com/2019/05/07/nginx-502/)
* [Linux内核调优部分参数说明 – 运维生存时间](http://www.ttlsa.com/linux/linux-kernel-tuning-section-parameter-description/)
* [Collection of /etc/sysctl.conf networking notes. - bl.ocks.org](https://bl.ocks.org/magnetikonline/2760f98f6bf654d5ad79)
* [High Volume Incoming Connections - Level Up Coding](https://levelup.gitconnected.com/linux-kernel-tuning-for-high-performance-networking-high-volume-incoming-connections-196e863d458a)
* [58房产Nginx 网络调优实践](https://mp.weixin.qq.com/s/A3e9eT_FWvl1am134Pv0AQ)
* [解Bug之路-Nginx 502 Bad Gateway](https://mp.weixin.qq.com/s/9IwiOgQzu63qB_hfOeOMLg)
* [Nginx+Tomcat偶现502分析](https://iyaozhen.com/nginx-tomcat-502-and-proxy_next_upstream.html)