1、首先docker日志
1
2
3
4
5
6
7
8
9
10
11
| Jul 16 14:41:46 worker1 dockerd[2396]: time="2020-07-16T14:41:46.453596079+08:00" level=error msg="Download failed, retrying: unexpected EOF"
Jul 16 14:42:06 worker1 dockerd[2396]: time="2020-07-16T14:42:06.483391207+08:00" level=error msg="Download failed: unexpected EOF"
Jul 16 14:42:29 worker1 dockerd[2396]: time="2020-07-16T14:42:29.220439089+08:00" level=warning msg="Error getting v2 registry: Get https://af.hikvision.com.cn/v2/: dial tcp: lookup af.hikvision.com.cn on
Jul 16 14:42:29 worker1 dockerd[2396]: time="2020-07-16T14:42:29.245091739+08:00" level=warning msg="Error getting v2 registry: Get http://af.hikvision.com.cn/v2/: dial tcp: lookup af.hikvision.com.cn on
Jul 16 14:42:29 worker1 dockerd[2396]: time="2020-07-16T14:42:29.245196494+08:00" level=error msg="Handler for POST /images/create returned error: Get http://af.hikvision.com.cn/v2/: dial tcp: lookup af.h
Jul 16 14:46:33 worker1 dockerd[2396]: time="2020-07-16T14:46:33.972502089+08:00" level=error msg="Download failed, retrying: unexpected EOF"
Jul 16 14:46:38 worker1 dockerd[2396]: time="2020-07-16T14:46:38.999069663+08:00" level=error msg="Download failed, retrying: unexpected EOF"
Jul 16 14:46:49 worker1 dockerd[2396]: time="2020-07-16T14:46:49.014718214+08:00" level=error msg="Download failed, retrying: unexpected EOF"
Jul 16 14:47:04 worker1 dockerd[2396]: time="2020-07-16T14:47:04.041825197+08:00" level=error msg="Download failed, retrying: unexpected EOF"
Jul 16 14:47:24 worker1 dockerd[2396]: time="2020-07-16T14:47:24.070685084+08:00" level=error msg="Download failed: unexpected EOF"
|
2、修改域名解析问题
1
2
3
4
5
6
| nameserver 114.114.114.114
nameserver 10.96.0.10
nameserver 10.1.7.77
nameserver 10.1.7.97
nameserver 10.1.7.98
nameserver 10.1.7.88
|
修改为
3、查看harbor日志
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| 命令：
k logs harbor-harbor-registry-68c9db8d9c-m76x8 -c registry --tail=100
输出：
10.19.123.165 - - [16/Jul/2020:07:30:39 +0000] "GET /v2 HTTP/1.1" 301 39 "" "Go-http-client/1.1"
time="2020-07-16T07:30:39.192445209Z" level=debug msg="authorizing request" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=ce22d938-3e7e-4164-b1c8-0d312306a6e0 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="10.19.123.165:7611" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
time="2020-07-16T07:30:39.192513529Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=ce22d938-3e7e-4164-b1c8-0d312306a6e0 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="10.19.123.165:7611" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
10.19.123.165 - - [16/Jul/2020:07:30:39 +0000] "GET /v2/ HTTP/1.1" 401 87 "http://harbor-harbor-registry:9188/v2" "Go-http-client/1.1"
172.30.5.3 - - [16/Jul/2020:07:30:47 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.17"
time="2020-07-16T07:30:47.219604256Z" level=info msg="response completed" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=736a16dc-64c3-4aff-98d3-c091919b9665 http.request.method=GET http.request.remoteaddr="172.30.5.2:30577" http.request.uri="/v2" http.request.useragent="Go-http-client/1.1" http.response.contenttype="text/html; charset=utf-8" http.response.duration="85.835µs" http.response.status=301 http.response.written=39
172.30.5.2 - - [16/Jul/2020:07:30:47 +0000] "GET /v2 HTTP/1.1" 301 39 "" "Go-http-client/1.1"
time="2020-07-16T07:30:47.221183139Z" level=debug msg="authorizing request" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=ccd84eed-8eee-4585-b861-e6fef8e95cc4 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="172.30.5.2:30577" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
time="2020-07-16T07:30:47.221269133Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=ccd84eed-8eee-4585-b861-e6fef8e95cc4 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="172.30.5.2:30577" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
172.30.5.2 - - [16/Jul/2020:07:30:47 +0000] "GET /v2/ HTTP/1.1" 401 87 "http://harbor-harbor-registry:9188/v2" "Go-http-client/1.1"
172.30.5.3 - - [16/Jul/2020:07:30:47 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.17"
time="2020-07-16T07:30:49.190353024Z" level=info msg="response completed" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=6e4083f2-529c-4c35-b07e-99afeb491aa7 http.request.method=GET http.request.remoteaddr="10.19.123.165:63600" http.request.uri="/v2" http.request.useragent="Go-http-client/1.1" http.response.contenttype="text/html; charset=utf-8" http.response.duration="108.978µs" http.response.status=301 http.response.written=39
10.19.123.165 - - [16/Jul/2020:07:30:49 +0000] "GET /v2 HTTP/1.1" 301 39 "" "Go-http-client/1.1"
time="2020-07-16T07:30:49.191510503Z" level=debug msg="authorizing request" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=c33522c3-2afb-4a8f-b422-e60b5aeae7af http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="10.19.123.165:63600" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
time="2020-07-16T07:30:49.191572656Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=c33522c3-2afb-4a8f-b422-e60b5aeae7af http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="10.19.123.165:63600" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
10.19.123.165 - - [16/Jul/2020:07:30:49 +0000] "GET /v2/ HTTP/1.1" 401 87 "http://harbor-harbor-registry:9188/v2" "Go-http-client/1.1"
172.30.5.3 - - [16/Jul/2020:07:30:57 +0000] "GET / HTTP/1.1" 200 0 "" "kube-probe/1.17"
time="2020-07-16T07:30:57.219890218Z" level=info msg="response completed" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=b463ef5e-7723-4a94-90e1-ca1163550553 http.request.method=GET http.request.remoteaddr="172.30.5.2:8799" http.request.uri="/v2" http.request.useragent="Go-http-client/1.1" http.response.contenttype="text/html; charset=utf-8" http.response.duration="178.483µs" http.response.status=301 http.response.written=39
172.30.5.2 - - [16/Jul/2020:07:30:57 +0000] "GET /v2 HTTP/1.1" 301 39 "" "Go-http-client/1.1"
time="2020-07-16T07:30:57.221293323Z" level=debug msg="authorizing request" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=a3964e76-656f-4b74-b1a7-18d72ece4d39 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="172.30.5.2:8799" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
time="2020-07-16T07:30:57.221359025Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.11.8 http.request.host="harbor-harbor-registry:9188" http.request.id=a3964e76-656f-4b74-b1a7-18d72ece4d39 http.request.method=GET http.request.referer="http://harbor-harbor-registry:9188/v2" http.request.remoteaddr="172.30.5.2:8799" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"
172.30.5.2 - - [16/Jul/2020:07:30:57 +0000] "GET /v2/ HTTP/1.1" 401 87 "http://harbor-harbor-registry:9188/v2" "Go-http-client/1.1"
|
harbor运行正常
4、harbor
文件
1
2
3
| cd /opt/hikcloud/harbor/registry/docker/registry/v2/repositories/docker-pbg/ydic/_layers/sha256
ll查看 失败layer存在
|
5、nginx-controller
查看nginx日志
1
2
3
4
5
6
7
8
9
10
11
12
13
14
| 2020/07/16 07:22:36 [error] 2043#2043: *197739729 upstream prematurely closed connection while reading upstream, client: 172.30.5.3, server: docker.hikcloud, request: "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1", upstream: "http://172.30.5.3:9185/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db", host: "docker.hikcloud:30001"
172.30.5.3 - - [16/Jul/2020:07:22:36 +0000] "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1" 200 0 "-" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" 1465 0.016 [kube-system-harbor-harbor-core-80] [] 172.30.5.3:9185 0 0.017 200 ce6fa0910582345c45ded8d5b25ae0d3
2020/07/16 07:22:45 [error] 2046#2046: *197739952 upstream prematurely closed connection while reading upstream, client: 172.30.5.3, server: docker.hikcloud, request: "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1", upstream: "http://172.30.5.2:9185/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db", host: "docker.hikcloud:30001"
172.30.5.3 - - [16/Jul/2020:07:22:45 +0000] "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1" 200 0 "-" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" 1465 0.015 [kube-system-harbor-harbor-core-80] [] 172.30.5.2:9185 0 0.014 200 54a5962ae803cfaf593c720b4d8f4171
2020/07/16 07:22:51 [error] 2049#2049: *197740104 upstream prematurely closed connection while reading upstream, client: 172.30.5.3, server: docker.hikcloud, request: "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1", upstream: "http://172.30.5.2:9185/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db", host: "docker.hikcloud:30001"
172.30.5.3 - - [16/Jul/2020:07:22:51 +0000] "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1" 200 0 "-" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" 1465 0.025 [kube-system-harbor-harbor-core-80] [] 172.30.5.2:9185 0 0.025 200 12d8f488258785ebf59bbae48881c3de
[root@worker1 ~]# kubectl logs --tail=100 -n kube-system harbor-harbor-registry-68c9db8d9c-m76x8 -c registry | grep 674
time="2020-07-16T07:23:11.349040402Z" level=debug msg="authorizing request" go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" vars.digest="sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" vars.name="docker-pbg/ydic"
time="2020-07-16T07:23:11.349875921Z" level=info msg="authorized request" go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" vars.digest="sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" vars.name="docker-pbg/ydic"
time="2020-07-16T07:23:11.349994691Z" level=debug msg=GetBlob auth.user.name= go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" vars.digest="sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" vars.name="docker-pbg/ydic"
time="2020-07-16T07:23:11.353393785Z" level=debug msg="filesystem.URLFor("/docker/registry/v2/blobs/sha256/67/674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db/data")" auth.user.name= go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" trace.duration=36.014µs trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).URLFor" trace.id=587d96eb-1aae-4d8d-8abc-cc0cc0c3d972 trace.line=217 vars.digest="sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" vars.name="docker-pbg/ydic"
time="2020-07-16T07:23:11.36010056Z" level=debug msg="filesystem.Reader("/docker/registry/v2/blobs/sha256/67/674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db/data", 0)" auth.user.name= go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" trace.duration=6.584606ms trace.file="/go/src/github.com/docker/distribution/registry/storage/driver/base/base.go" trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Reader" trace.id=bc92eae0-665a-42e5-8a2b-42a69ce3b03c trace.line=125 vars.digest="sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" vars.name="docker-pbg/ydic"
time="2020-07-16T07:23:11.361415935Z" level=info msg="response completed" go.version=go1.11.8 http.request.host="docker.hikcloud:30001" http.request.id=39de630c-307d-4ae0-9d98-d86ba76303e0 http.request.method=GET http.request.remoteaddr=172.30.5.2 http.request.uri="/v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db" http.request.useragent="docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" http.response.contenttype="application/octet-stream" http.response.duration=14.387271ms http.response.status=200 http.response.written=0
172.30.5.2 - - [16/Jul/2020:07:23:11 +0000] "GET /v2/docker-pbg/ydic/blobs/sha256:674a59b4e199228c78a5882ede113516d45d2eadac407f1828d03dda775875db HTTP/1.1" 200 0 "" "docker/19.03.4 go/go1.12.10 git-commit/9013bf583a kernel/3.10.0-1062.18.1.el7.x86_64 os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)"
|
upstream prematurely closed connection while reading upstream
错误原因：后端服务异常