记一次解vCenter 6.x WEB 503的故障

今天下午发现vCenter访问报503了,页面报

1
503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000fbb1883210] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

想到之前遇到过同样的问题,那次是服务器意外断电导致的,进命令行启动vpxd服务直接就可以了,但是这次好像有些区别,同时检查磁盘空间并没有占满。

观察到服务器HTTPS返回的证书中,到期日期已经是昨天了,感觉可能是ssl证书过期的问题。

上服务器上看到服务挂了一半:

1
2
3
4
5
PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --status
Running:
EsxAgentManager VMWareAfdService VMWareCertificateService MWareDirectoryService VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vPostgres vmon vmonapi vmware-cis-config vmware-license vmware-psc-client vsphere-ui vspherewebclientsvc
Stopped:
VMWareCAMService VMwareComponentManager VServiceManager content-library mbcs apiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health

观察到vpxdvpxd-svcs都已经down掉了,手动敲命令开启失败。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --start vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2021-12-13T11:18:51.550Z Service vpxd-svcs state STOPPED
Error executing start on service vpxd-svcs. Details {
"resolution": null,
"detail": [
{
"args": [
"vpxd-svcs"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vpxd-svcs'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"vpxd-svcs"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vpxd-svcs'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}

同时使用命令查看所有证书均在2021/12/12日过期:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
PS C:\> $VCInstallHome = [System.Environment]::ExpandEnvironmentVariables("%VMWARE_CIS_HOME%");foreach ($STORE in & "$VCInstallHome\vmafdd\vecs-cli" store list){Write-host STORE: $STORE;& $VCInstallHome\vmafdd\vecs-cli" entry list --store $STORE --text | findstr /C:"Alias" /C:"Not After"}

STORE: MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : Dec 12 05:36:33 2021 GMT
STORE: TRUSTED_ROOTS
Alias : d586b1b----------f72a3747----------603c2
Not After : Dec 6 17:36:32 2029 GMT
STORE: TRUSTED_ROOT_CRLS
Alias : 03ad6cd----------f6e18e6f----------32f72
STORE: machine
Alias : machine
Not After : Dec 11 17:27:58 2021 GMT
STORE: vsphere-webclient
Alias : vsphere-webclient
Not After : Dec 11 17:28:00 2021 GMT
STORE: vpxd
Alias : vpxd
Not After : Dec 11 17:28:01 2021 GMT
STORE: vpxd-extension
Alias : vpxd-extension
Not After : Dec 11 17:28:03 2021 GMT
STORE: SMS
Alias : sms_self_signed
Not After : Dec 12 17:43:12 2029 GMT
STORE: BACKUP_STORE
Alias : bkp___MACHINE_CERT
Not After : Dec 12 05:36:33 2021 GMT
Alias : bkp_machine
Not After : Dec 11 17:27:58 2021 GMT
Alias : bkp_vsphere-webclient
Not After : Dec 11 17:28:00 2021 GMT
Alias : bkp_vpxd
Not After : Dec 11 17:28:01 2021 GMT
Alias : bkp_vpxd-extension
Not After : Dec 11 17:28:03 2021 GMT

原本想仅续签证书,但发现根ca也过期了,遂执行所有证书全部重新注册操作:

注意:重新签发所有证书会导致解决方案包需要重新注册

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
PS C:\Program Files\VMware\vCenter Server\vmcad> .\certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 6.5 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : y

Press Enter key to skip optional parameters or use Previous value.

Enter proper value for 'Country' [Previous value : US] :

Enter proper value for 'Name' [Previous value : CA] :

Enter proper value for 'Organization' [Previous value : VMware] :

Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] :

Enter proper value for 'State' [Previous value : California] :

Enter proper value for 'Locality' [Previous value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 10.-.-.---

Enter proper value for 'Email' [Previous value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qual
ified Domain Name(FQDN), For Example : example.domain.com] : 10.-.-.---

Enter proper value for VMCA 'Name' :CA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Reset Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:9c6d---------------------------4cc76
Update service default-first-site:9c6d---------------------------4cc76; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ovwy7o
Get service default-first-site:5a5f---------------------------84beb
Update service default-first-site:5a5f---------------------------84beb; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ylk5yr
Get service default-first-site:083e---------------------------f3d01
Update service default-first-site:083e---------------------------f3d01; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_fsnv6_
Get service 76a5---------------------------c12b4
Update service 76a5---------------------------c12b4; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_yn5fml
Get service 68e8---------------------------83c0b
Update service 68e8---------------------------83c0b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_oysooz
Get service 6308---------------------------c6df1
Update service 6308---------------------------c6df1; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_quehgr
Get service d116---------------------------c5db8
Update service d116---------------------------c5db8; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vqxfvo
Get service 67f6---------------------------f72de
Update service 67f6---------------------------f72de; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_x0zbvu
Get service f648---------------------------3cec2
Update service f648---------------------------3cec2; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hspktf
Get service 8d3a---------------------------130e5
Update service 8d3a---------------------------130e5; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hzviao
Get service 5d75---------------------------5ec4b
Update service 5d75---------------------------5ec4b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_cqua3q
Get service 53ca---------------------------0dfb3
Update service 53ca---------------------------0dfb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9_c9pf
Get service 8891---------------------------c9003
Update service 8891---------------------------c9003; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_up4pcf
Get service c9ca---------------------------7d3dd
Update service c9ca---------------------------7d3dd; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9cyuzx
Get service d3ef---------------------------a1fdf
Update service d3ef---------------------------a1fdf; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hv6aa6
Get service 56e9---------------------------a9693
Update service 56e9---------------------------a9693; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ql6qpn
Get service a83b---------------------------09eea
Update service a83b---------------------------09eea; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_jjncqp
Get service 1975---------------------------65177
Update service 1975---------------------------65177; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ualmfy
Get service c835b---------------------------59bb3_kv
Update service c835b---------------------------59bb3_kv; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hwrqpu
Get service 12b7---------------------------68e3a
Update service 12b7---------------------------68e3a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ubyrj4
Get service 4ca6---------------------------fed53
Update service 4ca6---------------------------fed53; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vd5us4
Get service fba5---------------------------653e6
Update service fba5---------------------------653e6; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_m70tvt
Get service 8704---------------------------fa638
Update service 8704---------------------------fa638; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ygr5ee
Get service bd6f---------------------------abf9a
Update service bd6f---------------------------abf9a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec__6rsae
Get service c835b---------------------------9bb3_authz
Update service c835b---------------------------9bb3_authz; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_rw31ym
Get service c835b---------------------------9bb3
Update service c835b---------------------------9bb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_plshh5
Updated 26 service(s)
Status : 60% Completed [Reset vpxd-extension Cert...]
2021-12-13T11:46:20.423Z Updating certificate for "com.vmware.vim.eam" extension


2021-12-13T11:46:23.173Z Updating certificate for "com.vmware.rbd" extension

Reset status : 100% Completed [Reset completed successfully]

此时访问页面已经全部恢复正常。

问题解决

2021/12/13

  • 注:日志中所有信息已抹去关键字段,请根据实际情况填写

参考文献:

https://kb.vmware.com/s/article/82332?lang=zh_cn

https://kb.vmware.com/s/article/2112277?lang=zh_cn