记一次解vCenter 6.x WEB 503的故障

今天下午发现vCenter访问报503了，页面报

1
503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x000000fbb1883210] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

想到之前遇到过同样的问题，那次是服务器意外断电导致的，进命令行启动vpxd服务直接就可以了，但是这次好像有些区别，同时检查磁盘空间并没有占满。

观察到服务器HTTPS返回的证书中，到期日期已经是昨天了，感觉可能是ssl证书过期的问题。

上服务器上看到服务挂了一半：

1
2
3
4
5
PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --status
Running:
 EsxAgentManager VMWareAfdService VMWareCertificateService MWareDirectoryService VMwareDNSService VMwareIdentityMgmtService VMwareSTS rhttpproxy vPostgres vmon vmonapi vmware-cis-config vmware-license vmware-psc-client vsphere-ui vspherewebclientsvc
Stopped:
 VMWareCAMService VMwareComponentManager VServiceManager content-library mbcs apiEndpoint vimPBSM vmsyslogcollector vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vmware-perfcharts vmwareServiceControlAgent vpxd vpxd-svcs vsan-health

观察到vpxd和vpxd-svcs都已经down掉了，手动敲命令开启失败。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
PS C:\Program Files\VMware\vCenter Server\bin> .\service-control.bat --start vpxd-svcs
Perform start operation. vmon_profile=None, svc_names=['vpxd-svcs'], include_coreossvcs=False, include_leafossvcs=False
2021-12-13T11:18:51.550Z   Service vpxd-svcs state STOPPED
Error executing start on service vpxd-svcs. Details {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd-svcs"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd-svcs'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Service-control failed. Error {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vpxd-svcs"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vpxd-svcs'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

同时使用命令查看所有证书均在2021/12/12日过期：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
PS C:\> $VCInstallHome = [System.Environment]::ExpandEnvironmentVariables("%VMWARE_CIS_HOME%");foreach ($STORE in & "$VCInstallHome\vmafdd\vecs-cli" store list){Write-host STORE: $STORE;& $VCInstallHome\vmafdd\vecs-cli" entry list --store $STORE --text | findstr /C:"Alias" /C:"Not After"}

STORE: MACHINE_SSL_CERT
Alias : __MACHINE_CERT
            Not After : Dec 12 05:36:33 2021 GMT
STORE: TRUSTED_ROOTS
Alias : d586b1b----------f72a3747----------603c2
            Not After : Dec  6 17:36:32 2029 GMT
STORE: TRUSTED_ROOT_CRLS
Alias : 03ad6cd----------f6e18e6f----------32f72
STORE: machine
Alias : machine
            Not After : Dec 11 17:27:58 2021 GMT
STORE: vsphere-webclient
Alias : vsphere-webclient
            Not After : Dec 11 17:28:00 2021 GMT
STORE: vpxd
Alias : vpxd
            Not After : Dec 11 17:28:01 2021 GMT
STORE: vpxd-extension
Alias : vpxd-extension
            Not After : Dec 11 17:28:03 2021 GMT
STORE: SMS
Alias : sms_self_signed
            Not After : Dec 12 17:43:12 2029 GMT
STORE: BACKUP_STORE
Alias : bkp___MACHINE_CERT
            Not After : Dec 12 05:36:33 2021 GMT
Alias : bkp_machine
            Not After : Dec 11 17:27:58 2021 GMT
Alias : bkp_vsphere-webclient
            Not After : Dec 11 17:28:00 2021 GMT
Alias : bkp_vpxd
            Not After : Dec 11 17:28:01 2021 GMT
Alias : bkp_vpxd-extension
            Not After : Dec 11 17:28:03 2021 GMT

原本想仅续签证书，但发现根ca也过期了，遂执行所有证书全部重新注册操作：

注意：重新签发所有证书会导致解决方案包需要重新注册

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
PS C:\Program Files\VMware\vCenter Server\vmcad> .\certificate-manager
                 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
                |                                                                     |
                |      *** Welcome to the vSphere 6.5 Certificate Manager  ***        |
                |                                                                     |
                |                   -- Select Operation --                            |
                |                                                                     |
                |      1. Replace Machine SSL certificate with Custom Certificate     |
                |                                                                     |
                |      2. Replace VMCA Root certificate with Custom Signing           |
                |         Certificate and replace all Certificates                    |
                |                                                                     |
                |      3. Replace Machine SSL certificate with VMCA Certificate       |
                |                                                                     |
                |      4. Regenerate a new VMCA Root Certificate and                  |
                |         replace all certificates                                    |
                |                                                                     |
                |      5. Replace Solution user certificates with                     |
                |         Custom Certificate                                          |
                |                                                                     |
                |      6. Replace Solution user certificates with VMCA certificates   |
                |                                                                     |
                |      7. Revert last performed operation by re-publishing old        |
                |         certificates                                                |
                |                                                                     |
                |      8. Reset all Certificates                                      |
                |_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-Z and hit Enter to exit.
Option[1 to 8]: 8
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : y

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : y

Press Enter key to skip optional parameters or use Previous value.

Enter proper value for 'Country' [Previous value : US] :

Enter proper value for 'Name' [Previous value : CA] :

Enter proper value for 'Organization' [Previous value : VMware] :

Enter proper value for 'OrgUnit' [Previous value : VMware Engineering] :

Enter proper value for 'State' [Previous value : California] :

Enter proper value for 'Locality' [Previous value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 10.-.-.---

Enter proper value for 'Email' [Previous value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qual
ified Domain Name(FQDN), For Example : example.domain.com] : 10.-.-.---

Enter proper value for VMCA 'Name' :CA
Continue operation : Option[Y/N] ? : y

You are going to reset by regenerating Root Certificate and replace all certificates using VMCA
Continue operation : Option[Y/N] ? : y
Get site nameCompleted [Reset Machine SSL Cert...]
default-first-site
Lookup all services
Get service default-first-site:9c6d---------------------------4cc76
Update service default-first-site:9c6d---------------------------4cc76; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ovwy7o
Get service default-first-site:5a5f---------------------------84beb
Update service default-first-site:5a5f---------------------------84beb; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ylk5yr
Get service default-first-site:083e---------------------------f3d01
Update service default-first-site:083e---------------------------f3d01; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_fsnv6_
Get service 76a5---------------------------c12b4
Update service 76a5---------------------------c12b4; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_yn5fml
Get service 68e8---------------------------83c0b
Update service 68e8---------------------------83c0b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_oysooz
Get service 6308---------------------------c6df1
Update service 6308---------------------------c6df1; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_quehgr
Get service d116---------------------------c5db8
Update service d116---------------------------c5db8; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vqxfvo
Get service 67f6---------------------------f72de
Update service 67f6---------------------------f72de; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_x0zbvu
Get service f648---------------------------3cec2
Update service f648---------------------------3cec2; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hspktf
Get service 8d3a---------------------------130e5
Update service 8d3a---------------------------130e5; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hzviao
Get service 5d75---------------------------5ec4b
Update service 5d75---------------------------5ec4b; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_cqua3q
Get service 53ca---------------------------0dfb3
Update service 53ca---------------------------0dfb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9_c9pf
Get service 8891---------------------------c9003
Update service 8891---------------------------c9003; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_up4pcf
Get service c9ca---------------------------7d3dd
Update service c9ca---------------------------7d3dd; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_9cyuzx
Get service d3ef---------------------------a1fdf
Update service d3ef---------------------------a1fdf; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hv6aa6
Get service 56e9---------------------------a9693
Update service 56e9---------------------------a9693; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ql6qpn
Get service a83b---------------------------09eea
Update service a83b---------------------------09eea; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_jjncqp
Get service 1975---------------------------65177
Update service 1975---------------------------65177; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ualmfy
Get service c835b---------------------------59bb3_kv
Update service c835b---------------------------59bb3_kv; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_hwrqpu
Get service 12b7---------------------------68e3a
Update service 12b7---------------------------68e3a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ubyrj4
Get service 4ca6---------------------------fed53
Update service 4ca6---------------------------fed53; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_vd5us4
Get service fba5---------------------------653e6
Update service fba5---------------------------653e6; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_m70tvt
Get service 8704---------------------------fa638
Update service 8704---------------------------fa638; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_ygr5ee
Get service bd6f---------------------------abf9a
Update service bd6f---------------------------abf9a; spec: c:\users\admini~1\appdata\local\temp\2\svcspec__6rsae
Get service c835b---------------------------9bb3_authz
Update service c835b---------------------------9bb3_authz; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_rw31ym
Get service c835b---------------------------9bb3
Update service c835b---------------------------9bb3; spec: c:\users\admini~1\appdata\local\temp\2\svcspec_plshh5
Updated 26 service(s)
Status : 60% Completed [Reset vpxd-extension Cert...]
2021-12-13T11:46:20.423Z   Updating certificate for "com.vmware.vim.eam" extension


2021-12-13T11:46:23.173Z   Updating certificate for "com.vmware.rbd" extension

Reset status : 100% Completed [Reset completed successfully]

此时访问页面已经全部恢复正常。

问题解决

2021/12/13

  • 注：日志中所有信息已抹去关键字段，请根据实际情况填写

参考文献：

https://kb.vmware.com/s/article/82332?lang=zh_cn

https://kb.vmware.com/s/article/2112277?lang=zh_cn