rendered paste bodyJust try a simple nova list. It works at times, and then fails now and then randomly. The behavior is random and really strange.
I have enabled HTTPS on horizon dashboard, and it seems to made this very big issue. Liberty/CentOS7
nova list
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------+--------+------------+-------------+-----------------------+
| d025b105-33c7-4f7e-a560-c0cf58acccfd | myinstance | ACTIVE | - | Running | ext-net=192.168.2.185 |
| fbf3d852-37cd-4486-8afb-35fcd92605f1 | myinstance2 | ACTIVE | - | Running | ext-net=192.168.2.186 |
| 1b24edc9-1b78-4992-8a87-bf5389f377c8 | myinstance3 | ACTIVE | - | Running | ext-net=192.168.2.187 |
+--------------------------------------+-------------+--------+------------+-------------+------------------------
# NOTE HERE IT GIVES 503
nova list
ERROR (ClientException): Unknown Error (HTTP 503) (Request-ID: req-7a5de02a-be7b-410e-9bb0-a6a178121483)
# AND HERE 500
nova list
ERROR (InternalServerError): An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-2c978de4-3d13-4b10-aae4-a27c32a9cf83)
nova --debug list
DEBUG (session:198) REQ: curl -g -i -X GET http://maccess:35357/v3 -H "Accept: application/json" -H "User-Agent: python-keystoneclient"
INFO (connectionpool:213) Starting new HTTP connection (1): maccess
DEBUG (connectionpool:395) "GET /v3 HTTP/1.1" 200 247
DEBUG (session:215) RESP: [200] Date: Wed, 06 Jul 2016 09:26:06 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token x-openstack-request-id: req-eb553805-2a25-4a8b-9fd4-206e1646fa79 Content-Length: 247 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: application/json
RESP BODY: {"version": {"status": "stable", "updated": "2015-03-30T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.4", "links": [{"href": "http://maccess:35357/v3/", "rel": "self"}]}}
DEBUG (base:188) Making authentication request to http://maccess:35357/v3/auth/tokens
DEBUG (connectionpool:395) "POST /v3/auth/tokens HTTP/1.1" 500 143
DEBUG (session:400) Request returned failure status: 500
DEBUG (shell:905) An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-115e277e-bc20-4804-913d-959dcd0f471a)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 903, in main
OpenStackComputeShell().main(argv)
File "/usr/lib/python2.7/site-packages/novaclient/shell.py", line 737, in main
api_version = api_versions.discover_version(self.cs, api_version)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 253, in discover_version
client)
File "/usr/lib/python2.7/site-packages/novaclient/api_versions.py", line 235, in _get_server_version_range
version = client.versions.get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 62, in get_current
return self._get_current()
File "/usr/lib/python2.7/site-packages/novaclient/v2/versions.py", line 43, in _get_current
url = self.api.client.get_endpoint().rsplit("/", 1)[0]
File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 129, in get_endpoint
return self.session.get_endpoint(auth or self.auth, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 660, in get_endpoint
return auth.get_endpoint(self, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 315, in get_endpoint
service_catalog = self.get_access(session).service_catalog
File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
self.auth_ref = self.get_auth_ref(session)
File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref
return self._plugin.get_auth_ref(session, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
authenticated=False, log=False, **rkwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
return self.request(url, 'POST', **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
raise exceptions.from_response(resp, method, url)
InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-115e277e-bc20-4804-913d-959dcd0f471a)
ERROR (InternalServerError): An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-115e277e-bc20-4804-913d-959dcd0f471a)
ENDPOINTS
openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------+
| 157eff41f9a84b6e9f32098bfc928f2a | RegionOne | neutron | network | True | public | http://maccess:9696 |
| 16d9aac4397f404b83a928070e6edff2 | RegionOne | keystone | identity | True | internal | http://maccess:5000/v2.0 |
| 225ff697043a4017af6def34147b66b5 | RegionOne | glance | image | True | public | http://maccess:9292 |
| 26c3d8e9b33143dc8d055df57a2ec18d | RegionOne | nova | compute | True | admin | http://maccess:8774/v2/%(tenant_id)s |
| 2fddb74d31164630975c0c7417cea872 | RegionOne | keystone | identity | True | public | http://maccess:5000/v2.0 |
| 3d0d0cc1ead442a5a1bc02a817d77fdf | RegionOne | cinder | volume | True | public | http://maccess:8776/v1/%(tenant_id)s |
| 4255ecd52a4442b8a2c358e07dd2c84b | RegionOne | glance | image | True | internal | http://maccess:9292 |
| 4ed9606cd0d945f39945de4da063ece5 | RegionOne | keystone | identity | True | admin | http://maccess:35357/v2.0 |
| 5388e157b29f49c5bec91853d0e606d8 | RegionOne | nova | compute | True | internal | http://maccess:8774/v2/%(tenant_id)s |
| 5e5e60beeb354498b73c38f455dd2d5b | RegionOne | cinder | volume | True | admin | http://maccess:8776/v1/%(tenant_id)s |
| 6095cc4c40554bf9a34c9c81b04482c0 | RegionOne | glance | image | True | admin | http://maccess:9292 |
| 72b46318c5f046b5a3fd18f6eb5ff1a9 | RegionOne | cinderv2 | volumev2 | True | internal | http://maccess:8776/v2/%(tenant_id)s |
| 78f14f9a7a024d6989e0bec13b18ec65 | RegionOne | nova | compute | True | public | http://maccess:8774/v2/%(tenant_id)s |
| 98b28a0a2ef74f75a6b708459b80e8b4 | RegionOne | cinder | volume | True | internal | http://maccess:8776/v1/%(tenant_id)s |
| 9ec6d236a0094d03a5ff06c864d1e637 | RegionOne | cinderv2 | volumev2 | True | admin | http://maccess:8776/v2/%(tenant_id)s |
| 9f7ed766db7044bf925660d0dad0e60e | RegionOne | neutron | network | True | internal | http://maccess:9696 |
| a598ed316aa742b78aec603123eafffa | RegionOne | neutron | network | True | admin | http://maccess:9696 |
| c811ff05a3dd4d449ff5da3ae0b530f0 | RegionOne | cinderv2 | volumev2 | True | public | http://maccess:8776/v2/%(tenant_id)s |
+----------------------------------+-----------+--------------+--------------+---------+-----------+--------------------------------------+