WSO2 API Manager Store - Access token key generation failure

Question

I have setup an API Manager 1.10 cluster, with each component in a separate VM, fronted by an nginx proxy.

Whenever I try to generate a key I get the following error in store (logs from store):

ERROR - AMDefaultKeyManagerImpl Error while creating tokens - Connection timed out
java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
   ...
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

[2016-12-14 16:25:03,376] ERROR - APIConsumerImpl Error while re-generating AccessToken
org.wso2.carbon.apimgt.api.APIManagementException: Error while creating tokens - Connection timed out
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.handleException(AMDefaultKeyManagerImpl.java:592)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:369)
    at org.wso2.carbon.apimgt.impl.APIConsumerImpl.renewAccessToken(APIConsumerImpl.java:659)
    at org.wso2.carbon.apimgt.impl.UserAwareAPIConsumer.renewAccessToken(UserAwareAPIConsumer.java:36)
    at org.wso2.carbon.apimgt.hostobjects.APIStoreHostObject.jsFunction_refreshToken(APIStoreHostObject.java:3800)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    .....
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
    at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:683)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection timed out
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:522)
    at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:401)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:178)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:610)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:445)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
    at org.wso2.carbon.apimgt.impl.AMDefaultKeyManagerImpl.getNewApplicationAccessToken(AMDefaultKeyManagerImpl.java:303)
    ... 70 more
[2016-12-14 16:25:03,377] ERROR - APIStoreHostObject Error while renewing Access Token for Consumer Key qZf8VvT4ts7a0TT6nE6ZHfY9t5sa and user admin
[2016-12-14 16:25:03,377] ERROR - subscription-add:jag org.wso2.carbon.apimgt.api.APIManagementException: Error while renewing Access Token for Consumer Key 88 and user admin

I have configured connections among the components by following this article The logs above are only showing up in the API store console.

I have also configured the Keymanager host and port in API Gateway in the following locations:

/repository/deployment/server/synapse-configs/default/api
AuthorizeAPI.xml
RevokeAPI.xml
TokenAPI.xml

What do I need to do to fix this?

Answer 1

In store's api-manager.xml, <RevokeAPIURL> should be pointing to gateway. make sure you have that configured properly.

Answer 2

I didn't have the port 8243 opened in my security group for the API gateway. So the problem had nothing to do with WSO2, it was my infrastructure.