Python - Connecting to AWS Neptune

Question

I have created a neptune instance in aws. How can I connect to it now?

I tried the the example given in the documentation locally from my laptop.

from gremlin_python.structure.graph import Graph
from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection

graph = Graph()

g = graph.traversal().withRemote(DriverRemoteConnection('ws://my_endpoint:8182/gremlin','g'))

print(g.V().limit(2).toList())

But I get Timeout exception with the following stacktrace

File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/driver_remote_connection.py", line 45, in __init__
    password=password)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 76, in __init__
    self._fill_pool()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 88, in _fill_pool
    conn = self._get_connection()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/client.py", line 101, in _get_connection
    self._transport_factory, self._executor, self._pool)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/connection.py", line 40, in __init__
    self.connect()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/connection.py", line 46, in connect
    self._transport.connect(self._url)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/gremlin_python/driver/tornado/transport.py", line 33, in connect
    lambda: websocket.websocket_connect(url))
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/ioloop.py", line 458, in run_sync
    return future_cell[0].result()
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/stack_context.py", line 316, in wrapped
    ret = fn(*args, **kwargs)
  File "/home/cegprakash/.virtualenvs/cegprakash-6Wq6Rd61/lib/python3.5/site-packages/tornado/simple_httpclient.py", line 307, in _on_timeout
    raise HTTPError(599, error_message)
tornado.httpclient.HTTPError: HTTP 599: Timeout while connecting

Is there any authentication that I'm missing for the DB to get connected?

Answer 1

Connectivity problems are generally attributed to some issue with your security group settings. This was already answered in another question [1]. Posting the response here, in case it helps.

---

If you are seeing timeouts while connecting to the database, the first step would be to check if you have network connectivity to the endpoint.

Try: telnet endpoint port

If you have connectivity, you would see something like this:

Trying 172.217.5.110...
Connected to endpoint (172.217.5.110).
Escape character is '^]'

If this does work, then any HTTP client should be able to connect to your database. (CURL, POSTMAN etc)

If telnet does not work, then it is almost certain that you have not configured your EC2 Security Groups correctly. The gist of what you need to do is:

1. Create a security Group (say 'ec2') and attach that to your EC2 client instance. By default, this security group should allow outbound connections to all IPs. If that is not the case, add it.

2. Create a security Group (say 'db'). In Inbound rules, add a rule that allows inbound TCP connections to your database port, and source as the security group created in #1.

3. Now modify your Neptune Cluster, and attach 'db' to it.

4. Security Group changes propagate pretty fast, so you should be able to test this using telnet.

You may find other answers that say that you need the database and the EC2 instance to be in the same security group. That is not entirely true, it is just a special case of the steps mentioned above where instead of creating 2 security groups, you can use a single security group for both - db and the client instance. From a security and design perspective, its best if you have separate security groups for your DB and your client instances.

Hope this helps.

[1] https://stackoverflow.com/a/51940587/3069919

Answer 2

AWS Neptune is only accessible from an EC2 instance (or something similar) running in the VPC in which you set up your cluster.

https://docs.aws.amazon.com/neptune/latest/userguide/security-vpc.html

If this proves to be a barrier, you can rapidly prototype using AWS Lambda, which allows access to Neptune via this tutorial.

https://docs.aws.amazon.com/neptune/latest/userguide/get-started-cfn-lambda.html