Can I programmatically log into a website without storing the password in plaintext?

Question

I do a number of projects that involve automatically submitting forms and/or retrieving data from websites. Some of these sites require username/password authentication. (These sites do not have APIs, so I am relying on screen scraping.)

Most of the tutorials I've seen store the username and password in the source code like any other POST data, e.g.:

string username = "someUserName";
string password = "somePassword";
// submit POST data...

But I know storing passwords in plain text is generally frowned upon. Is there an alternative method I should use?

Answer 1

The common way of storing a password is by hashing it. As most algorithms for hashing passwords are destructive, that is they can't be reversed, this wouldn't work for you.

An option would be to use a reversible hash, such as to base64 encode the password, but it isn't really a lot safer than storing it in plain text.

The best solution as far as I can see, would be to store the passwords in a database. If you are really worried about someone getting the usernames and passwords, you could encrypt them in the DB with encryption functions, or you could use a SQLite database which you would encrypt directly on the disk.

This way your code and login credentials are separated, and you can safely share your code with others without worrying about security.

Answer 2

A pattern we use is:

In your database table you have an encrypted column. This column contains data encrypted with a system-wide, long (128 bit), random secret key (usually stored in a configuration file). The data in this encrypted column contains a separate (random) secret key used for each thirdparty service. With this password we encrypt the authentication details related to this thirdparty service.

Why this double encrypting?

You reduce the amount of passwords in plain text to a single one (the system-wide password). Because of this, key management is easier. We create a long random secret key for each thirdparty service so that we can selectively decrypt the credentials for each thirdparty service and transfer them between systems if necessary. Having one of our secret keys stored outside the database also reduces the risk associated with both SQL-injection attacks (they 'only' get the database data) and with backups (configuration files are not included in the regular backup data).

The weakness is obviously the system-wide password. It needs to be in memory somewhere.

I'm no cryptographer, and I'm pretty sure the above is sub-optimal. However, it works, is manageable and lot safer than just storing the thirdparty service credentials in plain text.

Answer 3

I have a scraping project that needed this problem solved. My setup includes two separate servers. The first is the user front end web app. the second is a nodejs server that handles the scraping.

I handle encryption with openssl key pair encryption. I generate a key pair for the nodejs machine and give the public key to the front end web app. When a user registers their 3rd party credentials those credentials are encrypted with the public key and stored in a database.

The web app regularly selects a user's encrypted credentials and sends them to the node server where they are decrypted with the private key and used with the 3rd party for scraping.

After a quick search I found this article about using openssl and encrypting strings.

I realize this is a very old post but hopefully it helps the next person that stumbles onto this problem.