Importing all variables from a file into a list

Question

Is there a way to import all variables from a file into a list?

Simplifying, this is precisely what I'm trying to achieve:

• I have a config.py file containing some variables like this, which strictly contains variables only:

  # File: config.py
  
  NOSCRIPT = True
  TRANSFER_PB = False
  TIMEOUT = 7
  ...
  

• Now I have to import all of these variables from config.py file into process.py, so that I can get something similar to this when I run process.py

  # File: process.py
  
  from config import *
  # Stuff to convert all imports into a list
  x = [NOSCRIPT, TRANSFER_PB, TIMEOUT, ...]
  

I know using wildcards are not really a great practice but I am left with no other option now due to the huge number of variables inside config.py (approx. 500+). Also I cannot edit the config file anyway. I'm probably missing something obvious, however it would be great if someone can help me out with this.

Answer 1

from config import * performs the following steps:

1. It imports config (without binding it to that name),
2. It checks whether config.__all__ exists,
3. If it exists, use those names to import objects,
4. Otherwise, import everything that doesn't start with an underscore.

So you can do the following:

import config

if hasattr(config, '__all__'):
    x = [getattr(config, name) for name in config.__all__]
else:
    x = [getattr(config, name) for name in dir(config) if not name.startswith('_')]

This is only approximately equivalent though since if config.py defines __dir__ then this is used for the result of dir(config) while from config import * still imports every name that doesn't start with an underscore, irregardless of __dir__.

Note: The result of dir(config) is alphabetized, i.e. dir(config) == sorted(dir(config)). That means the elements of the list do not necessarily appear in the order that they were defined in the original config.py file. Since dictionaries preserve insertion order since Python 3.7 you can use for name in config.__dict__ instead, if that's important (this also sidesteps the __dir__ difference).

Answer 2

You could easily read them as CSV where = would be the separator. Here's a csv reader function that additionnaly deals with # comments, and provides a generator.

import csv


def csv_dict_reader(file, has_header=False, skip_comment_char=None, **kwargs):
    """
    Reads CSV file into memory


    :param file: (str) path to csv file to read
    :param has_header: (bool) skip first line
    :param skip_comment_char: (str) optional character which, if found on first row, will skip row
    :param delimiter: (char) CSV delimiter char
    :param fieldnames: (list) CSV field names for dictionnary creation
    :param kwargs:
    :return: csv object that can be iterated
    """
    with open(file) as fp:
        csv_data = csv.DictReader(fp, **kwargs)
        # Skip header
        if has_header:
            next(csv_data)

        fieldnames = kwargs.get('fieldnames')
        for row in csv_data:
            # Skip commented out entries
            if fieldnames is not None:
                if skip_comment_char is not None:
                    if not row[fieldnames[0]].startswith(skip_comment_char):
                        yield row
                else:
                    yield row
            else:
                # list(row)[0] is key from row, works with Python 3.7+
                if skip_comment_char is not None:
                    if not row[list(row)[0]].startswith(skip_comment_char):
                        yield row
                else:
                    yield row

You can than use that function to read your config.py file, which you can convert to a dict like:

conf = csv_dict_reader('config.py', skip_comment_char='#', delimiter='=', fieldnames=['name', 'value'])

my_dict = {}

for line in conf:
    my_dict [line['name']] = line['value']

print(my_dict)

You may have to add .strip() to the names and values in order to remove trailing / ending white spaces.

Bear in mind that reading a python config file this way isn't very elegant. You might consider using a 'real' config file format, with ConfigParser or so.

[EDIT] Just read your commment that you cannot change the config file format. So at least this should work[/EDIT]