Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Can I use abstract methods to import file-specific formatting of (Python) pandas data?

I have a class FileSet with a method _process_series, which contains a bunch of if-elif blocks doing filetag-specific formatting of different pandas.Series:

    elif filetag == "EntityA":
        ps[filetag+"_Id"] = str(ps[filetag+"_Id"]).strip()
        ps[filetag+"_DateOfBirth"] = str(pd.to_datetime(ps[filetag+"_DateOfBirth"]).strftime('%Y-%m-%d')).strip()
        ps[filetag+"_FirstName"] = str(ps[filetag+"_FirstName"]).strip().capitalize()
        ps[filetag+"_LastName"] = str(ps[filetag+"_LastName"]).strip().capitalize()
        ps[filetag+"_Age"] = relativedelta(datetime.today(), datetime.strptime(ps[filetag+"_DateOfBirth"], "%Y-%m-%d")).years
        return ps

I'd like to define an abstract format method in the class and keep these blocks of formatting in separate modules that are imported when _process_series is called for a given filetag. Forgive the pseudo-code, but something like:

for tag in filetag:
    from my_formatters import tag+'_formatter' as fmt
    ps = self.format(pandas_series, fmt)
    return ps

And the module would contain the formatting block:

# my_formatters.EntityA_formatter
    ps[filetag+"_Id"] = str(ps[filetag+"_Id"]).strip()
    ps[filetag+"_DateOfBirth"] = str(pd.to_datetime(ps[filetag+"_DateOfBirth"]).strftime('%Y-%m-%d')).strip()
    ps[filetag+"_FirstName"] = str(ps[filetag+"_FirstName"]).strip().capitalize()
    ps[filetag+"_LastName"] = str(ps[filetag+"_LastName"]).strip().capitalize()
    ps[filetag+"_Age"] = relativedelta(datetime.today(), datetime.strptime(ps[filetag+"_DateOfBirth"], "%Y-%m-%d")).years
    return ps
like image 767
d8aninja Avatar asked Aug 09 '21 00:08

d8aninja


People also ask

Which of the following file formats does Pandas support for data importing?

Plain Text (txt) JSON. XML. HTML.

What is abstraction in Python?

In Python, Abstraction works by incorporating abstract classes and methods. Abstract Class: A class specified in the code that has abstract methods is named Abstract Class. Abstract Method: Here, it doesn’t have any implementation. All the implementations are done inside the sub-classes.

How to import a CSV file into Python using PANDAS?

Steps to import a CSV file into Python using pandas. File name (as highlighted in green). You may choose a different file name, but make sure that the file name specified in the code matches with the actual file name File extension (as highlighted in blue). The file extension should always be ‘.csv’ when importing CSV files (2)...

Why do we use abstract classes as templates in Python?

If python allows creating an object for abstract classes then using that object if anyone calls the abstract method, but there is no actual implementation to invoke. So we use an abstract class as a template and according to the need, we extend it and build on it before we can use it.

How to import class from file1 dynamically in Python?

We can use the following code in file2.py to import the class from file1 dynamically. How to import class dynamically? We can import class dynamically using __import__ magic function or imp module.


Video Answer


2 Answers

You can create a function in it's own .py file and import it. If you create the same function in each file you can then call it.

here is f1.py:

def gimme():
    return 'format 1'

here is f2.py:

def gimme():
    return 'format 2'

Then you main file:

module_names = ['f1','f2']

for module_name in module_names:
    import_test = __import__(module_name)
    result = import_test.gimme()
    result = import_test.gimme()
    print(result)

Which gives the output:

format 1
format 2
like image 175
Brian Z Avatar answered Oct 21 '22 10:10

Brian Z


Why not use globals with asterisk:

from my_formatters import *

for tag in filetag:
    fmt = globals()[tag + '_formatter']
    ps = self.format(pandas_series, fmt)
    return ps

I converted your pseudocode to real code.

globals documentation:

Return a dictionary representing the current global symbol table. This is always the dictionary of the current module (inside a function or method, this is the module where it is defined, not the module from which it is called).

like image 22
U12-Forward Avatar answered Oct 21 '22 12:10

U12-Forward