I have a number of very large text files which I need to process, the largest being about 60GB.
Each line has 54 characters in seven fields and I want to remove the last three characters from each of the first three fields - which should reduce the file size by about 20%.
I am brand new to Python and have a code which will do what I want to do at about 3.4 GB per hour, but to be a worthwhile exercise I really need to be getting at least 10 GB/hr - is there any way to speed this up? This code doesn't come close to challenging my processor, so I am making an uneducated guess that it is limited by the read and write speed to the internal hard drive?
def ProcessLargeTextFile(): r = open("filepath", "r") w = open("filepath", "w") l = r.readline() while l: x = l.split(' ')[0] y = l.split(' ')[1] z = l.split(' ')[2] w.write(l.replace(x,x[:-3]).replace(y,y[:-3]).replace(z,z[:-3])) l = r.readline() r.close() w.close()
Any help would be really appreciated. I am using the IDLE Python GUI on Windows 7 and have 16GB of memory - perhaps a different OS would be more efficient?.
Edit: Here is an extract of the file to be processed.
70700.642014 31207.277115 -0.054123 -1585 255 255 255 70512.301468 31227.990799 -0.255600 -1655 155 158 158 70515.727097 31223.828659 -0.066727 -1734 191 187 180 70566.756699 31217.065598 -0.205673 -1727 254 255 255 70566.695938 31218.030807 -0.047928 -1689 249 251 249 70536.117874 31227.837662 -0.033096 -1548 251 252 252 70536.773270 31212.970322 -0.115891 -1434 155 158 163 70533.530777 31215.270828 -0.154770 -1550 148 152 156 70533.555923 31215.341599 -0.138809 -1480 150 154 158
Reading Large Text Files in Python We can use the file object as an iterator. The iterator will return each line one by one, which can be processed. This will not read the whole file into memory and it's suitable to read large files in Python.
In Python, files are read by using the readlines() method. The readlines() method returns a list where each item of the list is a complete sentence in the file. This method is useful when the file size is small.
The steps to import a TXT or CSV file into Excel are similar for Excel 2007, 2010, 2013, and 2016: Open the Excel spreadsheet where you want to save the data and click the Data tab. In the Get External Data group, click From Text. Select the TXT or CSV file you want to convert and click Import.
It's more idiomatic to write your code like this
def ProcessLargeTextFile(): with open("filepath", "r") as r, open("outfilepath", "w") as w: for line in r: x, y, z = line.split(' ')[:3] w.write(line.replace(x,x[:-3]).replace(y,y[:-3]).replace(z,z[:-3]))
The main saving here is to just do the split
once, but if the CPU is not being taxed, this is likely to make very little difference
It may help to save up a few thousand lines at a time and write them in one hit to reduce thrashing of your harddrive. A million lines is only 54MB of RAM!
def ProcessLargeTextFile(): bunchsize = 1000000 # Experiment with different sizes bunch = [] with open("filepath", "r") as r, open("outfilepath", "w") as w: for line in r: x, y, z = line.split(' ')[:3] bunch.append(line.replace(x,x[:-3]).replace(y,y[:-3]).replace(z,z[:-3])) if len(bunch) == bunchsize: w.writelines(bunch) bunch = [] w.writelines(bunch)
suggested by @Janne, an alternative way to generate the lines
def ProcessLargeTextFile(): bunchsize = 1000000 # Experiment with different sizes bunch = [] with open("filepath", "r") as r, open("outfilepath", "w") as w: for line in r: x, y, z, rest = line.split(' ', 3) bunch.append(' '.join((x[:-3], y[:-3], z[:-3], rest))) if len(bunch) == bunchsize: w.writelines(bunch) bunch = [] w.writelines(bunch)
Measure! You got quite some useful hints how to improve your python code and I agree with them. But you should first figure out, what your real problem is. My first steps to find your bottleneck would be:
If you figured out the exact problem, ask again for optimizations of that problem.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With