Automatic Detection/Parsing of tables in Python

Question

I have to parse a lot of UNIX CLI output, most of the time, the output is a table, but they always differ in number of columns/format...etc, I have to keep modifying my parsing code for every command. TxtFSM makes life easy but i still have to write the template.

I would like to write a script that will automatically detect the CLI output and when it detects a table, it would parse it. As a newbie to the world of Parsing, I wonder How hard/complicated could that be ? If anyone have seen similar efforts/projects before ?

I always get confused with advanced parsing topics like BNF, YACC...etc

Thanks

Answer 1

Here is how you can parse df output into table using perl and Marpa::R2

script (~ denotes lexical rules):

use 5.010;
use strict;
use warnings;

use Data::Dumper;
$Data::Dumper::Indent = 1;
$Data::Dumper::Terse = 1;
$Data::Dumper::Deepcopy = 1;

use Marpa::R2;

my $g = Marpa::R2::Scanless::G->new( { source => \(<<'END_OF_SOURCE'),

    :default ::= action => [ name, value]
    lexeme default = action => [ name, value] latm => 1

        table       ::= (header [\n]) rows
        header      ~ 'Filesystem      Size  Used Avail Use% Mounted on'
        rows        ::= row+ separator => [\n]
        row         ::= fs size used avail use_percent mounted_on

        fs          ~ [A-Z] ':'
        size        ~ digits 'G'
        used        ~ digits 'G'
        avail       ~ digits 'G'
        use_percent ~ digits '%'
        mounted_on  ~ [/\w]+

        digits ~ [\d\.]+

    :discard ~ whitespace
    whitespace ~ [\s]+

END_OF_SOURCE
} );

my $input = <<EOI;
Filesystem      Size  Used Avail Use% Mounted on
C:              101G   90G   11G  90% /cygdrive/c
D:              366G  230G  137G  63% /cygdrive/d
E:               38G  9.6G   28G  26% /cygdrive/e
EOI

say Dumper $g->parse( \$input, { trace_terminals => 0 } ); exit;

output:

\[
        'table',
        [
            'rows',
            [
                'row',
                [
                    'fs',
                    'C:'
                ],
                [
                    'size',
                    '101G'
                ],
                [
                    'used',
                    '90G'
                ],
                [
                    'avail',
                    '11G'
                ],
                [
                    'use_percent',
                    '90%'
                ],
                [
                    'mounted_on',
                    '/cygdrive/c'
                ]
            ],
            [
                'row',
                [
                    'fs',
                    'D:'
                ],
                [
                    'size',
                    '366G'
                ],
                [
                    'used',
                    '230G'
                ],
                [
                    'avail',
                    '137G'
                ],
                [
                    'use_percent',
                    '63%'
                ],
                [
                    'mounted_on',
                    '/cygdrive/d'
                ]
            ],
            [
                'row',
                [
                    'fs',
                    'E:'
                ],
                [
                    'size',
                    '38G'
                ],
                [
                    'used',
                    '9.6G'
                ],
                [
                    'avail',
                    '28G'
                ],
                [
                    'use_percent',
                    '26%'
                ],
                [
                    'mounted_on',
                    '/cygdrive/e'
                ]
            ]
        ]
    ]