boost::program_options and multiple sections in ini-file

Question

I'm trying to get boost::program_options to read a ini file with multiple sections:

[slave]
address=localhost
port=1111

[slave]
address=192.168.0.1
port=2222

Is there any solution?

Thanks in advance!

Answer 1

There are a few solutions to this problem. While it may initially appear that this should be an easy task, it is often fairly involved. This is because sections are roughly equivalent to namespaces; sections are not equivalent to objects.

[slave]
address=localhost
port=1111

[slave]
address=192.168.0.1
port=2222

The above configuration has a single slave namespace, that contains two address values and two port values. There are not two slave objects that each have an address and port. Due to this distinction, associating values, or pairing, must be done in the application code. This presenting the following options:

• Use the configuration file's layout to imply pairing.
• Perform explicit pairing by merging multiple values into single field value.

---

Implied Pairing

With this approach, the configuration file can remain as-is. The simplicity of this approach depends on:

• The behavior of a few Boost.ProgramOption components.
• Each object represented as a section having no optional fields and a small number of fields.

[slave]
address=localhost    # slave.address[0]
port=1111            # slave.port[0]

[slave]
address=192.168.0.1  # slave.address[1]
port=2222            # slave.port[1]

Without modifying the configuration, the following code:

#include <algorithm>
#include <fstream>
#include <iostream>
#include <iterator>
#include <string>
#include <vector>

#include <boost/program_options.hpp>

/// @brief Convenience function for when a 'store_to' value is being provided
///        to typed_value.
///
/// @param store_to The variable that will hold the parsed value upon notify.
///
/// @return Pointer to a type_value.
template < typename T >
boost::program_options::typed_value< T >* make_value( T* store_to )
{
  return boost::program_options::value< T >( store_to );
}

/// @brief Slave type that contains an address and port.
struct slave
{
  std::string    address;
  unsigned short port;

  /// @brief Constructor.
  slave( std::string address, 
         unsigned short port )
    : address( address ),
      port( port )
  {}
};

/// @brief Stream insertion operator for slave.
///
/// @param stream The stream into which slave is being inserted.
/// @param s The slave object.
///
/// @return Reference to the ostream.
std::ostream& operator<<( std::ostream& stream, 
                          const slave& slave )
{
  return stream << "Slave address: " << slave.address 
                << ", port: "        << slave.port;
}

/// @brief Makes a slave given an address and port.
slave make_slave( const std::string& address,
                  unsigned short port )
{
  return slave( address, port );
}

int main()
{
  // Variables that will store parsed values.
  std::vector< std::string >    addresses;
  std::vector< unsigned short > ports;

  // Setup options.
  namespace po = boost::program_options;
  po::options_description desc( "Options" );
  desc.add_options()
    ( "slave.address", make_value( &addresses ),
                       "slave's hostname or ip address" )
    ( "slave.port"   , make_value( &ports ),
                       "plugin id" );

  // Load setting file.
  po::variables_map vm;
  std::ifstream settings_file( "config.ini", std::ifstream::in );
  po::store( po::parse_config_file( settings_file , desc ), vm );
  settings_file.close();
  po::notify( vm );

  // Transform each address and port pair into a slave via make_slave,
  // inserting each object into the slaves vector.
  std::vector< slave > slaves;
  std::transform( addresses.begin(), addresses.end(),
                  ports.begin(),
                  std::back_inserter( slaves ),
                  make_slave );

  // Print slaves.
  std::copy( slaves.begin(), slaves.end(), 
             std::ostream_iterator< slave >( std::cout, "\n" ) );
}

Produces this output:

Slave address: localhost, port: 1111
Slave address: 192.168.0.1, port: 2222

---

Basic Explicit Pairing

Multiple values can be occasionally be represented within a single field in a meaningful way. One common representation of both address and port is address:port. With this pairing, the resulting configuration file would like:

[slaves]
slave=localhost:1111
slave=192.168.0.1:2222

This simplicity of this approach depends upon:

• Being able to represent multiple-values as a single meaningful value without key specifiers.
• Each object having no optional values.

The updated code:

#include <algorithm>
#include <fstream>
#include <iostream>
#include <iterator>
#include <string>
#include <vector>

#include <boost/algorithm/string/classification.hpp>
#include <boost/algorithm/string/split.hpp>
#include <boost/lexical_cast.hpp>
#include <boost/program_options.hpp>

/// @brief Convenience function for when a 'store_to' value is being provided
///        to typed_value.
///
/// @param store_to The variable that will hold the parsed value upon notify.
///
/// @return Pointer to a type_value.
template < typename T >
boost::program_options::typed_value< T >* make_value( T* store_to )
{
  return boost::program_options::value< T >( store_to );
}

/// @brief Slave type that contains an address and port.
struct slave
{
  std::string    address;
  unsigned short port;

  /// @brief Constructor.
  slave( std::string address, 
         unsigned short port )
    : address( address ),
      port( port )
  {}
};

/// @brief Stream insertion operator for slave.
///
/// @param stream The stream into which slave is being inserted.
/// @param s The slave object.
///
/// @return Reference to the ostream.
std::ostream& operator<<( std::ostream& stream, 
                          const slave& slave )
{
  return stream << "Slave address: " << slave.address 
                << ", port: "        << slave.port;
}

/// @brief Makes a slave given an address and port.
slave make_slave( const std::string& address_and_port )
{
  // Tokenize the string on the ":" delimiter. 
  std::vector< std::string > tokens;
  boost::split( tokens, address_and_port, boost::is_any_of( ":" ) );

  // If the split did not result in exactly 2 tokens, then the value
  // is formatted wrong.
  if ( 2 != tokens.size() )
  {
     using boost::program_options::validation_error;
     throw validation_error( validation_error::invalid_option_value,
                             "slaves.slave",
                             address_and_port );
  }

  // Create a slave from the token values.
  return slave( tokens[0],
                boost::lexical_cast< unsigned short >( tokens[1] ) );
}

int main()
{
  // Variables that will store parsed values.
  std::vector< std::string > slave_configs;

  // Setup options.
  namespace po = boost::program_options;
  po::options_description desc( "Options" );
  desc.add_options()
    ( "slaves.slave", make_value( &slave_configs ),
                      "slave's address@port" );

  // Load setting file.
  po::variables_map vm;
  std::ifstream settings_file( "config.ini", std::ifstream::in );
  po::store( po::parse_config_file( settings_file , desc ), vm );
  settings_file.close();
  po::notify( vm );

  // Transform each config into a slave via make_slave, inserting each 
  // object into the slaves vector.
  std::vector< slave > slaves;
  std::transform( slave_configs.begin(), slave_configs.end(),
                  std::back_inserter( slaves ),
                  make_slave );

  // Print slaves.
  std::copy( slaves.begin(), slaves.end(), 
             std::ostream_iterator< slave >( std::cout, "\n" ) );
}

Produces the same output:

Slave address: localhost, port: 1111
Slave address: 192.168.0.1, port: 2222

And the notable code modifications are as follows:

• The options_description's options need to be reading slaves.slave as a std::vector< std::string >.
• make_slave will take a single std::string argument, from which it will extract address and port.
• Update the std::transform call to only iterate over one range.

---

Advanced Explicit Pairing

Often, multiple fields cannot be represented meaningfully as a single key-less value, or an object has optional fields. For these cases, an additional level of syntax and parsing needs to occur. While applications can introduce their own syntax and parsers, I suggest leveraging Boost.ProgramOption's command line syntax (--key value and --key=value) and parsers. The resulting configuration file could look like:

[slaves]
slave= --address localhost --port 1111
slave= --address = 192.168.0.1 --port=2222

The updated code:

#include <algorithm>
#include <fstream>
#include <iostream>
#include <iterator>
#include <string>
#include <vector>

#include <boost/bind.hpp>
#include <boost/program_options.hpp>
#include <boost/tokenizer.hpp>

// copy_if was accidently left out of the C++03 standard, so mimic the
// C++11 behavior to support all predicate types.  The alternative is to
// use remove_copy_if, but it only works for adaptable functors.
template < typename InputIterator,
           typename OutputIterator, 
           typename Predicate >
OutputIterator 
copy_if( InputIterator first,
         InputIterator last,
         OutputIterator result,
         Predicate pred )
{
  while( first != last )
  {
    if( pred( *first ) )
      *result++ = *first;
    ++first;
  }
  return result;
}

/// @brief Tokenize a string.  The tokens will be separated by each non-quoted
///        character in @c separator_characters.  Empty tokens are removed.
///
/// @param input The string to tokenize.
/// @param separator_characters The characters on which to delimit.
///
/// @return Vector of tokens.
std::vector< std::string > tokenize( const std::string& input,
                                     const std::string& separator_characters )
{
   typedef boost::escaped_list_separator< char > separator_type;
   separator_type separator( "\\", // The escape characters.
                             separator_characters,
                             "\"\'" ); // The quote characters.

   // Tokenize the intput.
   boost::tokenizer< separator_type > tokens( input, separator );

   // Copy non-empty tokens from the tokenizer into the result.
   std::vector< std::string > result;
   copy_if( tokens.begin(), tokens.end(), std::back_inserter( result ), 
            !boost::bind( &std::string::empty, _1 ) );
   return result;
}

/// @brief option_builder provides a unary operator that can be used within
///        stl::algorithms.
template < typename ResultType,
           typename Builder >
class option_builder
{
public:

  typedef ResultType result_type;

public:

  /// @brief Constructor
  option_builder( const boost::program_options::options_description& options,
                  Builder builder )
    : options_( options ),
      builder_( builder )
  {}

  /// @brief Unary operator that will parse @c value, then delegate the
  ///        construction of @c result_type to the builder.
  template < typename T >
  result_type operator()( const T& value )
  {
    // Tokenize the value so that the command line parser can be used.
    std::vector< std::string > tokens = tokenize( value, "= " );

    // Parse the tokens.
    namespace po = boost::program_options;
    po::variables_map vm;
    po::store( po::command_line_parser( tokens ).options( options_ ).run(),
               vm );
    po::notify( vm );

    // Delegate object construction to the builder.
    return builder_( vm );
  }

private:

  const boost::program_options::options_description& options_;
  Builder builder_;

};

/// @brief  Convenience function used to create option_builder types.
template < typename T,
           typename Builder >
option_builder< T, Builder > make_option_builder(
  const boost::program_options::options_description& options,
  Builder builder )
{
  return option_builder< T, Builder >( options, builder );
}

/// @brief Convenience function for when a 'store_to' value is being provided
///        to typed_value.
///
/// @param store_to The variable that will hold the parsed value upon notify.
///
/// @return Pointer to a type_value.
template < typename T >
boost::program_options::typed_value< T >* make_value( T* store_to )
{
  return boost::program_options::value< T >( store_to );
}

/// @brief Slave type that contains an address and port.
struct slave
{
  std::string    address;
  unsigned short port;

  /// @brief Constructor.
  slave( std::string address, 
         unsigned short port )
    : address( address ),
      port( port )
  {}
};

/// @brief Stream insertion operator for slave.
///
/// @param stream The stream into which slave is being inserted.
/// @param s The slave object.
///
/// @return Reference to the ostream.
std::ostream& operator<<( std::ostream& stream, 
                          const slave& slave )
{
  return stream << "Slave address: " << slave.address 
                << ", port: "        << slave.port;
}

/// @brief Makes a slave given an address and port.
slave make_slave( const boost::program_options::variables_map& vm )
{
  // Create a slave from the variable map.
  return slave( vm["address"].as< std::string >(),
                vm["port"].as< unsigned short >() );
}

int main()
{
  // Variables that will store parsed values.
  std::vector< std::string > slave_configs;

  // Setup options.
  namespace po = boost::program_options;
  po::options_description desc( "Options" );
  desc.add_options()
    ( "slaves.slave", make_value( &slave_configs ),
                      "slave's --address ip/hostname --port num" );

  // Load setting file.
  po::variables_map vm;
  std::ifstream settings_file( "config.ini", std::ifstream::in );
  po::store( po::parse_config_file( settings_file , desc ), vm );
  settings_file.close();
  po::notify( vm );

  // Create options for slaves.slave.
  po::options_description slave_desc( "Slave Options" );
  slave_desc.add_options()
    ( "address", po::value< std::string >(),
                 "slave's hostname or ip address" )
    ( "port"   , po::value< unsigned short >(),
                 "slave's port" );

  // Transform each config into a slave via creating an option_builder that
  // will use the slave_desc and make_slave to create slave objects.  These
  // objects will be inserted into the slaves vector.
  std::vector< slave > slaves;
  std::transform( slave_configs.begin(), slave_configs.end(),
                  std::back_inserter( slaves ),
                  make_option_builder< slave >( slave_desc, make_slave ) );

  // Print slaves.
  std::copy( slaves.begin(), slaves.end(), 
             std::ostream_iterator< slave >( std::cout, "\n" ) ); 
}

Produces the same output as the previous approaches:

Slave address: localhost, port: 1111
Slave address: 192.168.0.1, port: 2222

And the notable code modifications are as follows:

• Created copy_if since it was an overlooked algorithm in C++03.
• Using Boost.Tokenizer instead of Boost.StringAlgo since Boost.Tokenizer handles quoted escapes easier.
• Created a option_builder unary functor to help provide idiomatic reuse for applying transformations.
• make_slave now takes a boost::program_options::variables_map from which it will construct a slave object.

This approach can also easily be extended to support the following variations:

• Supporting multiple command-lines for a single value. For example, a configuration could support two slaves, with one of the slaves having a secondary configuration in case the first fails. This requires performing an initial tokenization on the , delimiter.

  [slaves]
  slave = --address localhost --port 1111, --address 127.0.0.1 --port 1112
  slave = --address 192.168.0.1 --port 2222

• Declaring the options for slave_desc as typed_value with variables provided to the store_to argument. These same variables can then be bound with boost::ref via boost::bind to the make_slave factory function. While this decouples make_slave from Boost.ProgramOptions types, it may become difficult to maintain for types with many fields.

---

Alternatives Approaches

Alternative approaches still need explicit pairing to be done via placing multiple values into a single value. However, transformations can occur during the parsing phase by inheriting from either boost::program_options::typed_value or boost::program_options::untyped_value.

• When inheriting from typed_value, override the parse function. One consequence of using typed_value is that the template parameter must meet all the requirements for typed_value. For example, if typed_value< slave > was used, then it would require making slave default constructable, and defining both istream extraction (>>) and ostream insertion (<<) operators for slave.
• When inheriting from untyped_value, override both the parse and notify functions. This approach does not impose type requirements like typed_value, but it does require that the derived class maintain its own store_to variable.

---

Suggestions

• When it is absolute certain that there will never be an optional field and the amount of fields will be minimal (2~), then use the implied pairing approach.
• If there will be a minimal amount of fields (2~) and the values can be represented in a meaningful way without field name identifiers, then use the basic explicit pairing. Optional fields can be supported, but it increases the complexity of both the syntax and parser.
• For all other cases, or when there is any uncertainty, use the advanced explicit pairing. While it may take a little more work, it provides greater reuseability. For example, if the slave configurations become so complex that each slave has its own configuration file, then the code changes are minimal, as only the parser type and call need to be changed.