C++ debug/print custom type with GDB : the case of nlohmann json library

Question

I'm working on a project using nlohmann's json C++ implementation.

How can one easily explore nlohmann's JSON keys/vals in GDB ?

I tried to use this STL gdb wrapping since it provides helpers to explore standard C++ library structures that nlohmann's JSON lib is using. But I don't find it convenient.

Here is a simple use case:

json foo;
foo["flex"] = 0.2;
foo["awesome_str"] = "bleh";
foo["nested"] = {{"bar", "barz"}}; 

What I would like to have in GDB:

(gdb) p foo
{
    "flex" : 0.2,
    "awesome_str": "bleh",
    "nested": etc.
}

Current behavior

(gdb) p foo
$1 = {
  m_type = nlohmann::detail::value_t::object, 
  m_value = {
    object = 0x129ccdd0, 
    array = 0x129ccdd0, 
    string = 0x129ccdd0, 
    boolean = 208, 
    number_integer = 312266192, 
    number_unsigned = 312266192, 
    number_float = 1.5427999782486669e-315
  }
}
(gdb) p foo.at("flex")
Cannot evaluate function -- may be inlined // I suppose it depends on my compilation process. But I guess it does not invalidate the question.
(gdb) p *foo.m_value.object
$2 = {
  _M_t = {
    _M_impl = {
      <std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator, nlohmann::adl_serializer> > > >> = {
        <__gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator, nlohmann::adl_serializer> > > >> = {<No data fields>}, <No data fields>}, 
      <std::_Rb_tree_key_compare<std::less<void> >> = {
        _M_key_compare = {<No data fields>}
      }, 
      <std::_Rb_tree_header> = {
        _M_header = {
          _M_color = std::_S_red, 
          _M_parent = 0x4d72d0, 
          _M_left = 0x4d7210, 
          _M_right = 0x4d7270
        }, 
        _M_node_count = 5
      }, <No data fields>}
  }
}

Answer 1

I found my own answer reading further the GDB capabilities and stack overflow questions concerning print of std::string. The short path is the easiest. The other path was hard, but I'm glad I managed to do this. There is lots of room for improvements.

• there is an open issue for this particular matter here https://github.com/nlohmann/json/issues/1952*

Short path v3.1.2

I simply defined a gdb command as follows:

# this is a gdb script
# can be loaded from gdb using
# source my_script.txt (or. gdb or whatever you like)
define pjson
# use the lohmann's builtin dump method, ident 4 and use space separator
printf "%s\n", $arg0.dump(4, ' ', true).c_str()
end
# configure command helper (text displayed when typing 'help pjson' in gdb)
document pjson
Prints a lohmann's JSON C++ variable as a human-readable JSON string
end

Using it in gdb:

(gdb) source my_custom_script.gdb
(gdb) pjson foo
{
    "flex" : 0.2,
    "awesome_str": "bleh",
    "nested": {
        "bar": "barz"
    }
}

Short path v3.7.0 [EDIT] 2019-onv-06 One may also use the new to_string() method,but I could not get it to work withing GDB with a live inferior process. Method below still works.

# this is a gdb script
# can be loaded from gdb using
# source my_script.txt (or. gdb or whatever you like)
define pjson
# use the lohmann's builtin dump method, ident 4 and use space separator
printf "%s\n", $arg0.dump(4, ' ', true, json::error_handler_t::strict).c_str()
end
# configure command helper (text displayed when typing 'help pjson' in gdb)
document pjson
Prints a lohmann's JSON C++ variable as a human-readable JSON string
end

---

April 18th 2020: WORKING FULL PYTHON GDB (with live inferior process and debug symbols)

Edit 2020-april-26: the code (offsets) here are out of blue and NOT compatible for all platforms/JSON lib compilations. The github project is much more mature regarding this matter (3 platforms tested so far). Code is left there as is since I won't maintain 2 codebases.

versions:

• https://github.com/nlohmann/json version 3.7.3
• GNU gdb (GDB) 8.3 for GNAT Community 2019 [rev=gdb-8.3-ref-194-g3fc1095]
• c++ project built with GPRBUILD/ GNAT Community 2019 (20190517) (x86_64-pc-mingw32)

The following python code shall be loaded within gdb. I use a .gdbinit file sourced in gdb.

Github repo: https://github.com/LoneWanderer-GH/nlohmann-json-gdb

GDB script

Feel free to adopt the loading method of your choice (auto, or not, or IDE plugin, whatever)

set print pretty
# source stl_parser.gdb # if you like the good work done with those STL containers GDB parsers

source printer.py # the python file is given below
python gdb.printing.register_pretty_printer(gdb.current_objfile(), build_pretty_printer())

Python script

import gdb

import platform
import sys
import traceback

# adapted from https://github.com/hugsy/gef/blob/dev/gef.py
# their rights are theirs
HORIZONTAL_LINE = "_"  # u"\u2500"
LEFT_ARROW = "<-"  # "\u2190 "
RIGHT_ARROW = "->"  # " \u2192 "
DOWN_ARROW = "|"  # "\u21b3"

nlohmann_json_type_namespace = \
    r"nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, " \
    r"std::allocator<char> >, bool, long long, unsigned long long, double, std::allocator, nlohmann::adl_serializer>"

# STD black magic
MAGIC_STD_VECTOR_OFFSET = 16  # win 10 x64 values, beware on your platform
MAGIC_OFFSET_STD_MAP = 32  # win 10 x64 values, beware on your platform

""""""
# GDB black magic
""""""
nlohmann_json_type = gdb.lookup_type(nlohmann_json_type_namespace).pointer()
# for in memory direct jumps. cast to type is still necessary yet to obtain values, but this could be changed by chaning the types to simpler ones ?
std_rb_tree_node_type = gdb.lookup_type("std::_Rb_tree_node_base::_Base_ptr").pointer()
std_rb_tree_size_type = gdb.lookup_type("std::size_t").pointer()

""""""
# nlohmann_json reminder. any interface change should be reflected here
# enum class value_t : std::uint8_t
# {
#     null,             ///< null value
#     object,           ///< object (unordered set of name/value pairs)
#     array,            ///< array (ordered collection of values)
#     string,           ///< string value
#     boolean,          ///< boolean value
#     number_integer,   ///< number value (signed integer)
#     number_unsigned,  ///< number value (unsigned integer)
#     number_float,     ///< number value (floating-point)
#     discarded         ///< discarded by the the parser callback function
# };

""""""
enum_literals_namespace = ["nlohmann::detail::value_t::null",
                            "nlohmann::detail::value_t::object",
                            "nlohmann::detail::value_t::array",
                            "nlohmann::detail::value_t::string",
                            "nlohmann::detail::value_t::boolean",
                            "nlohmann::detail::value_t::number_integer",
                            "nlohmann::detail::value_t::number_unsigned",
                            "nlohmann::detail::value_t::number_float",
                            "nlohmann::detail::value_t::discarded"]

enum_literal_namespace_to_literal = dict([(e, e.split("::")[-1]) for e in enum_literals_namespace])

INDENT = 4 # beautiful isn't it ?

def std_stl_item_to_int_address(node):
    return int(str(node), 0)


def parse_std_str_from_hexa_address(hexa_str):
    # https://stackoverflow.com/questions/6776961/how-to-inspect-stdstring-in-gdb-with-no-source-code
    return '"{}"'.format(gdb.parse_and_eval("*(char**){}".format(hexa_str)).string())


class LohmannJSONPrinter(object):
    """Print a nlohmann::json in GDB python
    BEWARE :
     - Contains shitty string formatting (defining lists and playing with ",".join(...) could be better; ident management is stoneage style)
     - Parsing barely tested only with a live inferior process.
     - It could possibly work with a core dump + debug symbols. TODO: read that stuff
     https://doc.ecoscentric.com/gnutools/doc/gdb/Core-File-Generation.html
     - Not idea what happens with no symbols available, lots of fields are retrieved by name and should be changed to offsets if possible
     - NO LIB VERSION MANAGEMENT. TODO: determine if there are serious variants in nlohmann data structures that would justify working with strucutres
     - PLATFORM DEPENDANT TODO: remove the black magic offsets or handle them in a nicer way
    NB: If you are python-kaizer-style-guru, please consider helping or teaching how to improve all that mess
    """

    def __init__(self, val, indent_level=0):
        self.val = val
        self.field_type_full_namespace = None
        self.field_type_short = None
        self.indent_level = indent_level

        self.function_map = {"nlohmann::detail::value_t::null": self.parse_as_leaf,
                            "nlohmann::detail::value_t::object": self.parse_as_object,
                            "nlohmann::detail::value_t::array": self.parse_as_array,
                            "nlohmann::detail::value_t::string": self.parse_as_str,
                            "nlohmann::detail::value_t::boolean": self.parse_as_leaf,
                            "nlohmann::detail::value_t::number_integer": self.parse_as_leaf,
                            "nlohmann::detail::value_t::number_unsigned": self.parse_as_leaf,
                            "nlohmann::detail::value_t::number_float": self.parse_as_leaf,
                            "nlohmann::detail::value_t::discarded": self.parse_as_leaf}

    def parse_as_object(self):
        assert (self.field_type_short == "object")

        o = self.val["m_value"][self.field_type_short]

        # traversing tree is a an adapted copy pasta from STL gdb parser
        # (http://www.yolinux.com/TUTORIALS/src/dbinit_stl_views-1.03.txt and similar links)

        #   Simple GDB Macros writen by Dan Marinescu (H-PhD) - License GPL
        #   Inspired by intial work of Tom Malnar,
        #     Tony Novac (PhD) / Cornell / Stanford,
        #     Gilad Mishne (PhD) and Many Many Others.
        #   Contact: [email protected] (Subject: STL)
        #
        #   Modified to work with g++ 4.3 by Anders Elton
        #   Also added _member functions, that instead of printing the entire class in map, prints a member.

        node = o["_M_t"]["_M_impl"]["_M_header"]["_M_left"]
        # end = o["_M_t"]["_M_impl"]["_M_header"]
        tree_size = o["_M_t"]["_M_impl"]["_M_node_count"]


        # in memory alternatives:

        _M_t = std_stl_item_to_int_address(o.referenced_value().address)
        _M_t_M_impl_M_header_M_left = _M_t + 8 + 16 # adding bits
        _M_t_M_impl_M_node_count    = _M_t + 8 + 16 + 16 # adding bits

        node = gdb.Value(long(_M_t_M_impl_M_header_M_left)).cast(std_rb_tree_node_type).referenced_value()
        tree_size = gdb.Value(long(_M_t_M_impl_M_node_count)).cast(std_rb_tree_size_type).referenced_value()

        i = 0

        if tree_size == 0:
            return "{}"
        else:
            s = "{\n"
            self.indent_level += 1
            while i < tree_size:
                # STL GDB scripts write "+1" which in my w10 x64 GDB makes a +32 bits move ...
                # may be platform dependant and should be taken with caution
                key_address = std_stl_item_to_int_address(node) + MAGIC_OFFSET_STD_MAP

                # print(key_object['_M_dataplus']['_M_p'])

                k_str = parse_std_str_from_hexa_address(hex(key_address))

                # offset = MAGIC_OFFSET_STD_MAP
                value_address = key_address + MAGIC_OFFSET_STD_MAP
                value_object = gdb.Value(long(value_address)).cast(nlohmann_json_type)

                v_str = LohmannJSONPrinter(value_object, self.indent_level + 1).to_string()

                k_v_str = "{} : {}".format(k_str, v_str)
                end_of_line = "\n" if tree_size <= 1 or i == tree_size else ",\n"

                s = s + (" " * (self.indent_level * INDENT)) + k_v_str + end_of_line  # ",\n"

                if std_stl_item_to_int_address(node["_M_right"]) != 0:
                    node = node["_M_right"]
                    while std_stl_item_to_int_address(node["_M_left"]) != 0:
                        node = node["_M_left"]
                else:
                    tmp_node = node["_M_parent"]
                    while std_stl_item_to_int_address(node) == std_stl_item_to_int_address(tmp_node["_M_right"]):
                        node = tmp_node
                        tmp_node = tmp_node["_M_parent"]

                    if std_stl_item_to_int_address(node["_M_right"]) != std_stl_item_to_int_address(tmp_node):
                        node = tmp_node
                i += 1
            self.indent_level -= 2
            s = s + (" " * (self.indent_level * INDENT)) + "}"
            return s

    def parse_as_str(self):
        return parse_std_str_from_hexa_address(str(self.val["m_value"][self.field_type_short]))

    def parse_as_leaf(self):
        s = "WTFBBQ !"
        if self.field_type_short == "null" or self.field_type_short == "discarded":
            s = self.field_type_short
        elif self.field_type_short == "string":
            s = self.parse_as_str()
        else:
            s = str(self.val["m_value"][self.field_type_short])
        return s

    def parse_as_array(self):
        assert (self.field_type_short == "array")
        o = self.val["m_value"][self.field_type_short]
        start = o["_M_impl"]["_M_start"]
        size = o["_M_impl"]["_M_finish"] - start
        # capacity = o["_M_impl"]["_M_end_of_storage"] - start
        # size_max = size - 1
        i = 0
        start_address = std_stl_item_to_int_address(start)
        if size == 0:
            s = "[]"
        else:
            self.indent_level += 1
            s = "[\n"
            while i < size:
                # STL GDB scripts write "+1" which in my w10 x64 GDB makes a +16 bits move ...
                offset = i * MAGIC_STD_VECTOR_OFFSET
                i_address = start_address + offset
                value_object = gdb.Value(long(i_address)).cast(nlohmann_json_type)
                v_str = LohmannJSONPrinter(value_object, self.indent_level + 1).to_string()
                end_of_line = "\n" if size <= 1 or i == size else ",\n"
                s = s + (" " * (self.indent_level * INDENT)) + v_str + end_of_line
                i += 1
            self.indent_level -= 2
            s = s + (" " * (self.indent_level * INDENT)) + "]"
        return s

    def is_leaf(self):
        return self.field_type_short != "object" and self.field_type_short != "array"

    def parse_as_aggregate(self):
        if self.field_type_short == "object":
            s = self.parse_as_object()
        elif self.field_type_short == "array":
            s = self.parse_as_array()
        else:
            s = "WTFBBQ !"
        return s

    def parse(self):
        # s = "WTFBBQ !"
        if self.is_leaf():
            s = self.parse_as_leaf()
        else:
            s = self.parse_as_aggregate()
        return s

    def to_string(self):
        try:
            self.field_type_full_namespace = self.val["m_type"]
            str_val = str(self.field_type_full_namespace)
            if not str_val in enum_literal_namespace_to_literal:
                return "TIMMY !"
            self.field_type_short = enum_literal_namespace_to_literal[str_val]
            return self.function_map[str_val]()
            # return self.parse()
        except:
            show_last_exception()
            return "NOT A JSON OBJECT // CORRUPTED ?"

    def display_hint(self):
        return self.val.type


# adapted from https://github.com/hugsy/gef/blob/dev/gef.py
# inspired by https://stackoverflow.com/questions/44733195/gdb-python-api-getting-the-python-api-of-gdb-to-print-the-offending-line-numbe
def show_last_exception():
    """Display the last Python exception."""
    print("")
    exc_type, exc_value, exc_traceback = sys.exc_info()

    print(" Exception raised ".center(80, HORIZONTAL_LINE))
    print("{}: {}".format(exc_type.__name__, exc_value))
    print(" Detailed stacktrace ".center(80, HORIZONTAL_LINE))
    for (filename, lineno, method, code) in traceback.extract_tb(exc_traceback)[::-1]:
        print("""{} File "{}", line {:d}, in {}()""".format(DOWN_ARROW, filename, lineno, method))
        print("   {}    {}".format(RIGHT_ARROW, code))
    print(" Last 10 GDB commands ".center(80, HORIZONTAL_LINE))
    gdb.execute("show commands")
    print(" Runtime environment ".center(80, HORIZONTAL_LINE))
    print("* GDB: {}".format(gdb.VERSION))
    print("* Python: {:d}.{:d}.{:d} - {:s}".format(sys.version_info.major, sys.version_info.minor,
                                                   sys.version_info.micro, sys.version_info.releaselevel))
    print("* OS: {:s} - {:s} ({:s}) on {:s}".format(platform.system(), platform.release(),
                                                    platform.architecture()[0],
                                                    " ".join(platform.dist())))
    print(horizontal_line * 80)
    print("")
    exit(-6000)


def build_pretty_printer():
    pp = gdb.printing.RegexpCollectionPrettyPrinter("nlohmann_json")
    pp.add_printer(nlohmann_json_type_namespace, "^{}$".format(nlohmann_json_type_namespace), LohmannJSONPrinter)
    return pp


######
# executed at autoload (or to be executed by in GDB)
# gdb.printing.register_pretty_printer(gdb.current_objfile(),build_pretty_printer())

BEWARE :
 - Contains shitty string formatting (defining lists and playing with ",".join(...) could be better; ident management is stoneage style)
 - Parsing barely tested only with a live inferior process.
 - It could possibly work with a core dump + debug symbols. TODO: read that stuff
 https://doc.ecoscentric.com/gnutools/doc/gdb/Core-File-Generation.html
 - Not idea what happens with no symbols available, lots of fields are retrieved by name and should be changed to offsets if possible
 - NO LIB VERSION MANAGEMENT. TODO: determine if there are serious variants in nlohmann data structures that would justify working with structures
 - PLATFORM DEPENDANT TODO: remove the black magic offsets or handle them in a nicer way
NB: If you are python-kaizer-style-guru, please consider helping or teaching how to improve all that mess

some (light tests):

gpr file:

project Debug_Printer is

   for Source_Dirs use ("src", "include");
   for Object_Dir use "obj";
   for Main use ("main.cpp");
   for Languages use ("C++");

   package Naming is
      for Spec_Suffix ("c++") use ".hpp";
   end Naming;

   package Compiler is
      for Switches ("c++") use ("-O3", "-Wall", "-Woverloaded-virtual", "-g");
   end Compiler;

   package Linker is
      for Switches ("c++") use ("-g");
   end Linker;

end Debug_Printer;

main.cpp

#include // i am using the standalone json.hpp from the repo release #include

using json = nlohmann::json;

int main() {
  json fooz;
  fooz = 0.7;

  json arr = {3, "25", 0.5};

  json one;
  one["first"] = "second";

  json foo;
  foo["flex"] = 0.2;
  foo["bool"] = true;
  foo["int"] = 5;
  foo["float"] = 5.22;
  foo["trap "] = "you fell";
  foo["awesome_str"] = "bleh";
  foo["nested"] = {{"bar", "barz"}};
  foo["array"] = { 1, 0, 2 };

  std::cout << "fooz" << std::endl;
  std::cout << fooz.dump(4) << std::endl << std::endl;

  std::cout << "arr" << std::endl;
  std::cout << arr.dump(4) << std::endl << std::endl;

  std::cout << "one" << std::endl;
  std::cout << one.dump(4) << std::endl << std::endl;

  std::cout << "foo" << std::endl;
  std::cout << foo.dump(4) << std::endl << std::endl;

  json mixed_nested;

  mixed_nested["Jean"] = fooz;
  mixed_nested["Baptiste"] = one;
  mixed_nested["Emmanuel"] = arr;
  mixed_nested["Zorg"] = foo;

  std::cout << "5th element" << std::endl;
  std::cout << mixed_nested.dump(4) << std::endl << std::endl;

  return 0;
}

outputs:

(gdb) source .gdbinit

Breakpoint 1, main () at F:\DEV\Projets\nlohmann.json\src\main.cpp:45
(gdb) p mixed_nested
$1 = {
    "Baptiste" : {
            "first" : "second"
    },
    "Emmanuel" : [
            3,
            "25",
            0.5,
    ],
    "Jean" : 0.69999999999999996,
    "Zorg" : {
            "array" : [
                    1,
                    0,
                    2,
            ],
            "awesome_str" : "bleh",
            "bool" : true,
            "flex" : 0.20000000000000001,
            "float" : 5.2199999999999998,
            "int" : 5,
            "nested" : {
                    "bar" : "barz"
            },
            "trap " : "you fell",
    },
}

---

Edit 2019-march-24 : add precision given by employed russian.

Edit 2020-april-18 : after a long night of struggling with python/gdb/stl I had something working by the ways of the GDB documentation for python pretty printers. Please forgive any mistakes or misconceptions, I banged my head a whole night on this and everything is flurry-blurry now.

Edit 2020-april-18 (2): rb tree node and tree_size could be traversed in a more "in-memory" way (see above)

Edit 2020-april-26: add warning concerning the GDB python pretty printer.

Answer 2

My solution was to edit the ~/.gdbinit file.

define jsontostring 
   printf "%s\n", $arg0.dump(2, ' ', true, nlohmann::detail::error_handler_t::strict).c_str()
end

This makes the "jsontostring" command available on every gdb session without the need of sourcing any files.

(gdb) jsontostring object