Concept of getter in TensorFlow

Question

In TensorFlow what is the concept and use of getter ?

The signature of tf.get_variable() is :

get_variable(
    name,
    shape=None,
    dtype=None,
    initializer=None,
    regularizer=None,
    trainable=True,
    collections=None,
    caching_device=None,
    partitioner=None,
    validate_shape=True,
    use_resource=None,
    custom_getter=None
)

The definition of custom_getter is given in the documentation as follows :

custom_getter: Callable that takes as a first argument the true getter, and allows overwriting the internal get_variable method. The signature of custom_getter should match that of this method, but the most future-proof version will allow for changes: def custom_getter(getter, *args, **kwargs). Direct access to all get_variable parameters is also allowed: def custom_getter(getter, name, *args, **kwargs). A simple identity custom getter that simply creates variables with modified names is: python def custom_getter(getter, name, *args, **kwargs): return getter(name + '_suffix', *args, **kwargs)

Unfortunately it is not very clear. Could someone please expand on it ?

Answer 1

The idea is that it provides a "hook" into the variable creation process, a way to potentially override some things when variables are created. This can be very convenient for a specific (narrow) set of issues.

Conceptually a "custom getter" is similar to a Python decorator: you write a function that gets the original function and its arguments as arguments, except you have to return the result not the modified function.

You can also specify the custom getter as an argument in tf.variable_scope() which allows you to apply it to multiple variables at once.

As a somewhat arbitrary example, let's suppose you have a large amount of code that creates a network of some sort. You wake up one morning and want to try if L2 normalizing all variables might help with the network's performance. Instead of editing all the created layers and the variables inside them, you can enclose the whole network in a scope and do the following (pseudo code):

with tf.variable_scope( "L2", custom_getter =
    lambda getter, name, shape, *args, **kwargs:
        tf.nn.l2_normalize( getter( name = name, shape = shape, *args, **kwargs ) ) ):
    # the original network here

This will automagically L2 normalize ALL variables in your network and keep them like that. Of course, if you don't want to do that to all of them, you can write more code and filter by name or other arguments. You also don't have to necessarily do this as a lambda function, you can write a normal function with def and pass that as the custom_getter argument.

Also please note this simple example commits a sin: the returned variable will be a tensor and not a variable so a tf.assign() would fail on this variable, for example.

For reference, the default getter is defined in tensorflow/python/ops/variable_scope.py in the _VariableStore class' get_variable() method, at this exact link for r1.8.

Answer 2

Take this custom getter for example, it rewrites caching_device according to variable shape size.