How do I add a new feature column to a tf.data.Dataset object?

Question

I am building an input pipeline for proprietary data using Tensorflow 2.0's data module and using the tf.data.Dataset object to store my features. Here is my issue - the data source is a CSV file that has only 3 columns, a label column and then two columns which just hold strings referring to JSON files where that data is stored. I have developed functions that access all the data I need, and am able to use Dataset's map function on the columns to get the data, but I don't see how I can add a new column to my tf.data.Dataset object to hold the new data. So if anyone could help with the following questions, it would really help:

1. How can a new feature be appended to a tf.data.Dataset object?
2. Should this process be done on the entire Dataset before iterating through it, or during (I think during iteration would allow utilization of the performance boost, but I don't know how this functionality works)?

I have all the methods for taking the input as the elements from the columns and performing everything required to get the features for each element, I just don't understand how to get this data into the dataset. I could do "hacky" workarounds, using a Pandas Dataframe as a "mediator" or something along those lines, but I want to keep everything within the Tensorflow Dataset and pipeline process, for both performance gains and higher quality code.

I have looked through the Tensorflow 2.0 documentation for the Dataset class (https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/data/Dataset), but haven't been able to find a method that can manipulate the structure of the object.

Here is the function I use to load the original dataset:

def load_dataset(self):
    # TODO: Function to get max number of available CPU threads
    dataset = tf.data.experimental.make_csv_dataset(self.dataset_path,
                                                    self.batch_size,
                                                    label_name='score',
                                                    shuffle_buffer_size=self.get_dataset_size(),
                                                    shuffle_seed=self.seed,
                                                    num_parallel_reads=1)
    return dataset

Then, I have methods which allow me to take a string input (column element) and return the actual feature data. And I am able to access the elements from the Dataset using a function like ".map". But how do I add that as a column?

Answer 1

Wow, this is embarassing, but I have found the solution and it's simplicity literally makes me feel like an idiot for asking this. But I will leave the answer up just in case anyone else is ever facing this issue.

You first create a new tf.data.Dataset object using any function that returns a Dataset, such as ".map".

Then you create a new Dataset by zipping the original and the one with the new data:

dataset3 = tf.data.Dataset.zip((dataset1, dataset2))