Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to save/load a tensorflow hub module to/from a custom path?

The tensorflow_hub library maintainers has made it every easy for users to download and use the pre-trained tensorflow modules, e.g.:

import tensorflow_hub as hub

embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder/1")

But from the sys.stderr it seemed like it was saving the module locally to a temporary directory, i.e.

INFO:tensorflow:Using /var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules to cache modules. INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_0:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_0 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_1:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_1 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_10:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_10 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_11:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_11 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_12:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_12 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_13:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_13 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_14:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_14 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_15:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_15 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_16:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_16 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_2:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_2 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_3:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_3 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_4:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_4 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_5:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_5 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_6:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_6 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_7:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_7 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_8:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_8 INFO:tensorflow:Initialize variable module/Embeddings_en/sharded_9:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Embeddings_en/sharded_9 INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_0/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Encoder_en/DNN/ResidualHidden_0/weights INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_1/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Encoder_en/DNN/ResidualHidden_1/weights INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_2/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Encoder_en/DNN/ResidualHidden_2/weights INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_3/projection:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Encoder_en/DNN/ResidualHidden_3/projection INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_3/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with Encoder_en/DNN/ResidualHidden_3/weights INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/bias:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/bias INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0/weights INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/bias:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/bias INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1/weights INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/bias:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/bias INFO:tensorflow:Initialize variable module/SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2/weights INFO:tensorflow:Initialize variable module/SNLI/Classifier/LinearLayer/bias:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SNLI/Classifier/LinearLayer/bias INFO:tensorflow:Initialize variable module/SNLI/Classifier/LinearLayer/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SNLI/Classifier/LinearLayer/weights INFO:tensorflow:Initialize variable module/SNLI/Classifier/tanh_layer_0/bias:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SNLI/Classifier/tanh_layer_0/bias INFO:tensorflow:Initialize variable module/SNLI/Classifier/tanh_layer_0/weights:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with SNLI/Classifier/tanh_layer_0/weights INFO:tensorflow:Initialize variable module/global_step:0 from checkpoint b'/var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables' with global_step

After a machine reboot, the module gets deleted and running the hub.Module('...') code again would re-download the module.

Is is possible to save the module to a custom directory and afterwards load from the custom directory?

If it's possible, how to save/load a tensorflow hub module to/from a custom path?

like image 502
alvas Avatar asked May 14 '18 01:05

alvas


People also ask

What is hub load?

Hub load is created by belt. tension. Belt tension, or the difference between. the tight side and slack side tension called the. effective tension, Te, is what creates torque.

What does the hub Keraslayer do?

This layer wraps a callable object for use as a Keras layer. The callable object can be passed directly, or be specified by a Python string with a handle that gets passed to hub. load() . This is the preferred API to load a TF2-style SavedModel from TF Hub into a Keras model.

What is hub layer in TensorFlow?

TensorFlow Hub is a repository of trained machine learning models ready for fine-tuning and deployable anywhere. Reuse trained models like BERT and Faster R-CNN with just a few lines of code.


1 Answers

  • Take the URL and change it from:

https://tfhub.dev/google/universal-sentence-encoder/1

To:

https://storage.googleapis.com/tfhub-modules/google/universal-sentence-encoder/1.tar.gz

  • Download using Curl or browser.
  • Extract to a prefered location (e.g. /home/admin/embed/)
  • Change your code to:

.

import tensorflow_hub as hub 
embed = hub.load('/home/admin/embed/')
like image 94
Alon Lavian Avatar answered Sep 19 '22 18:09

Alon Lavian