=================================================================================
TensorFlow introduced estimators that were designed for scaling and distributed training of machine learning models. For tf.estimator, for loop is not needed. It handles iteration just as Keras. tf.estimator API is used to train models. Estimators had a significant advantage as they offered fault tolerance training in a distributed environment, but its APIs were not very user-friendly and often caused confusion. Fortunately, TensorFlow 2.0 has introduced the standardized version of tf.keras, which combines the simplicity of Keras and power of estimators.
In tf.estimator API:
i) model_fn defines the computation graph, which returns a different tf.estimator.EstimatorSpec.
ii) In mode == tf.estimator.ModeKeys.TRAIN, you can specify a train_op to be called at each training iteration, which in turns changes trainable instances of tf.Variable, to optimize a certain loss.
ii) model_fn defines the computation graph and is passed as a parameter to the init method of a tf.estimator.Estimator.
iii) If you need the auxiliary variable only after training, tf.estimator.Estimator.get_variable_value can be used to extract the values of variables A and B after training as numpy arrays and do your computations to get C. However then C would not be part of the model.
iv) Hook:
iv.a) A hook can be written with an end method that will be called at the end of the session (i.e. when training stops).
iv.b) Since the hooks needs access to the variables, you would need to define the hook inside the model function.
iv.c) tf.train.SessionRunHook is an interface for defining session-related hooks.
v) If you want to change the value of variables, you must not change in the end() function since the results of changing cannot be stored in the checkpoint file. If you change the value in the, for example, after_run function, then the results will be stored in the checkpoint.
vi) Both tf.contrib.estimator.stop_if_no_decrease_hook and tf.estimator.experimental.stop_if_no_decrease_hook are used for early stopping for the training based on the evaluation loss. The early-stopping hook uses the evaluation results to decide when it's time to cut the training, but you need to pass in the number of training steps that you want to monitor and keep in mind how many evaluations will happen in that number of steps. For instance, if you set the hook as hook = early_stopping.stop_if_no_decrease_hook(estimator, "loss", 10000) the hook will consider the evaluations happening in a range of 10k steps.
vii) estimator automatically saves an restores the model for you during the training.
viii) tf.estimator.Estimator. When the tf.estimator.Estimator is used, then you don't need to add the summaries to a FileWriter, since it's done automatically (merging and saving them every 100 steps by default).
viii.a) tf.estimator.Estimator.get_variable_names() can be used in order to get all estimator variables and to get all variable names.
viii.b) tf.estimator.Estimator.get_variable_value(name) can be used in order to get variables values by name.
ix) Both tf.estimator.Estimator.evaluate and tf.estimator.Estimator.predict have a checkpoint_path argument. You should be able to supply the path to model.ckpt-1 here to use this checkpoint for evaluation.
ix.a) tf.estimator.evaluate() returns a dictionary contained fields defined by the eval_metrics. tf.estimator.Estimator.evaluate() can be used to evaluate the model.
ix.b) tf.estimator.Estimator.evaluate() accepts checkpoint_path and restores variables.
x) tf.estimator has early stopping support on master. To use this function, you must first name the loss to make it available to the early stopping call. If your loss variable is named "loss" in the estimator, then add the line below to have the function available.
copyloss = tf.identity(loss, name="loss")
============================================
|
