Problem with Sweep; how to use run.finish() and log without error + Question about defined metric

Hi all,

Nice to meet you!

Currently I’m not understanding how to use run.finish() and wandb.init for logging correctly. I’m constantly getting an error when the wandb.agent sweeps to another model configuration. It’s successfully doing the K-fold split, I dont see any errors. But it’s right after the K-fold split when the now model configuration is applied by the sweep.

Information about my code:
My code is a bit messy. I think there is no way to use K cross validation from scikitlearn. I’ve tried it many times, but my input and output are (with N = number of datasets):

Input 1: N datasets of 1000 numbers (x-axis)
Input2 : N datasets of 1000 numbers (y-axis)
Output1: 1 number for each N’th dataset
Output2: 1 number for each N’th dataset

Input 1 and 2 are concatenated to produce 2 outputs. Lets say N is 300 and split is 0.2 then:
Output1.shape, Output2.shape, Output1_test.shape, Output2_test.shape, X.shape, Y.shape, X_test.shape, Y_test.shape

In the same order, their shapes: ((240,), (240,), (60,), (60,), (240, 1000), (240, 1000), (60, 1000), (60, 1000))
I think there is just no way I can define the cross validation with sklearn with this type of data I think…

Error
I’ve introduced to save to model each time it’s configured. Then load the model in each for loop with zero weigths. This way may cross validation succeeds. However, I’m not sure how to correctly log my files. This code is doing bad at producing the groups I want them to be in; it’s just overwriting them. Also, and as I mentioned; every time when a new model is initiated by the sweep, I get and error:

display

wandb: Sweep Agent: Waiting for job.
wandb: Job received.
wandb: While tearing down the service manager. The following error has occured: [WinError 10054] De externe host heeft een verbinding verbroken
wandb: Agent Starting Run: ekc1s4gm with config:
wandb: 	batch_size: 6
wandb: 	dense_units: 63.48472025895866
wandb: 	dense_units2: 81.47931201263756
wandb: 	learning_rate: 0.0007466619646085462
wandb: 	num_layers: 10
wandb: 	optimizer: Adam
wandb: WARNING Ignored wandb.init() arg project when running a sweep.

Exception in thread ChkStopThr: 
.....
Some file information
.....

Exception in thread NetStatThr::
.....
Some file information
.....

Ending in: 

ConnectionResetError: [WinError 10054] De externe host heeft een verbinding verbroken
    sent = self._sock.send(data)
ConnectionResetError: [WinError 10054] De externe host heeft een verbinding verbroken

So when this error occurs, it just continues right after it created the new model configuration. The last time this error occurs is when the last loop and last model gets evaluated. Also this error occurs every loop

wandb: WARNING Ignored wandb.init() arg project when running a sweep.

I think it definitely has to do something with the Groups I want certain logs to be in and therefore also my run.finish() command (The error does not error without any wandb.init()!!! The error also occurs with just the wandb.init() in the for loop, also if I add this, each sweep is overwritten by the other sweep so it’s not creating groups aswell!!. I’m unsure if I placed them correctly. it sounded logical to me to define run just once and to have to others as just wandb.init()… (see me code). I just don’t understand how to use it in this case… I hope you do ? How can I group my folds and my validation seperately without using wandb.init() ? Any recommendations on this would be very welcome!

My code
Here is my code… A little messy, sorry.

def seed_all(seed):
    np.random.seed(seed)
    os.environ['PYTHONHASHSEED'] = str(seed)
    tf.random.set_seed(seed)
def build_model(config):
  
    activ = config.activation   
    dense_units = config.dense_units  
    dense_units2 = config.dense_units2
    num_layers = config.num_layers
    batch_size = config.batch_size
    batch_norm = False
    optimizer = config.optimizer
    learning_rate = config.learning_rate
    
    x = tf.keras.layers.Concatenate()([input1, input2])
                   
        #input_layer = Input(shape=(len(norm_train_X .columns), len(norm_train_X.iloc[0][0]))
    x = Dense(units=dense_units, activation=activ)(x)
        
    for _ in range(num_layers):
        x = Dense(units=dense_units, activation=activ)(x)
        
        
    x1 = Dense(units=dense_units2, activation=activ)(x)
    
        # Y1 output will be fed directly from the second dense
    
    y1_output = Dense(units='1', name='y1_output')(x1)

    third_dense = Dense(units=dense_units2, activation=activ)(x1)

         # Y2 output will come via the third dense
    y2_output = Dense(units='1', name='y2_output')(third_dense)
        
    model = Model(inputs=[input1, input2], outputs=[y1_output, y2_output])
    print(model.summary())   
    model.save_weights('model.h5', overwrite = True)
    return model
def train():
    test_loss_sum =np.array([0])
    Hp_loss_sum = np.array([0])
    MDp_loss_sum = np.array([0])
    Hp_rmse_sum = np.array([0])
    MDp_rmse_sum = np.array([0])
    loss_sum = np.array([0])
    loss_sum_tot =0 
    Hp_R2_append = []
    test_loss_sum_tot =0
    hp_append = []
    MDp_append = []
    hp_pred_append = []
    MDp_R2_append = []
    MDp_sum = 0
    Hp_sum = 0
    test_loss_mean = 0 
    hp_pred_append = []
    MDp_pred_append = []
    Hp_loss_sum_tot = 0
    MDp_loss_sum_tot =0
    Hp_rmse_sum_tot =0
    MDp_rmse_sum_tot=0
    
    X = np.vstack(np.asarray(norm1.numpy()[:]))
    Y = np.vstack(np.asarray(norm2.numpy()[:]))
    max_trials = 2
    epochs = 100
    test_loss_sum = np.array([0])
    Hp = train_Y_1_t
    MDp = train_Y_2_t
    Hp_test = test_Y_1_t
    MDp_test = test_Y_2_t
    hyperparams = dict(
        lr = 0.0001,
        optimizer = 'Adam',
        dense_units = 256,
        batch_size = 64,
        epochs = 1,        
        ense_units2 = 64,
        activation = 'relu',)
    
    cb_reducelr = tf.keras.callbacks.ReduceLROnPlateau(
        monitor = "val_loss",
        mode = 'auto',
        factor = 0.1,
        patience = 20,
        min_delta = 1e-04, #default
        min_lr = 1e-07,
        verbose = 1)

    cb_earlystop= tf.keras.callbacks.EarlyStopping(
        monitor="val_loss",
        mode="auto",
        min_delta = 0,
        patience=25,
        verbose=1)
    
    run = wandb.init(project="custom-charts", config=hyperparams, reinit = True) #Note the Reinit here!
    config = wandb.config
    
    Wandcalback = WandbCallback(monitor='val_loss')
    
    model =  build_model(config=config)
    
    LR = config.learning_rate     # 0.001
    
    if config.optimizer=='Adam':
        optimizer = tf.keras.optimizers.Adam(lr = LR)
    elif config.optimizer=='RMSprop':
        optimizer = tf.keras.optimizers.RMSprop(lr=LR, rho=0.9, epsilon=1e-08, decay=0.0)
    else: 
        raise
    
    # Compile the model
    model.compile(optimizer=optimizer,
    loss={'y1_output': 'mse', 'y2_output': 'mse'},
    metrics={'y1_output': tf.keras.metrics.RootMeanSquaredError(),'y2_output': tf.keras.metrics.RootMeanSquaredError()})
   
    n_splits = 4
    skf = KFold(n_splits, shuffle = True)
    skf.get_n_splits(X, Y)
    i=0
    vall_loss = []
    for train_index, test_index in skf.split(X, Y):
        
        wandb.init(project="custom-charts", group = "folds_experiment", job_type = "fold{}".format(i)) 
        model.load_weights('model.h5')
        
        train_index = train_index.astype(int)
        test_index = test_index.astype(int)
        X = np.array(X)
        Y = np.array(Y)
        Hp = np.array(Hp)
        MDp = np.array(MDp)
        X_train, X_test = X[train_index], X[test_index]
        Y_train, Y_test = Y[train_index], Y[test_index]
        Hp_train, Hp_test = Hp[train_index], Hp[test_index]
        MDp_train, MDp_test = MDp[train_index],MDp[test_index]      
        
        history = model.fit([tf.convert_to_tensor(X), tf.convert_to_tensor(Y)], [Hp, MDp], validation_data = ([tf.convert_to_tensor(X_test), tf.convert_to_tensor(Y_test)], [Hp_test, MDp_test]), 
                     batch_size=config.batch_size,    
                     epochs=2,             
                     callbacks=[Wandcalback,cb_earlystop,cb_reducelr],
                     verbose=1)
        
        loss_sum = pd.DataFrame(history.history)['loss'].iloc[-1]  + loss_sum
        test_loss_sum = pd.DataFrame(history.history)['val_loss'].iloc[-1]  + test_loss_sum
        Hp_loss_sum = pd.DataFrame(history.history)['val_y1_output_loss'].iloc[-1]  + Hp_loss_sum
        MDp_loss_sum = pd.DataFrame(history.history)['val_y2_output_loss'].iloc[-1]  + MDp_loss_sum
        Hp_rmse_sum = pd.DataFrame(history.history)['val_y1_output_root_mean_squared_error'].iloc[-1]  + Hp_rmse_sum
        MDp_rmse_sum = pd.DataFrame(history.history)['val_y2_output_root_mean_squared_error'].iloc[-1]  + MDp_rmse_sum
            
        loss_sum_tot = pd.DataFrame(history.history)['loss']  + loss_sum_tot
        test_loss_sum_tot = pd.DataFrame(history.history)['val_loss']  + test_loss_sum_tot
        Hp_loss_sum_tot = pd.DataFrame(history.history)['val_y1_output_loss'] +  Hp_loss_sum_tot
        MDp_loss_sum_tot = pd.DataFrame(history.history)['val_y2_output_loss']  + MDp_loss_sum_tot
        Hp_rmse_sum_tot = pd.DataFrame(history.history)['val_y1_output_root_mean_squared_error']  +  Hp_rmse_sum_tot
        MDp_rmse_sum_tot = pd.DataFrame(history.history)['val_y2_output_root_mean_squared_error']  +  MDp_rmse_sum_tot   

        Y_pred = model.predict([tf.convert_to_tensor(X_test), tf.convert_to_tensor(Y_test)])
        metric = tfa.metrics.r_square.RSquare()
            
        metric.update_state(Hp_test, Y_pred[0].flatten())
        result = metric.result()
        R_2_Hp = result.numpy()
        Hp_R2_append.append(R_2_Hp)
           
        metric.update_state(MDp_test, Y_pred[1].flatten())
        result = metric.result()
        R_2_MDp = result.numpy()
        MDp_R2_append.append(R_2_MDp)
        
        hp_append.append(Hp_test)
        MDp_append.append(MDp_test)
        hp_pred_append.append(Y_pred[0])
        MDp_pred_append.append(Y_pred[1])
        MDp_sum = MDp_sum + Y_pred[1]
        Hp_sum = Hp_sum + Y_pred[0]
        i = i + 1 
    
    test_loss_mean = test_loss_sum/n_splits
    loss_sum_mean = loss_sum/n_splits
    test_loss_sum_mean =  test_loss_sum/n_splits
    Hp_loss_sum_mean = Hp_loss_sum/n_splits
    MDp_loss_sum_mean = MDp_loss_sum/n_splits
    Hp_rmse_sum_mean = Hp_rmse_sum/n_splits
    MDp_rmse_sum_mean = MDp_rmse_sum/n_splits
         
    test_MDp_R2 = np.mean(MDp_R2_append)
    test_Hp_R2 = np.mean(Hp_R2_append)
                
        
    Hp_mean = Hp_sum/n_splits
    MDp_mean = MDp_sum/n_splits    
        # wandb.init(project= "sweep & optimalisation RandomSearch", group="experimentfold{}".format(i), job_type="validation")
    wandb.init(project="custom-charts", group ="folds_experiment", job_type = "validation")
    for val_los in range(len(test_loss_sum_tot)):
        wandb.log({"val_loss_mean" : test_loss_sum_tot[val_los]/n_splits})
    for loss in range(len(loss_sum_tot)):
        wandb.log({"loss_mean": loss_sum_tot[loss]/n_splits})
    for val_MDp_los in range(len(test_loss_sum_tot)):
        wandb.log({"val_MDp_loss_mean" : test_loss_sum_tot[val_MDp_los]/n_splits})
    for val_hp_los in range(len( Hp_loss_sum_tot)):
        wandb.log({"val_hp_loss_mean": Hp_loss_sum_tot[val_hp_los]/n_splits})
    for val_MDp_rmse in range(len(MDp_rmse_sum_tot)):
        wandb.log({"val_MDp_rmse_mean" : MDp_loss_sum_tot[val_MDp_rmse]/n_splits})
    for val_hp_rmse in range(len(Hp_rmse_sum_tot)):
        wandb.log({"val_MDp_rmse_mean": Hp_rmse_sum_tot[val_hp_rmse]/n_splits})
       
    hp_append = np.concatenate(hp_append)
    MDp_append = np.concatenate(MDp_append)
    hp_pred_append = np.concatenate(hp_pred_append)
    MDp_pred_append = np.concatenate(MDp_pred_append)
            
    Hp_score = np.sqrt(mean_squared_error(hp_pred_append,hp_append))
    MDp_score = np.sqrt(mean_squared_error(MDp_pred_append,MDp_append))  
    test_MDp_R2 = np.mean(MDp_R2_append)
    test_Hp_R2 = np.mean(Hp_R2_append)
        
    wandb.log({"R2_score_hp":Hp_score, "R2_score_MDp":MDp_score, "R2_hp":test_Hp_R2, "R2_MDp":test_MDp_R2})
    Hp = np.asarray(Hp_mean.flatten())
    MDp = np.asarray(MDp_mean.flatten())
    Hp_testt = np.asarray(Hp_test.flatten())
    MDp_testt = np.asarray(MDp_test.flatten())
        
        
    fd = pd.DataFrame({"pred": Hp,"actual":Hp_testt})
    print(fd)
    table = wandb.Table(dataframe=fd)
    wandb.log({'scatter-plot1': wandb.plot.scatter(table, "pred", "actual")})
        
    fd2 = pd.DataFrame({"pred": MDp,"actual":MDp_testt})
    print(fd2)
    table2 = wandb.Table(dataframe=fd2)
    wandb.log({'scatter-plot2': wandb.plot.scatter(table2, "pred", "actual")})
        
    predictions_h = [s for s in Hp_mean]
    predictions_h
    table2 = wandb.Table(data=predictions_h, columns=["h_predictions"])
    wandb.log({'my_histogramM': wandb.plot.histogram(table2, "h_predictions",
    title="Prediction Score Distribution Hubble Parameter")})
        # hist = np.histogram(predictions_h)
        # wandb.log({'Hubble parameter': wandb.plot.histogram(hist)})
        
        
    predictions_hh = [ s for s in MDp_mean]
    predictions_hh
    table3 = wandb.Table(data=predictions_hh, columns=["h_predictions"])
    wandb.log({'my_histogram': wandb.plot.histogram(table3, "h_predictions",
    title="Prediction Score Distribution Mass Density")})
        # hist = np.histogram(predictions_hh)
        # wandb.log({'Mass Density parameter': wandb.plot.histogram(hist)})
    run.finish()
sweep_config = {
    'method': 'random',         
    'metric': {
        'name': 'test_loss_mean',     
        'goal': 'minimize'      
    },
    'parameters': {
        'dense_units': {
            'distribution': 'log_uniform_values',
            'min': 32,
            'max': 256
        },
        'learning_rate': {
            'distribution': 'log_uniform_values',
            'min': 0.0000001,
            'max': 0.1
        },
        'dense_units2': {
            'distribution': 'log_uniform_values',
            'min': 32,
            'max': 256
        },
        'batch_size': {
            #Integers between 32 and 256 
            # with evenly distributed logarithms
            'values': [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20],
               
        },
        'optimizer': {
            'values': ['Adam', 'RMSprop']
        },
        'num_layers': {
            'values': [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
        }

    }
}
sweep_id = wandb.sweep(sweep_config, entity="stijnvdbosch", project="custom-charts")
wandb.agent(sweep_id, function=train, count=2, project="custom-charts")

You can also see i’m updating my own defined metric (not from model.fit) called test_loss_mean. I suppose I did that correct?
If there is any more information you need to help me, then, please, send me a message and I will reply in a blink. :blush: