Failed to upsert bucket: returned error 403 Forbidden:

W&B Help
,
1

Hello,
I am a new using wanb. I want to experiment tracking by using wandb and executing python commands. However I get this error.

wandb: Currently logged in as: bassfaye10. Use wandb login --relogin to force relogin
wandb: ERROR failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
How to solve this error?
Thanks in advance,

2

Hi @bassfaye10 ,happy to help. Could you please share the following to better assess the issue:

-Kindly expound your specific use case/experiment
-Code Snippet
-Environment Set up
-the debug.log and debug-internal.log. These files are under your local folder wandb/run-<date>_<time>-<run-id>/logs in the same directory where you’re running your code.

3

Hi @JoanaMarieL,
I want to use machine learning for tooth segmentation. So I’ve found an example on github on this link: GitHub - limhoyeon/ToothGroupNetwork: 3D Dental surface segmentation with Tooth Group Network. I want to initiate myself on machine learning by compiling this project.
Here are informations that you need:

  1. Code Snippet:

train.py

if config[“wandb”][“wandb_on”]:
wandb.init(
entity=self.config[“wandb”][“wandb_entity”],
project=self.config[“wandb”][“project”],
notes=self.config[“wandb”][“notes”],
tags=self.config[“wandb”][“tags”],
name=self.config[“wandb”][“name”],
config=self.config,
)

  1. Full stack trace error:

_wandb: ERROR failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
Traceback (most recent call last):
File “start_train.py”, line 51, in
runner(config, model)
File “F:\ToothGroupNetwork\runner.py”, line 56, in runner
trainner = Trainer(config=config, model = model, gen_set=gen_set)
File “F:\ToothGroupNetwork\trainer.py”, line 15, in init
wandb.init(
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1266, in init
wandb.sentry.reraise(e)
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\analytics\sentry.py”, line 155, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1252, in init
return wi.init()
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 844, in init
raise error
wandb.errors.errors.CommError: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}

  1. Debug.log
    _### 2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Current SDK version is 0.18.3
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Configure stats pid to 15652
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from C:\Users\serigne_admin.config\wandb\settings
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from F:\ToothGroupNetwork\wandb\settings
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from environment variables: {}
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Applying setup settings: {‘mode’: None, ‘_disable_service’: None}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_setup.py:_flush():79] Inferring run settings from compute environment: {‘program_relpath’: ‘start_train.py’, ‘program_abspath’: ‘F:\ToothGroupNetwork\start_train.py’, ‘program’: ‘start_train.py’}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_setup.py:_flush():79] Applying login settings: {}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:_log_setup():532] Logging user logs to F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug.log
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:_log_setup():533] Logging internal logs to F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug-internal.log
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():617] calling init triggers
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():624] wandb.init called with sweep_config: {}
    config: {‘wandb’: {‘entity’: ‘bassfaye10’, ‘wandb_on’: True, ‘project’: ‘3d tooth mesh segmentation’, ‘tags’: ‘3d tooth mesh segmentation’, ‘notes’: ‘3d tooth mesh segmentation’, ‘name’: ‘tgnet_0620’}, ‘generator’: {‘input_data_dir_path’: ‘data_preprocessed_directory’, ‘train_data_split_txt_path’: ‘base_name_train_fold.txt’, ‘val_data_split_txt_path’: ‘base_name_val_fold.txt’, ‘aug_obj_str’: “aug.Augmentator([aug.Scaling([0.85, 1.15]), aug.Rotation([-30,30], ‘fixed’), aug.Translation([-0.2, 0.2])])”, ‘train_batch_size’: 1, ‘val_batch_size’: 1}, ‘checkpoint_path’: ‘ckpts/tgnet_0620’, ‘tr_set’: {‘optimizer’: {‘lr’: 0.1, ‘NAME’: ‘sgd’, ‘momentum’: 0.9, ‘weight_decay’: 0.0001}, ‘scheduler’: {‘sched’: ‘cosine’, ‘warmup_epochs’: 0, ‘full_steps’: 40, ‘schedueler_step’: 15000000, ‘min_lr’: 1e-05}, ‘loss’: {‘cbl_loss_1’: 1, ‘cbl_loss_2’: 1, ‘tooth_class_loss_1’: 1, ‘tooth_class_loss_2’: 1, ‘offset_1_loss’: 0.03, ‘offset_1_dir_loss’: 0.03, ‘chamf_1_loss’: 0.15}}, ‘model_parameter’: {‘input_feat’: 6, ‘stride’: [1, 4, 4, 4, 4], ‘nsample’: [36, 24, 24, 24, 24], ‘blocks’: [2, 3, 4, 6, 3], ‘block_num’: 5, ‘planes’: [32, 64, 128, 256, 512], ‘crop_sample_size’: 3072}}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():667] starting backend
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():671] sending inform_init request
    2024-10-14 11:16:39,652 INFO MainThread:15652 [backend.py:multiprocessing_setup():104] multiprocessing start_methods=spawn, using: spawn
    2024-10-14 11:16:39,656 INFO MainThread:15652 [wandb_init.py:init():684] backend started and connected
    2024-10-14 11:16:39,663 INFO MainThread:15652 [wandb_init.py:init():779] updated telemetry
    2024-10-14 11:16:39,663 INFO MainThread:15652 [wandb_init.py:init():812] communicating run to backend with 90.0 second timeout
    2024-10-14 11:16:40,077 ERROR MainThread:15652 [wandb_init.py:init():838] encountered error: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
    2024-10-14 11:16:40,077 ERROR MainThread:15652 [wandb_init.py:init():1262] error in wandb.init()
    Traceback (most recent call last):
    File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1252, in init
    return wi.init()
    File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 844, in init
    raise error
    wandb.errors.errors.CommError: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
    2024-10-14 11:16:40,147 WARNING MsgRouterThr:15652 [router.py:message_loop():77] message_loop has been closed
  2. Debug-internal.log:

{“time”:“2024-10-14T11:16:39.6571469+02:00”,“level”:“INFO”,“msg”:“using version”,“core version”:“0.18.3”}
{“time”:“2024-10-14T11:16:39.6571469+02:00”,“level”:“INFO”,“msg”:“created symlink”,“path”:“F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug-core.log”}
{“time”:“2024-10-14T11:16:39.667464+02:00”,“level”:“ERROR”,“msg”:“dialing: google: could not find default credentials. See … for more information”}
{“time”:“2024-10-14T11:16:39.9300979+02:00”,“level”:“INFO”,“msg”:“created new stream”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:39.9302162+02:00”,“level”:“INFO”,“msg”:“stream: started”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:39.9302162+02:00”,“level”:“INFO”,“msg”:“writer: Do: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:39.9303154+02:00”,“level”:“INFO”,“msg”:“sender: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:39.9303154+02:00”,“level”:“INFO”,“msg”:“handler: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.0765827+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:403,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.0765827+02:00”,“level”:“ERROR”,“msg”:“sender: upsertRun:”,“error”:“failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}”}
{“time”:“2024-10-14T11:16:40.1472315+02:00”,“level”:“INFO”,“msg”:“stream: closing”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:40.2625221+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:403,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.2697548+02:00”,“level”:“ERROR”,“msg”:“sender: sendConfig:”,“error”:“returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}”}
{“time”:“2024-10-14T11:16:40.2711115+02:00”,“level”:“WARN”,“msg”:“No requirements.txt found, not creating job artifact. See […”}](…
{“time”:“2024-10-14T11:16:40.2711115+02:00”,“level”:“INFO”,“msg”:“sender: sendDefer: no job artifact to save”}
{“time”:“2024-10-14T11:16:40.3161557+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:404,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.3161557+02:00”,“level”:“ERROR”,“msg”:“runfiles: CreateRunFiles returned error: returned error 404 Not Found: {“errors”:[{“message”:“project bassfaye10/3d tooth mesh segmentation not found during createRunFiles”,“path”:[“createRunFiles”]}],“data”:{“createRunFiles”:null}}”}
{“time”:“2024-10-14T11:16:40.4470144+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:404,“method”:“POST”,“url1”:“[…”}](…
{“time”:“2024-10-14T11:16:40.4470144+02:00”,“level”:“ERROR”,“msg”:“runfiles: CreateRunFiles returned error: returned error 404 Not Found: {“errors”:[{“message”:“project bassfaye10/3d tooth mesh segmentation not found during createRunFiles”,“path”:[“createRunFiles”]}],“data”:{“createRunFiles”:null}}”}
{“time”:“2024-10-14T11:16:40.4471349+02:00”,“level”:“INFO”,“msg”:“fileTransfer: Close: file transfer manager closed”}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“handler: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“writer: Close: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“sender: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4480132+02:00”,“level”:“INFO”,“msg”:“stream: closed”,“id”:“i5vb5pi5”}
Really appreciate your help,

4

I have the same issue on and off as well.
Sometimes after I delete the .netrc file, it works and sometime it does not. Help is appreciated.