Failed to upsert bucket: returned error 403 Forbidden:

W&B Help
,
1

Hello,
I am a new using wanb. I want to experiment tracking by using wandb and executing python commands. However I get this error.

wandb: Currently logged in as: bassfaye10. Use wandb login --relogin to force relogin
wandb: ERROR failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
How to solve this error?
Thanks in advance,

2

Hi @bassfaye10 ,happy to help. Could you please share the following to better assess the issue:

-Kindly expound your specific use case/experiment
-Code Snippet
-Environment Set up
-the debug.log and debug-internal.log. These files are under your local folder wandb/run-<date>_<time>-<run-id>/logs in the same directory where you’re running your code.

3

Hi @JoanaMarieL,
I want to use machine learning for tooth segmentation. So I’ve found an example on github on this link: GitHub - limhoyeon/ToothGroupNetwork: 3D Dental surface segmentation with Tooth Group Network. I want to initiate myself on machine learning by compiling this project.
Here are informations that you need:

  1. Code Snippet:

train.py

if config[“wandb”][“wandb_on”]:
wandb.init(
entity=self.config[“wandb”][“wandb_entity”],
project=self.config[“wandb”][“project”],
notes=self.config[“wandb”][“notes”],
tags=self.config[“wandb”][“tags”],
name=self.config[“wandb”][“name”],
config=self.config,
)

  1. Full stack trace error:

_wandb: ERROR failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
Traceback (most recent call last):
File “start_train.py”, line 51, in
runner(config, model)
File “F:\ToothGroupNetwork\runner.py”, line 56, in runner
trainner = Trainer(config=config, model = model, gen_set=gen_set)
File “F:\ToothGroupNetwork\trainer.py”, line 15, in init
wandb.init(
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1266, in init
wandb.sentry.reraise(e)
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\analytics\sentry.py”, line 155, in reraise
raise exc.with_traceback(sys.exc_info()[2])
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1252, in init
return wi.init()
File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 844, in init
raise error
wandb.errors.errors.CommError: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}

  1. Debug.log
    _### 2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Current SDK version is 0.18.3
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Configure stats pid to 15652
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from C:\Users\serigne_admin.config\wandb\settings
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from F:\ToothGroupNetwork\wandb\settings
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Loading settings from environment variables: {}
    2024-10-14 11:16:39,633 INFO MainThread:15652 [wandb_setup.py:_flush():79] Applying setup settings: {‘mode’: None, ‘_disable_service’: None}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_setup.py:_flush():79] Inferring run settings from compute environment: {‘program_relpath’: ‘start_train.py’, ‘program_abspath’: ‘F:\ToothGroupNetwork\start_train.py’, ‘program’: ‘start_train.py’}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_setup.py:_flush():79] Applying login settings: {}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:_log_setup():532] Logging user logs to F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug.log
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:_log_setup():533] Logging internal logs to F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug-internal.log
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():617] calling init triggers
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():624] wandb.init called with sweep_config: {}
    config: {‘wandb’: {‘entity’: ‘bassfaye10’, ‘wandb_on’: True, ‘project’: ‘3d tooth mesh segmentation’, ‘tags’: ‘3d tooth mesh segmentation’, ‘notes’: ‘3d tooth mesh segmentation’, ‘name’: ‘tgnet_0620’}, ‘generator’: {‘input_data_dir_path’: ‘data_preprocessed_directory’, ‘train_data_split_txt_path’: ‘base_name_train_fold.txt’, ‘val_data_split_txt_path’: ‘base_name_val_fold.txt’, ‘aug_obj_str’: “aug.Augmentator([aug.Scaling([0.85, 1.15]), aug.Rotation([-30,30], ‘fixed’), aug.Translation([-0.2, 0.2])])”, ‘train_batch_size’: 1, ‘val_batch_size’: 1}, ‘checkpoint_path’: ‘ckpts/tgnet_0620’, ‘tr_set’: {‘optimizer’: {‘lr’: 0.1, ‘NAME’: ‘sgd’, ‘momentum’: 0.9, ‘weight_decay’: 0.0001}, ‘scheduler’: {‘sched’: ‘cosine’, ‘warmup_epochs’: 0, ‘full_steps’: 40, ‘schedueler_step’: 15000000, ‘min_lr’: 1e-05}, ‘loss’: {‘cbl_loss_1’: 1, ‘cbl_loss_2’: 1, ‘tooth_class_loss_1’: 1, ‘tooth_class_loss_2’: 1, ‘offset_1_loss’: 0.03, ‘offset_1_dir_loss’: 0.03, ‘chamf_1_loss’: 0.15}}, ‘model_parameter’: {‘input_feat’: 6, ‘stride’: [1, 4, 4, 4, 4], ‘nsample’: [36, 24, 24, 24, 24], ‘blocks’: [2, 3, 4, 6, 3], ‘block_num’: 5, ‘planes’: [32, 64, 128, 256, 512], ‘crop_sample_size’: 3072}}
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():667] starting backend
    2024-10-14 11:16:39,652 INFO MainThread:15652 [wandb_init.py:init():671] sending inform_init request
    2024-10-14 11:16:39,652 INFO MainThread:15652 [backend.py:multiprocessing_setup():104] multiprocessing start_methods=spawn, using: spawn
    2024-10-14 11:16:39,656 INFO MainThread:15652 [wandb_init.py:init():684] backend started and connected
    2024-10-14 11:16:39,663 INFO MainThread:15652 [wandb_init.py:init():779] updated telemetry
    2024-10-14 11:16:39,663 INFO MainThread:15652 [wandb_init.py:init():812] communicating run to backend with 90.0 second timeout
    2024-10-14 11:16:40,077 ERROR MainThread:15652 [wandb_init.py:init():838] encountered error: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
    2024-10-14 11:16:40,077 ERROR MainThread:15652 [wandb_init.py:init():1262] error in wandb.init()
    Traceback (most recent call last):
    File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 1252, in init
    return wi.init()
    File “C:\Users\serigne_admin\AppData\Local\Programs\Python\Python38\lib\site-packages\wandb\sdk\wandb_init.py”, line 844, in init
    raise error
    wandb.errors.errors.CommError: failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}
    2024-10-14 11:16:40,147 WARNING MsgRouterThr:15652 [router.py:message_loop():77] message_loop has been closed
  2. Debug-internal.log:

{“time”:“2024-10-14T11:16:39.6571469+02:00”,“level”:“INFO”,“msg”:“using version”,“core version”:“0.18.3”}
{“time”:“2024-10-14T11:16:39.6571469+02:00”,“level”:“INFO”,“msg”:“created symlink”,“path”:“F:\ToothGroupNetwork\wandb\run-20241014_111639-i5vb5pi5\logs\debug-core.log”}
{“time”:“2024-10-14T11:16:39.667464+02:00”,“level”:“ERROR”,“msg”:“dialing: google: could not find default credentials. See … for more information”}
{“time”:“2024-10-14T11:16:39.9300979+02:00”,“level”:“INFO”,“msg”:“created new stream”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:39.9302162+02:00”,“level”:“INFO”,“msg”:“stream: started”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:39.9302162+02:00”,“level”:“INFO”,“msg”:“writer: Do: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:39.9303154+02:00”,“level”:“INFO”,“msg”:“sender: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:39.9303154+02:00”,“level”:“INFO”,“msg”:“handler: started”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.0765827+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:403,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.0765827+02:00”,“level”:“ERROR”,“msg”:“sender: upsertRun:”,“error”:“failed to upsert bucket: returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}”}
{“time”:“2024-10-14T11:16:40.1472315+02:00”,“level”:“INFO”,“msg”:“stream: closing”,“id”:“i5vb5pi5”}
{“time”:“2024-10-14T11:16:40.2625221+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:403,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.2697548+02:00”,“level”:“ERROR”,“msg”:“sender: sendConfig:”,“error”:“returned error 403 Forbidden: {“errors”:[{“message”:“permission denied”,“path”:[“upsertBucket”],“extensions”:{“code”:“PERMISSION_ERROR”}}],“data”:{“upsertBucket”:null}}”}
{“time”:“2024-10-14T11:16:40.2711115+02:00”,“level”:“WARN”,“msg”:“No requirements.txt found, not creating job artifact. See […”}](…
{“time”:“2024-10-14T11:16:40.2711115+02:00”,“level”:“INFO”,“msg”:“sender: sendDefer: no job artifact to save”}
{“time”:“2024-10-14T11:16:40.3161557+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:404,“method”:“POST”,“url”:“[…”}](…
{“time”:“2024-10-14T11:16:40.3161557+02:00”,“level”:“ERROR”,“msg”:“runfiles: CreateRunFiles returned error: returned error 404 Not Found: {“errors”:[{“message”:“project bassfaye10/3d tooth mesh segmentation not found during createRunFiles”,“path”:[“createRunFiles”]}],“data”:{“createRunFiles”:null}}”}
{“time”:“2024-10-14T11:16:40.4470144+02:00”,“level”:“ERROR”,“msg”:“HTTP error”,“status”:404,“method”:“POST”,“url1”:“[…”}](…
{“time”:“2024-10-14T11:16:40.4470144+02:00”,“level”:“ERROR”,“msg”:“runfiles: CreateRunFiles returned error: returned error 404 Not Found: {“errors”:[{“message”:“project bassfaye10/3d tooth mesh segmentation not found during createRunFiles”,“path”:[“createRunFiles”]}],“data”:{“createRunFiles”:null}}”}
{“time”:“2024-10-14T11:16:40.4471349+02:00”,“level”:“INFO”,“msg”:“fileTransfer: Close: file transfer manager closed”}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“handler: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“writer: Close: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4472412+02:00”,“level”:“INFO”,“msg”:“sender: closed”,“stream_id”:{“value”:“i5vb5pi5”}}
{“time”:“2024-10-14T11:16:40.4480132+02:00”,“level”:“INFO”,“msg”:“stream: closed”,“id”:“i5vb5pi5”}
Really appreciate your help,

4

I have the same issue on and off as well.
Sometimes after I delete the .netrc file, it works and sometime it does not. Help is appreciated.

5

Hey I was running into the same error message. How were you able to get this resolved?

6

Hi everyone ,

I encountered a similar error, it might be a permission error when you try to initialize WandB in your python script.

When you do:

wandb.init(
project=“My_project,
entity=“name_of_the_users_team”
)

Check what you have as entity, you might change it with your personal one in your WandB profile website.

Go to the account settings in your WandB profile, under the name ‘group’ you should find your entity name to insert there!

I solved my problem in this way, I hope it helped!