Unable to Delete Multiple Runs Simultaneously

W&B Help
1

Hi, I’m attempting to delete a large number of irrelevant runs to free up space, but the operation fails with the following error message:
"

Delete runs?

Delete 2670 runs? This affects all pages, including reports.
Also delete the 11250 artifacts this run created. Any references to these artifacts will be broken.
Any links from these artifacts to registries will be deleted. References to linked versions will also be broken.
"

Error:
"driver: bad connection
An application error occurred.
Click to refresh the page.
"

2

Hi @eyalcohen! thank you for writing in!

What experiment are you trying to delete these runs from? Could you please provide a link?

Could you also please send a screenshot of the error you are seeing as well as the browser console logs after running into the error?

Could you please collect the full browser logs for us to see if there are any related errors in there? That can be done by:

  • Right-clicking anywhere on the page of the website to open the options menu
  • Selecting “inspect”
  • And then going over to the console tab to see what errors you have in there
  • Once you are in the console tab, could you please right-click on it, save all of the logs into a file, and send it our way?

Warmly,
Artsiom

3

Hi @artsiom thank you for your prompt reply.

I’m attempting to delete runs where the “total_model_wer” is marked as “infinity.” These runs were early stopped and are no longer relevant to my experiments.

This platform is very difficult to work with. I’m unable to upload files, and when I try to upload more than one photo, I receive an error message stating that new users are limited to uploading only one photo. Additionally, I hit the maximum character limit per post when I attempt to paste the log files as plain text. Could you please provide an alternative way for me to share the information you requested?

Could you please provide an alternative way for me to share the information you requested?

Here’s one screenshot from the console:

image
image1540×1534 222 KB

4

Hey @eyalcohen! Thank you very much for the screenshot.

If you find discourse community forum hard to work with, you are more than welcome to reach out to us via support@wandb.com, and we can continue our conversation there :slight_smile:

Either in the new email thread you create, or here, would you mind sending a link to a project where you are trying to delete your runs from?

5

I still can’t attach more than 1 photo, even when attaching in the mail and not in the W&B forum.
"

We’re sorry, but your email message to [“incoming+31bb50f54aa2b337ab930c201849bb2d@wandb.discoursemail.com”] (titled Re: [W&B Community] [W&B Help] Unable to Delete Multiple Runs Simultaneously) didn’t work.

Reason:

Sorry, new users can only put one embedded media item in a post.

If you can correct the problem, please try again.
"

6

Ah, I see the misunderstanding. We can go ahead and create a new email thread with you simply emailing to support@wandb.com from your personal email and we can continue the conversation from there.

7

Hi Artsiom,

I’ve already sent you three messages about this issue, but I haven’t received a resolution yet.

Can you please help me solve this problem as soon as possible?

I will also open a new topic from my educational email address as requested.

Thanks,
Eyal Cohen

בתאריך יום ג׳, 27 באוג׳ 2024 ב-23:33 מאת Artsiom Skarakhod via W&B Community <notifications@wandb.discoursemail.com>:

8

Hi Artsiom,

I checked my email account and the account I’m using is linked with this email. Why do I need to open a new email thread?

Thanks,
Eyal Cohen

9

Thank you for your quick follow-up, Eyal.

I have followed up with you via email with the thread you have started. Let’s continue our conversation there to resolve this issue as soon as possible.

10

Hi Artsiom,

Regarding the issue I’ve been having with deleting multiple experiments simultaneously, I’ve already sent you several messages about this, and I’m concerned that we haven’t been able to resolve the issue yet.

I understand that you’ve asked me to open a new email thread from my educational email address, and I’ve done so. However, I’m not sure why this is necessary, as I’m already using the email address associated with my account.

I’m hoping that you can help me resolve this issue as soon as possible. If you’re unable to do so, please let me know who I can contact for assistance.

11

Hi Eyal,

Thank you for your response and hope you are having a great day.

I am escalating this thread to our Manager Anmol to help us look into this concern.

12

Hi Eyal,

Thank you for reaching out again and for your patience as we work to resolve the issue you’ve been experiencing with deleting multiple experiments simultaneously. I understand your frustration and the urgency of this matter.

Firstly, I want to apologize for the delay in providing a resolution. Your feedback is important to us, and I assure you that we are committed to resolving this as quickly as possible.

Secondly, I do want to note that the error *errors.errorString: driver: bad connection you’re encountering upon deleting runs in bulk is a potential locking issue in our DB (MySQL). This might take a bit more time to resolve on or end. However, there is a potential workaround you can use on your end via our API. Example script to parallelize the deletion of W&B runs:

import wandb
from concurrent.futures import ThreadPoolExecutor, as_completed


# Define a function to delete a single run
def delete_run(run):
    try:
        run.delete(delete_artifacts=True)
        return f"Run {run_id} deleted successfully."
    except Exception as e:
        return f"Error deleting run {run_id}: {e}"


# Initialize W&B API
api = wandb.Api()
runs = api.runs('entity_name/project_name', filters={"summary_metrics.total_model_wer": "infinity"})

# Set up the ThreadPoolExecutor to parallelize the deletion process
with ThreadPoolExecutor(max_workers=5) as executor:
    # Submit deletion tasks to the executor
    future_to_run = {executor.submit(delete_run, run): run for run in runs}

    # Process the results as they complete
    for future in as_completed(future_to_run):
        run_id = future_to_run[future]
        try:
            result = future.result()
        except Exception as exc:
            print(f"Run {run_id} generated an exception: {exc}")
        else:
            print(result)

Please let me know if the above script helps.

13

Hi @eyalcohen , I wanted to follow up regarding the above. Please let us know if we can be of further assistance or if your issue has been resolved.

14

Hi @eyalcohen , since we have not heard back from you we are going to close this request. If you would like to re-open the conversation, please let us know!

15

Hi Anmol,

I’m back from vacation and wanted to follow up on the issue I’m having with deleting multiple experiments simultaneously.

I think your solution makes sense, but I’m having trouble filtering the runs correctly. In my case, the attribute “total_model_wer” is logged using “wandb.log”. I’ve tried reading the API documentation, but I can’t figure out how to filter it correctly. It’s not part of the “config” or “summary_metrics”.

Can you assist me with this?

Thanks,
Eyal

16

Hi Anmol,

I’ve made some progress on filtering the runs correctly. I noticed that when I use:

api.runs("mlspeech/gpt_driven_asr", filters={"summary_metrics.total_model_wer": 1})

It filters correctly. This means that the string value “infinity” doesn’t work for some reason.

Can you please assist me with this?

Thanks,
Eyal

17

Hi Anmol,

I’ve made some progress on filtering the runs correctly.

After some debugging, I finally found the problem. When you store the infinity value (logged using float(“inf”)), you save it as a string but with capitalization. This is not the convention for storing infinity. If you choose to store it as a string, you should store it in lowercase. Use “infinity” instead of “Infinity.”

Fixing your suggested code, I finally succeeded in deleting the irrelevant runs.

Thanks,
Eyal Cohen

18

You’re correct, it should be not saved as a string. I’m glad though that you succeeded in deleting the irrelevant runs.