I spent many hours designing the workspace I wanted (columns, graphs, …). Worked nicely (beside a bug that I reported before). Now I run the first experiment on a more powerful machine and… ALL configuration is gone, I am presented with the initial/standard graphs. Please tell me that there is a way to get my designs back. This way I can’t use it.
Hi @hogru,
I’m sorry you had this issue. Was the new run sent to the same project as the other runs where you setup this workspace?
If so, you can change the workspace on the bottom left corner of the UI from default to the workspace you created.
Workspaces currently only apply to a single project but if you need, we should be able to move that run into the original project if you would like?
As a side note, this has been reported as a frustration before and we are currently working to allow workspaces to be used like templates that can be moved across projects. I can add your name to this feature request to help up the priority.
Thank you,
Nate
Hi @nathank,
I log the runs to the same project. And my issue is very basic. I don’t need to copy workspace settings (although
that might be convenient in the future).
On machine A I implement/debug and log to wandb. On machine B (with a GPU) I run the experiment. I have setup a workspace on machine A. After running the experiment on machine B the workspace setting is gone. It does not matter if I am in the default or my personal workspace. It’s as if it was never configured (but I can “prove” it since I did a couple of exports).
I will present the results to a couple of dozens lecturers and master students at an ML institute. The presentation will also cover my experience with wandb and a recommendation. I would really like to recommend it, but if the settings are gone this is a no go.
Even if I did something stupid it should give a warning about deleting/overwriting.
If it makes a difference, I log via Pytorch Lightning. And I have reported a bug (presumably) in the wandb UI a couple of days ago which I have not received a response for. I think it does not relate to this one, but who knows. Just so that you are aware.
Hope you can help.
Best,
Stephan
@hogru I see. The workspace should persist across different machines but it is tied to the username who is signed into the UI. Is there any chance that machine B is logged in as another user?
Also, are you on a local deployment or using our public cloud? If it is public cloud, could you share a link to the project and I can take a look?
Lastly, I apologize that you didn’t get a response on your bug report. Could you possibly share a link to where you posted it and I can address it there?
Thank you,
Nate
Hi @nathank,
thank for your quick responses. As a context information: I am new to wandb, this is my first project with it. So it might be that I don’t use the right wandb terms or do “stupid” things.
ad user name: I am not in front of machine B but I am pretty sure it is the same user X. I can’t remember having added a second user Y. I set up a team though, just to try it out (if that’s relevant). If machine B would log with user Y I guess I would not see the runs on machine A with user X. I see all runs from both machines on both machines.
ad cloud: I use your public cloud. The project is a bit of a mess right now (in terms of logging) but it does not contain any sensitive information and I just made it public: project
ad bug report: mea culpa, I saw that there IS an answer, I was just expecting/used to getting an e-mail after a thread update, which I didn’t. I will deal with my issue in the other thread.
Update: the bug (which was mine) is now resolved; just if you are curious: Logged value available in graph panel, but not in columns - #4 by hogru
Best,
Stephan
@hogru, no problem! And we’re glad you’re trying out W&B!
In the runs table you can see that all of the runs in this project were created by “hogru” but I just wanted to clarify a little about the usernames to make sure we’re understanding each other. When you log a run and give an API key when logging, the API key is tied to a username and will affect who “Created” the run. Regardless of who created the run, you should be able to access the UI from any computer and the workspace will be tied to the user name used to sign into the UI on that computer, not the username associated with the run.
I can see a workspace created by “hogru” that is different than the default. Here is what that looks like:
Is this the workspace you were looking for or is this different than the one you created?
Also, I see that you’ve created a report found here. I just wanted to double check that this isn’t what you are looking for.
Glad we were able to resolve the other bug you were experiencing!
Let me know if either of these were the workspace you wanted or if we have some more digging to do.
Thank you,
Nate
Hi @nathank,
thank you for the clarification concerning the user names. Makes sense. At the moment it is just me (creating runs and designing the UI) but for future work this will be important to understand.
The screenshot you showed is what I referred to as the “default”. Since my modifications are lost I was assuming that the UI was reset to the default. If this is not the default I can only assume it is a very early version of the UI when I started to experiment with it. My “last known good” UI shows additional sections starting with “10”, “20”, … (I refer to those in a presentation).
I created the report as a short status report for my supervisor at the university. It’s not what I am missing.
I attach a sample graph that I had in the UI, just to “prove” that something is gone. I had those until Monday.
So, unfortunately for me this is still unresolved and I still hope to get my Monday configuration back.
Best,
Stephan
Hi @hogru, sorry for the delay on this. Looking at your workspace now, it looks like you have the custom charts back. Did you have to recreate these?
The only way that the workspace should be able to be reset is by going to the workspace dialogue in the bottom left corner and clicking clear workspace. If you don’t think this could have happened then let me know and we can see if anything happened on our end.
Thank you,
Nate
Hi @nathank,
first, a status update:
- I had to recreate (a subset of) the custom charts
- I finished all runs on machine B (Ubuntu) and only then recreated the workspace on machine A (macOS); sure, should not make a difference, but I do not trust the UI / web service at this point
- During the runs of machine B (a number of loops over the same experiment setup to get the std.dev. of the results) the wandb process crashed in 2 out of 5 runs… I did not bother to report it since I needed to finish it and could not parallelize the work (see point above)
- As a consequence, I could not recommend wandb for “serious work” during my presentation to our ML institute
- I experienced wandb with a nice feature set and I like the product, but as a pre-sales consultant (not in the AI field) I would not recommend wandb to paying customers
That said, I like the feature set, I am glad to get it for free in the academic context and I need to decide if/how I will proceed with my usage of wandb from here on. My next and last step is the master thesis and as I said, I don’t know if I can trust wandb with my config data and the stability it needs for experiments that last longer hours.
I understand that “Clear workspace” should be the (only) way to delete the configuration, but as described in my initial bug report, the configuration was gone without me clearing the workspace. I can “offer” an experiment to try to reproduce the situation, i.e. running experiments from machine B (I only touched the wandb UI on machine A since recreating the workspace). I consider it unlikely that we can simply reproduce it, but it might be worth a try. Let me know what you think.
In the meantime, I have a bit more time until I am ready to log new experiments and still need to decide how I will track them.
Best regards,
Stephan
Hi @hogru,
I apologize for the delay in the response. I’m not able to replicate the issue unfortunately. We’ll make a note of this and watch for any other users experiencing a similar problem but I apologize that you lost time spent creating the original workspace.
We do have many user that run long experiments on our platform without any issues but I understand your concerns given the experience you had. If you would like we can try to debug what happened that caused your runs to crash? Was there any particular error you encountered? Linux machines sometimes do not like that we start new processes so often setting the env variable WANDB_START_METHOD="thread" can help resolve these problems.
Again, I apologize for the bad experience you had but I would love to hear your feedback and see if we can resolve your issues.
Thank you,
Nate
Hi @nathank,
I am still finishing the final touches on the work/report that I used wandb for. Until then, (when I get my grade) I will not touch wandb on Ubuntu, I am still afraid to lose work. For my master thesis (already started) I plan to use it for experiment tracking. I will switch between machines, set the WANDB_START_METHOD as suggested and depending on the experience, even do some graphs/reports that I plan to use in the thesis. I have also started to download the logged data as CSV files so that I can do the critical reporting with more conventional methods (matplotlib et al.). Let’s see how this goes, I’ll report back (in a couple of months).
Best regards,
Stephan
Hi @hogru,
I’m excited to hear that you plan to continue with W&B in your thesis work. Feel free to write us here or email us at support@wandb.com if you have any feedback along the way.
Wish you the best with your projects and excited to see how you incorporate W&B in your work!
Thank you,
Nate
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.