RB 6 Cloud sync file locations

Hi,

Thanks for the reply and the suggestion, but I dont think you understand what I am trying to articulate. I am not wanting the file location to change for the songs in RB. RB should always point to source and not the new location in Dropbox, this changes my complete library structure. RB should sync both ways, in Dropbox and sync the files back to the source or the original file. the RB sync makes no sense and its no different than just having my entire library in Dropbox. This feature is good until RB decides to change my source location of the files. I don't think much thought was put in this and I will not be able to use it until this a proper sync feature. Also, when i tried to revert or turn the sync off, I removed the files from Dropbox and tried to do the auto relocate, the auto relocate feature looked the references to the files in the hidden drobox folder as shown in my image. I would never get the auto relocate feature enabled after it added the referance files. I had to delete the files from the collection and re-import from the rekordbox.xml which I had a backup of. 

This feature really needs to be thought out properly end-to-end. 

If you moved the entire music folder to Dropbox, it wouldn't affect your library structure at all. All the files and folders would be intact in the same directory tree you have now - the only difference is they'd be within Dropbox.

If you don't want them in Dropbox, they can't be in the cloud.

Can I ask why it matters where the file is stored?

Hi,

I think I forgot to mention that my media is on an external drive, not in the laptop. I could probably achieve this by pointing Dropbox to an external drive, but not sure if that is suggested. also my library is about 48K large as well. I have a dropbox which is big enough, and space is not an issue, its just this whole workaround is the issue. As I said, I hope to see changes to this workflow for cloud sync, and more options. thanks for your help though

You could certainly add your external drive as the Dropbox location to avoid moving the songs internally; that would avoid some issues.

I noticed when adding the files to the cloud, it stores them in the rekordbox and then the folder structure is created based on the artist. If I do move the files to Dropbox in my current structure, when I do a sync with any of the files, will it still copy/move them in the above structure? If I copy I will have duplicates in Dropbox, if I move then will RB move the files to the Rekordbox folder under the artist subdirectory? Also, is there a way to change the this setting on the directory structure it uses.

No. As I said above, rekordbox recognizes that they're already there and doesn't need to do anything with them.

Ok, I think I will try this, I will have my files backed up to cloud this way as well. I will try it will some files and comeback with my findings. Thanks for your help. 

No problem!

HI, 

Ok, so this worked! Now, One requirement for Dropbox is to have the external file to be an APFS file format not FAT32 or exFAT, which my drive is, I can covert it to apple format and then try it. But my question is how do I load the entire library back into the collection with all the playlists? do I have it Auto locate, (which I have seen an take a while), any suggestions?

Sorry, you mentioned to have RB relocate, I have 48K songs in my library; (Yes, I know its too much, but Indian weddings and people ask for all random shit) that may take a very long time for RB to relocate. is there a way to bulk edit a xml file and import?

I'm in the same boat; 52K songs here!

The relocation process is actually pretty quick because everything is moved in the same scope; see here. 

Another thing I just figured out while testing this,

Below is my current structure

1. The files I added to the cloud sync, when I selected the option to Move them back to local storage, it removes the file from Dropbox location (Dropbox asks whether I want o remove the file and I click Move out of Dropbox) and adds the file to another location in my Music folder

2. When I select that same file to sync to cloud again, it it moves it from the last destination above to the rekordbox/unknownartist folder in Dropbox, Not my initial location in /djSameer/Music/mp3/Test Song, and the Files also stay in my Music/PioneerDJ folder

3. When I Click again to Move to local storage, then RB moves the files to its original location in Dropbox where it started, leaving the files orphaned in /Moved from Cloud folder 

Ok, I think you have to admit that this is a bug. 

Hi, 

Earlier in our conversation we talked about having dropbox point to an external drive. I tried to test with Dropbox and as I suspected, it does not work well with external Drives as this article mentions https://www.dropboxforum.com/t5/Dropbox-installs-integrations/Syncing-Dropbox-folder-to-external-hard-drive-Mac/td-p/221957

In my test I had to reboot my mac about twice to because it locked my external. One other requirement for Dropbox is for the external drive to be an APSF Apple format. So I tested with a new drive formatted in APSF. 

There is a work around to not launch Dropbox on startup but its really impractical to keep track of when doing gigs, and using the computer without the external drive on the go. How ever, Google drive and some other cloud spaces are a little more favorable towards external drives. 

It's not a bug that the file structure changes; rekordbox is migrating the files into a consolidated location. That is expected.

As for the external drive & Dropbox, it's entirely up to you as to how you manage Dropbox. There are a few tools such as Boxifier, or MountainDuck. Personally, I use CloudMounter and manage it that way.

I think with the moving back to local option, it is expected that it would go back to its original location, no? I think this has to be thought out and tested more with some realistic use cases. I cant risk my library (which I have maintained for years) to be all over the drives, specially orphaned. This should be looked at for a better end-to-end solution. 

I dont know how CloudMounter will help as it seems like a facade to DropBox, I will have to test it out. 

Also, the above issue happens if my library is located in Dropbox (ie. not via cloud sync, but library moved to Dropbox, and Dropbox pointing to external drive) .

(In this scenario below, Dropbox is NOT set to my external drive as above examples)

I just tested with a few songs in my currently library in my external drive, RB did not move them to the Moved from Cloud folder. It worked as expected. So the issue is if you move your library to Dropbox, and do the cloud sync and try to move to local storage*, it moves the files to a temp Moved from Cloud folder in /user/Music/PioneerDJ/Moved from Cloud folder. And if those files are re-synced with the cloud again and move to local storage*, they are pointed back to the Dropbox folder library, and the files are also remained orphaned if you selected the Copy Option. 

*Here "move to local storage" I am referring to the option in the Cloud Library Sync --> Move to local storage" menu option.

I think this scenario needs to be looked at. It would be great to store all my library in Dropbox or any cloud and be able to sync and un-sync without this run around to the temp holding directory. 

I hope I am making sense. Sorry if I am a little anal about this, outside of being a DJ I am also a Software Enterprise Architect, been in the software industry for over 20 years. 

Regards. 

Hi Pulse, 

So I tried to do what you had suggested me with regards to Auto Relocate and I'm stuck with some files not being able to relocate. Itseems that RB is looking at some refrence of the files and the Auto Relocate Option is no longer available. I also tried the File --> Display All Missing Files option an it does not seem to find the files. My new structure is in Dropbox on the external drive. It did find some files but not all. I am stuck and the only option is to restore to previous state. 

Hi,

I keep finding thing and realizing some of it is the environment I have (which I need to clean up) as I figure out in depth the collection I have had for years. But, it also seems that RB6, when doing the Auto Relocate has issues with duplicate files when its traversing through the files if they are the same name in different locations. Please see the images below:

However, it did Auto Relocate most of my 48K collection, has issues with only a few (354) files which in most case seem to be duplicates. 

Please let me know what you think. 

I am having this exact same issue and Pulse, your answers really aren't adequate. The cloud sync is a complete mess and has made my library unsusable.

Why not have Rekordbox function as normal with an external drive plugged in, and only use the cloud copy when necessary? I want to have my entire library in the cloud, but I don't want to have to download all the songs every time I export to a thumb drive when the original files are still available locally.

I had no idea this was going to change the way my library functions day-to-day, and if I'd have known that, I would have never attempted to use the cloud. Please provide a real solution.

@Christian > That's how it works now - you CAN keep your music on an external drive (as I do), and then only copy the music into the cloud (and thus to Dropbox) as needed. 

This will allow your DATABASE to be on the rekordbox server, but only the songs you indicate for cloud status would be uploaded.

One request I made early was to have a "pull" feature so the music didn't necessarily have to be in Dropbox, but then a cloud-sync'd device (eg your iPhone), could request the song which is in the database, then the host computer would add it to Dropbox automatically.

Appreciate the quick response. I guess I'm wondering where my process goes wrong then.

I saw it mentioned above that the interaction between Dropbox and external drives could cause issues. Is that the problem and reason why the file paths for every track that uploaded to the cloud now point to Dropbox? It seems the external drive has been completely cut out of the equation, and that's what I want to figure out.

My end goal here is to have my Rekordbox run no differently than before, with files stored on my external drive and all file paths pointing to said drive, but to also add an identical cloud copy of the library in Dropbox that enables me to access them on-the-go. Here's my process:

1. Library in Rekordbox working perfectly with files housed on external drive

2. Enable CloudSync and select the "copy" option rather than "move"

3. Select All on entire collection, right click, and upload to Dropbox.

It starts the syncing/uploading process, but eventually, the disc space on my computer (not external drive) becomes full and the process stops. This is what makes me think the issue is the aforementioned Dropbox/external drive incompatibility, because local disc space should not be involved at all...right?

Thanks in advance for the help. It's possible that the way I'm trying to use CloudSync is simply not it's intended use. If that's the case, I can adjust, but wanted to confirm with you before changing up my workflow.

Hi Christian,

So, Here is what I have found so far while working on this issue/workflow. I'll try to explain as much as I have understood about this and Pulse please help me if I am wrong anywhere. 

1. The above mention about Dropbox and external drive issue is correct, Dropbox does not behave properly when it is synched on an external drive. It does work, but the issue is when you try to remove the drive, you have to make sure that you exit Dropbox first before you unmount the external drive and DO NOT start Dropbox on startup. Personally I would only have Dropbox on if I were synchronyzing to backup my songs or make them available in the cloud for RB to pick up. Else it's just like having the files on the External drive to where Dropbox happens to be looking for synchronization. 

2. When you have Cloud Sync turned on, RB sees that its the path to Dropbox and adds the Dropbox Reference tag, if you also notice you will see that the Cloud icon in the cloud sync column be a solid icon and you dont have to have Dropbox running if all the files are in the Dropbox Cloud and RB thinks All the files are already Cloud Sync'ed for RB. When you turn off the Cloud Sync, the file location goes back to the full path of the file.

3. Now with that said, the above was with Dropbox being on the external drive. In your later scenario, you mention about syncing the files where dropbox is on your laptop HDD. when you sync with RB Cloud Sync, it will copy/move (I would select copy in your case since you dont want the directory structure of your songs on the external drive to be touched) base on what you select, it will create a rekordbox directory on Dropbox on the local HDD and copy the files based on the artist subdirectories, something to what iTunes does. This will fill up your local HDD. You can select Move to local storage from the RB Cloud Sync menu and it will change the reference in the location path back to the original path of the file (aka, your external target). 

My recommendation and this is what I have done after 4 days of continuous struggle with this is :

1. Backup all of your media from external drive to another external drive using rsync ( I can give you the commands if you are on a mac).

2. Format your main media file with AFPS (if apple, Dropbox requires this format).

3. Point Dropbox to the external Drive to what ever folder you want and Turn OFF Dropbox on startup.

4. Move all of your media to the Dropbox Folder (Dropbox will start to sync, You may want to increase the upload in the networks option to no limit as it limits the upload to only 10KBps be default). 

5. Open Rekordbox, turn on Cloud Sync and Add the new locations only to where your media are as custom locations (you can add multiple locations) in the Auto Relocate Search Folders. 

6. Restart RB 6 ( I have noticed this it does not pick up the settings after I save)

7. Select File - Display All Missing Files. and Auto Relocate them (depending on you collection it can take some time). 

If you see my above post, if your library/collection has multiple files that are the same name but in other folders, RB6 will have issues and will add a reference to your location such as .vol/2343423 as seen is some of my images above. They might be duplicate names or files that could not be found. This then is simply a way for you to work on your library to fix it manually. 

So jut to give you an example, I have 48K songs in my library, and once I went through the Auto relocate, it had issue with about 300 some files, which is not bad as I am seeing this was an issue with duplicates and same file names on other locations; my collection has been going on for 18 years now. this for me needed to be corrected anyways. i have also tested with the mobile device, as long as the song is in Dropbox cloud, it will download the file and you can modify in RB mobile if the RB sync is on, it will update the file on your system of your changes to Que points Grid etc, you dont need to upload the file back again. I will only sync with Dropbox when I have new material to upload to the Dropbox Cloud. 

NOTE: What ever you do, DO NOT select Move to local storage from the Cloud Sync Option, as it drops the file to a temp location in /Music/PIONEERDJ/Moved from Cloud and it remains there orphaned. You have to resync twice to get them to reference in RB again (Read my above issues). As long as you have the files in the external drive in the place you want them to be you should be OK, 

Hope this helps. 

Extremely helpful. Thanks for taking the time to write that up. I'll give it a go using these instructions tonight and report back.

Also, those rsync commands would be a huge help if you don't mind sharing them. Again, thanks so much for your willingness to help.

Hey, Christian,

I'm glad to help, here are the rsync codes for my purpose, you might want to check for other options on what you want to do.

sudo rsync -r -t -v -u -h -O --progress --delete --exclude='/*/.local' --exclude='/Volumes/7SUR_DISK1/.TemporaryItems' --exclude='/Volumes/7SUR_DISK1/.Spotlight-V100' --exclude='/Volumes/7SUR_DISK1/.Trashes' --log-file="/Users/USERNAME/rsync.log" '/Volumes/7SUR_DISK1/' '/Volumes/7SUR_DISK4'

remember above to change the source and target directories to fit to your environment. and the correct Username path for the log file.

Hi Chritian,

Also remember if you maintain your songs or media in iTunes/Apple Music, that will not work anymore since you are moving all your media to another location, if you need help with that i can help as well. 4 hours with Apple support and they could NOT help, I was able to get this resolved as well, its very simple. 

@Christian,

Let me know how things are going for you on this so far.

I've run into some new issues in trying to get it all resolved. Because it was corrupting a bunch of my tracks, I completely disabled CloudSync with the goal of simply restoring to my previous state, with my library working well in Rekordbox 6, but without CloudSync enabled.

I've run into tons of issues. First and foremost, Rekordbox now only recognizes ~8k of my library. The remaining 18k still have a filepath pointing to a nonexistent Dropbox folder. I have RB relocate the files on my external SSD and it finds most of them (the remaining not found seem to be in 'volume' folders you reference above, in total about 200 remaining that it couldn't find, but that's not a huge issue).

Problem is, when I restart Rekordbox, the 18k are once again missing files. Their file path stays the same, but adds a nonexistent Dropbox folder to the file path.

Any idea on the quickest way to resolve this? Once I get back to normal, I will attempt the cloud again using your instructions above. Thanks again for all your help.

Here is my suggestion, since RB 6 environment is not in proper state, I would either restore the RB 6 database from previous backups (Time Machine) if you made a backup of your library before working, or, completely remove RB 6 library/database and start over again by importing from RB 5. 

You will find the RB databases and library in /Users/USERNAME/Library/Pioneer/rekordbox

you can connect with me at djsameer@7sur.com outside and I am ok to help.