54 posts in this topic

Last night, I discovered that one of my source code repo's had been lost. 
Today, I confirmed that I had no backup of that repo.
I am feeling poorly now because I know that it was my fault. I should have had a backup process in place to prevent this type of loss.

So now I am wondering, what is your data backup process? How do you ensure your data is not lost in the worst case scenario?

 

Share this post


Link to post
Share on other sites
35 minutes ago, zeke said:

Last night, I discovered that one of my source code repo's had been lost. 
Today, I confirmed that I had no backup of that repo.
I am feeling poorly now because I know that it was my fault. I should have had a backup process in place to prevent this type of loss.

So now I am wondering, what is your data backup process? How do you ensure your data is not lost in the worst case scenario?

 

Which platform ? Windows ? Linux ? OSX ?

 

But rsync is a pretty good way if you use the correct strategy. This can be done on Windows too using the rsync application Deltacopy. Which runs services, and will automatically sync your watched directory, to the backup location.

bluehash and zeke like this

Share this post


Link to post
Share on other sites

All three platforms.

I have been trying to use freefilesync but it isn't automated.

I have bookmarked Deltacopy because I have never heard of it before. Thanks!

Share this post


Link to post
Share on other sites

As far as my process. I avoid RAID arrays like the plague. RAID in home systems is an additional point of failure, and is a recipe for disaster. So, I use USB 3.0 external drives. Just bare drives, well partitioned and formated of course. And I just hand copy important files to these drives. As for my code base. Well for my beaglebone I have a dedicated Debian system that acts as a NFS, and Samba file share. This means I can mount the NFS share on my beaglebone, and through Samba I can mount the file system on my Windows system. From here, I edit files directly off this share, in windows, using my text editor of choice.  Anyway, the files stay on this debian support system of mine, and never go away until I delete them .

Share this post


Link to post
Share on other sites
5 minutes ago, zeke said:

All three platforms.

I have been trying to use freefilesync but it isn't automated.

I have bookmarked Deltacopy because I have never heard of it before. Thanks!

It'll be a steep learning curve figuring it out. But once learned it's really awesome. Actually, I've known of it for years, and have used it a few times, and still do not know everything about it ;) There's guides online though.

As for Linux, use a systemd timer to fire off once a day or whatever, that calls a script to run your rsync's. If not using systemd, then use a cronjob once or so a day to fire off that same previously mentioned script.

zeke likes this

Share this post


Link to post
Share on other sites

LOL sorry I keep thinking of things that I've long forgotten about, until just ow thinking about all of this. I've heard tale of people setting up a local git repo, and using that to backup files too. The cool thing about using git in this manner, is that you would have file versioning too !

Share this post


Link to post
Share on other sites

I had git setup on a local linux box and I was using it.

It suffered disk corruption when the power went out while I was away on Christmas Vacation.

I am presently running efsck on it. It sure is taking a long time to finish.

Share this post


Link to post
Share on other sites
1 minute ago, zeke said:

I had git setup on a local linux box and I was using it.

It suffered disk corruption when the power went out while I was away on Christmas Vacation.

I am presently running efsck on it. It sure is taking a long time to finish.

Ok, so in that case, all you really need to do is rsync your git repo to another location. Some place *maybe* that is only mounted while backing up, and unmounted while not. This way perhaps you would avoid corrupting your backup. I'd have to think on this some more, to see if there is any holes in my idea here, then perhaps come up with a bulletproof strategy.

 

Share this post


Link to post
Share on other sites

I was wondering if a private github repo might make sense.  I don't like going offsite with my stuff so I'm not sure how to respond to my emotions just yet.

Share this post


Link to post
Share on other sites
2 hours ago, zeke said:

I was wondering if a private github repo might make sense.  I don't like going offsite with my stuff so I'm not sure how to respond to my emotions just yet.

The company I work for uses a private github account, and it seems to work great. I also think the guy I work with who manages the account says it only costs like $20 a month too. But this is for business. For personal use, I think it's a lot less. Couldn't hurt to check.

In either case. For a solid backup strategy, it's always a good idea to have offsite backup's too. In addition to local redundancy.

Share this post


Link to post
Share on other sites
Quote

So now I am wondering, what is your data backup process? How do you ensure your data is not lost in the worst case scenario?

Mac OS X: TimeMachine and SuperDuper!.

Every few days I plug in the external USB drive, and TimeMachine does its thing (backing up the files that changed). When that's done, I use SuperDuper! to make an image backup to a different partition on that same external drive. Once a month I plug in a different external drive (a WD My Passport Ultra) and clone the main drive to one of three partitions on the external - it's round-robin so I've got three monthly snapshots, plus TimeMachine, plus the other backup. I also have an old FireWire drive that gets an occasional copy of the main drive.

I should store them in different locations, but they're all in the same room.

For your case, I think TimeMachine could have helped, especially if, unlike my setup, you let it backup to a drive that's always connected - you would have had all the versions at your fingertips.

One more tip: always, always verify that the backups are good -- boot from them if they're supposed to be bootable. This is something that I'm lazy at, and I know one of these days it's going to bite me.

Share this post


Link to post
Share on other sites

I think it's important to keep OS and data separate always. So maybe once, after setting up a fresh install, make a bootable image of the OS partition. But after that, always keep your data separate from the OS. With Linux, I usually mount /home or /home/username on a separate partition. In windows I create an OS partition( ~ 500GB or something ) and partition the rest as a pure data drive.

Share this post


Link to post
Share on other sites

Ugh! It just gets worse the deeper I look into what data was lost.

I lost all of my altium files on my marquee clock project.  

All I have left over are the gerbers that I sent to the board fab house because they have them stored on their server under my account.

Sigh.

 

 

 

Share this post


Link to post
Share on other sites

Bummer is certainly an understatement.

As to my local backups I run Macrium reflect pro (now V7 home edition). I have all of my code and eagle files on a 1TB 10k  rpm hard drive. Macrium does a full backup every day around 02:00 to an internal 5TB drive, and then copies to my server's 24TB JOBD array.  Finally, the source files get posted to my AWS account for "offsite" storage.  My AWS account runs about $2.00/month. And the last upgrade for my 5 licenses for Macrium was about $150.  Money well spent.  All of the machines on my home network run daily backups. In general, I do a daily full system image for the boot drives, and then the data files are incremental folder/file backups.  I generally keep 8 full backups of my system images, and for the data files, there's a full backup done weekly, and I keep four full backups + the incrementals for a month.  FWIW, Macrium's Windows PE rescue disk saved my butt last week when the system files became corrupted due to a problem with my aging RAM.  I was able to recover the system and get completely back up and running after a day's effort (about 200 reboots while playing with BIOS HD settings...)

zeke and dubnet like this

Share this post


Link to post
Share on other sites
17 minutes ago, NurseBob said:

Bummer is certainly an understatement.

As to my local backups I run Macrium reflect pro (now V7 home edition). I have all of my code and eagle files on a 1TB 10k  rpm hard drive. Macrium does a full backup every day around 02:00 to an internal 5TB drive, and then copies to my server's 24TB JOBD array.  Finally, the source files get posted to my AWS account for "offsite" storage.  My AWS account runs about $2.00/month. And the last upgrade for my 5 licenses for Macrium was about $150.  Money well spent.  All of the machines on my home network run daily backups. In general, I do a daily full system image for the boot drives, and then the data files are incremental folder/file backups.  I generally keep 8 full backups of my system images, and for the data files, there's a full backup done weekly, and I keep four full backups + the incrementals for a month.  FWIW, Macrium's Windows PE rescue disk saved my butt last week when the system files became corrupted due to a problem with my aging RAM.  I was able to recover the system and get completely back up and running after a day's effort (about 200 reboots while playing with BIOS HD settings...)

A kindred spirit!

Multi-level, multi-device backups, bravo!  :biggrin:

Btw, 100% agree on Macrium. It is a great product. It also makes drive upgrades, or switching from spinning drives to SSD, a breeze.

Share this post


Link to post
Share on other sites
Just now, dubnet said:

It also makes drive upgrades, or switching from spinning drives to SSD, a breeze.

Yep.  I've been though a couple of cycles of boot and data drive upgrades.

I love being a kindred spirit! :)

My primary workstation is an i7 x980 with 24GB ram and 7 hard drives (21TB total storage).  I built this machine to do viideo and photo editing.  Even at 6+ years old, it will render an hour of 1080p HD video in just under an hour. Everytime I think about upgrading, I find myself looking at Xeon processors and about 4 - 6 grand to get any significant improvement in performance.  Since I really only do the video for non-commercial stuff, it's not worth the extra cost.  So, since the system is responsive, meets my current needs, I don't get too caught-up in getting the latest and greatest - though I am wanting one of the Wacom Cintiq HD tablets. Maybe for Christmas??? :)

Off topic: I've found that having multiple monitors and running two instances of CCS or IAR an interesting and productive way to debug comms between a pair of '430 devices.

zeke likes this

Share this post


Link to post
Share on other sites

Wow! My head is spinning trying to understand your description of your backup processes. 

It's cool though. It looks well thought out.

My challenge is trying to come up with a process that will work on windows, mac, and Linux simultaneously.

I am reading about AWS S3 services. It's bewildering because of all the options available.

It would be cool if there was an application that could run on windows, mac, and linux that would orchestrate the backup process.

NurseBob likes this

Share this post


Link to post
Share on other sites

 

@NurseBob

For your video work where is the bottleneck? If it's the drive subsystem then perhaps an upgrade to SSD would be helpful. Perhaps not for the whole 21TB but maybe as scratch drives for the rendering process.  Prices keep dropping and are now at the point, in my opinion at least, where the price difference between spinning drives and SSD is pretty easily justified by the performance gains. I have them in my laptops and main desktop and love the performance boost. 

In the past I have recommended adding memory, to a point (and you are north of the point with 24GB), as a way to inexpensively increase performance. Now, its a toss up between memory and SSD and I am leaning more toward the SSD and not solely due to lower prices. Reason is that even with less than optimal memory, the swap to virtual memory on the SSD is so much faster it mitigates the need for more system memory.

NurseBob likes this

Share this post


Link to post
Share on other sites
1 hour ago, zeke said:

I am reading about AWS S3 services. It's bewildering because of all the options available.

RE: AWS S3, I'm running with their simplest service - basically a dropbox type of situation.  AWS S3 suffers from a more complex interface than dropbox, but fits my needs.  When I was actively posting to my video blog, I used their service to store the downloadable files rather than paying Network Solutions for exorbitantly priced storage.

As to cross platform single backup solutions, I've not researched that, so I've really got nothing to offer there.  However, if your machines are co-located, you might consider setting-up some type of NAS storage that everyone writes to, and then upload those backups to a cloud-based option?

While my original plan was to upload all of my backups, they end up in the TB+ range, and uploading a TB of data for one day's worth of data would take days... Losing proposition. So, I limit what gets sent off site, and pray that having multiple locations (basement, and a couple of other locations at home), that my data will survive a local disaster. Bad choice, but I've yet to see any TB level solutions that involve consumer-based internet speeds. Fingers crossed in that regard.

Share this post


Link to post
Share on other sites

@dubnet

> For your video work where is the bottleneck?

CPU.  Disk and RAM are not a problem. HD video processing is very, very CPU intensive, and the graphics accelerators and GPUs really don't participate in the rendering process. Overall, I'm not at all unhappy.  I've seen other systems that take easily 8 to 10 times longer to render files similar to mine.  Aside from CPU, the real bottleneck is uploading to youtube, or any other service. A 90 minute video takes about 4-6 hours to upload. On more than one occasion I've had both a render and upload running. No stress for the machine (all twelve threads will be at 40-70% for the render, but the upload really doesn't register on the resources).

FWIW - the long videos were my recordings of lectures for my nursing students; they were subjected to 3 hour lectures on a weekly basis. Otherwise my goal is less than ten minutes for a topic of interest... :)

 

tripwire likes this

Share this post


Link to post
Share on other sites

@zeke

Do you use any version control software?  For me, that's yet another repository, though it's still locally stored on my home server. I should probabkly be backing that up to the cloud too...

Share this post


Link to post
Share on other sites
4 hours ago, NurseBob said:

Off topic: I've found that having multiple monitors and running two instances of CCS or IAR an interesting and productive way to debug comms between a pair of '430 devices.

...and a third monitor (or laptop/PC) for the logic analyzer watching the comms.  :)  I can truly relate as I had two laptops and a PC in play debugging comms not too long ago.

It is somewhat interesting that we need a handful of computers to play with very inexpensive MCUs.

Share this post


Link to post
Share on other sites

@NurseBob

I had been playing around with github but not every program that I use can use github i.e.: Altium.

I may have to setup a subversion server as well.

 

But, I am trying to not depend on an external cloud storage option so I went to a local supplier today and purchased a WD 8TB Red NAS drive today. This will allow me to setup my own cloud storage and sync files against it.

Since I want to set something up locally, I looked for the following options:

  1. Self-hosted
  2. Linux client
  3. Windows client
  4. Mac client
  5. Open source

These are the possible programs I found today:

  1. Nextcloud
  2. Owncloud
  3. Seafile
  4. Pydio
  5. Syncthing

I've installed Nextcloud but I haven't tried it out yet because I've run out of time today.

The important thing will be the automated process of syncing/backing up all the files on the different sources.

 

yyrkoon likes this

Share this post


Link to post
Share on other sites

Oh, I did find one more open source option worth mentioning called Unison.

It doesn't suit my use case but it is a well regarded possibility.

Share this post


Link to post
Share on other sites

@zeke

The programs used are less important than the results. Multiple tier local backup, and at least one stage offsite. So two copies local would be sufficient, but typically 3 is around the best, with at least one copy offsite. Offsite is for natural disaster, fire, etc. If you could somehow keep multiple copies "in house" with surety that at least one copy would survive after a disaster. That would probably be as good as offsite. Most people do not have that luxury however.

So going by what I know works. For Windows systems, you could use Deltacopy to a local *NIX* system, OSX to a local system, using rsync. Then from that local system to some form of remote system AWS S3 perhaps, but I have no hands on with that. Or, you cold stage a secondary system that stores a second redundant copy, then from that to "the cloud".

I would however advise against storing Operating system data on the cloud. The way I handle that is as I mentioned before I get my systems ready for *whatever*. Perhaps even after a fresh install with a minimal amount of tools installed. Then make a 1:1 disk image of that partition, After that, either keep your data on a second partition, or all remotely. That way, if you ever need to reinstall your OS, your data never goes away. Personally, I keep data on a second partition, then on a single local system, for important data. I've yet to have a problem with this method, and I do not use cloud storage, I figure if there is something catastrophic happening where my local systems are. I'm done too . . YMMV.

EDIT:

Oh and right, as also said before. RAID is not usually a good way to go for personal backup. JOBD is not terrible though, as there is no need for parity. e.g. "Just a Bunch of Disks". With parity, unless done properly, and using proper equipment. It turns into a money pit, that is unwieldy to maintain, and deal with in the event of disk failure. Plus, you end up losing space from the parity disks. God forbid, you have a controller fail, that turns out to be impossible to replace. You'll lose your data forever. You be better off investing into one or a few enterprise drives, and storing the important data on those. Using them as single drives.

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now