Jump to content
Force Gaia

What is the best way to protect >8TB of media data when using a gaming motherboard?

Recommended Posts

Posted (edited) · Original PosterOP

EDIT Jan 8 2018: The plan has changed quite a bit, so read my replies on the 31st Dec and 8th Jan to see the current state of my progress on deciding. The OP post had had some edits to allow a better picture of my situation.

 

This is a copy (with edits) of my old thread that got no responses, likely as it was in the wrong subforum.

So for a new gaming build I'm considering, I'd like to have a lot of storage as it'll also be used for some minor content creation as well as storing my media (and serve it to others via SMB and DLNA) - this pc truly will be used for anything I require of a pc. It will not be an always-on server; it'll only be on when being used directly, or when required to serve content.

The issue I have is storage of the aforementioned games, media and created content.

I already have 9TB across 4 drives in my current rig without any sort of RAID or backup, and all of them are getting old and full, worryingly so on both counts. The size of this has pretty much prohibited me from making backups which is why the age worries me, as I'm expecting one or more drives to die soon. 8TB of this - comprised of a single 2TB and two 3TB as separate drives in windows - is dedicated to my media collection, this is the important bit I need to protect. The 1TB with my data and games is a lower priority as my important data is on Dropbox and Google Drive, and my game data is in the steam cloud where possible and re-downloading games isn't an issue.

I would like at least 20TB in my new rig, with some protection from drive failure. My original plan was to grab 5x 5TB drives and RAID5 them on my motherboard controller, this was ditched when I realised that onboard controllers are usually crap.

 

I need some protection in case of drive failure, but I can't afford to do any backups or mirroring with this amount of storage (a future plan, but not at the outset). So RAID5 seems to be the best answer I can come up with for now. Not true anymore, I've checked prices and mirroring is a valid option.

 

So my question is: How should I do this?

 

Is what I'm hearing about onboard controllers bunk?

I honestly expect the answer to this to be "You hear correctly. They are indeed crap"

 

Should I get a RAID card, and if so which one?

Ones I'm looking at are pricey, require SAS-SATA cables, don't support the required RAID modes, or say that they are server motherboard compatible only - or even a combination of those. I care the least about the SAS-SATA bit and can afford to spend a bit if need be, but RAID 1, 10 or 5/6 and gaming motherboard compatibility is mandatory. If a model of motherboard is needed, my new board is the Asus ROG MAXIMUS X HERO. I'm also hearing of battery backups for RAID cards, why are these needed, and how much extra are they?

 

Software RAID, will it hurt my performance if my game files are in the RAID? Anything I should know about Software RAID in general?

This is unknown territory for me. I've stayed well away from the Disk manager if I can avoid it, and wouldn't know how to do this in windows without looking it up. But I'm hearing conflicting whispers about software raid and its performance, as well as the impact on the rest of the PC, so I'd like this straightened out, or links to an article about this. I am looking into Storage Spaces in Win 10

 

Am I overthinking this and there is another path I can take?

I am not keen on the idea of an external USB enclosure or a NAS enclosure but will hear Pros-Cons in comparisons to the options I'm already considering. Bear in mind that NAS isn't ideal as I'm not at Gigabit at all points in my network and the hope was this will store more than just the SMB and DLNA served content.

 

When answering assume the following facts:

I run windows, and am not comfortable enough with Linux to experiment with this kind of thing.

I currently have around 7-8TB of media.
My internet upstream isn't suitable to upload multi-terabyte amounts of data offsite.

My home network is gigabit in the same room as the PC, but is megabit elsewhere in the house.

Edited by Force Gaia
Updating situation, and changing the title to better represent the current thought process
Link to post
Share on other sites

Well first Raid is not a backup, secondly Raid 6 is a much better option for protection of data in the event of a failure. 

You have a few options you could run Freenas or Unraid with a HBA or raid card flashed to IT mode. Raid cards are expensive unless you buy them used.  There are lot of different brands out there, adaptec, LSI some LSI cards are rather inexpensive.  Software raid is a viable option but it depends on which one you use and secondly if write speed is important to you some software raids suffer a big write penalty.    USB is not a good idea internal storage is the way to go in my opinion. 

 

Just my opinion.

Link to post
Share on other sites

Id just use software raid. Its the pest option here. The cpu usage is very low(you won't notice it) and its better for proteting data as it can correct bitrot that hardware raid can't do. Its also easier to add drives of different sizes and expand. 

 

If using windows id use storage spaces or snapraid+drivepool, or zfs/btrfs/snapraid/md on linux

Link to post
Share on other sites
Posted · Original PosterOP

Ah yes, I didn't mention OS. This will be a windows box, I'm not up for messing with Linux for my day to day computing tasks.

I am aware that RAID is not a backup, but it's better than what I have at the moment: 4 disks of varying sizes, not backed up, cloned, or with any form of parity. This is why I'm worried and want to have some form of protection. RAID 5 has got to be better than what I currently have, I don't overly care about how many parity disks I have. My aim is that I can survive a single drive loss, and have time to order a new drive.

The other thing I was hoping to gain from RAID was pooling, as the multiple drives bit is a pest as I keep having to check what is on each.

 

The problem I see with software RAID is that I don't have the number of SATA ports on my motherboard that I need. the board I list has 6 ports, but it loses 2 when I use an m.2 SSD. Which is why I was looking at the hardware RAID possibility.

 

I may look at NAS in the end, as megabit should generally be fast enough to serve media until I can get around to upgrading it to gigabit. The issue I see is that many NAS devices seem to require utilities to add data (i may be wrong, but this is from a quick google). I'd much rather use a windows SMB network share and do it that way. I've also seen that pre-made ones to buy seem to be very limited on the number of drives or max hosted space. I could build one, but that's more hardware I'd need to buy, and a whole new batch of research.

 

So I'm thinking of changing my model as it were.

My original plan was: SSD for OS, RAID of internal drives for literally everything else.

My new plan looks like it may be: SSD for OS, a single internal drive for game files and general stuff, plus another solution for the media drives with some form of backup, cloning or parity.

 

The question then becomes what to do with my media. I don't need write performance, as while I will write semi-frequently to it when I add media, it's not time critical. I could have 3 drives on my motherboard and then use some non-raid SATA expansion card to give me more ports and then use a storage space to mirror/parity the data. I do also like the idea of protecting against bitrot and other silent errors.

My understanding of software RAID is that most can cope with an OS reinstall, I can see that storage spaces can, is this true for other software RAID solutions?

Link to post
Share on other sites

Unless you’re going to build a dedicated NAS, I’d recommend buying 1 or 2 larger disks that satisfy your storage need and sign up for an unlimited online backup service (Backblaze?) for backups. 


Desktop: MSI Z370M Gaming Pro | 8700k | 32GB Corsair Vengence 3000 | 1080 Ti | 2x 960 Pro 1TB | Corsair AX760 | Acer Predator X34 | 2x Dell U2415

Laptop: HP ZBook 17 G4 | i7-7820hq | 16GB | P3000 | 1080P | 1TB 600P | 500GB 850 EVO

Laptop: Dell XPS 13 9343 | i5-5200u | 8GB DDR3 | 256GB SSD | 4k Touchsceen

Laptop: HP Elitebook Folio G1 | M7-6Y75 | 8GB LPDDR3 | 600p 512GB SSD | FHD Touchsceen

Server(ESXi): Supermicro | 2x E5-2640v2 | 128GB DDR3 ECC 

Server(ESXi): Tyan | 2x E5-2640v2 | 128GB DDR3 ECC 

Nas: QNAP TVS-EC880 | E3-1246v3 | 16GB | 8x 3TB RAID6 | 2x 128GB SSD (system)

Nas: QNAP TS-EC1679U-RP | E3-1225v2 | 32GB | 8x 2TB 850 Pro RAID6 | 4x 60GB SSD RAID6 (system)

Nas: QNAP TS-451+ | J1900 | 8GB | 4x 4TB RAID6 (offsite backup)

Link to post
Share on other sites

Say no to RAID 5!  It was good when the drives were 20 GB, there is a very real risk of catastrophic failure during rebuild with current drive sizes.

 

Read more here: http://baarf.dk/BAARF/RAID5_versus_RAID10.txt

 

RAID 6 or RAID 10 would be better for data integrity.  The devil in the details of RAID 5 with huge volume sizes is that you don't know there is a problem until a failure / rebuild.  Then you lose the entire raid if you win the jackpot.

Link to post
Share on other sites
Posted · Original PosterOP

@Dark I hadn't considered offsite at all until you mentioned it, but after a moment thinking about it, I realised it wasn't a good idea in my situation. Offsite backups are pretty much a no-no for me. My upload isn't good enough that I'd have it upload nearly 10TB of existing data. Good idea though.

@Paul Vreeland See my last post, I'm thinking about dropping the idea of hardware RAID. On board isn't good enough, and LSI cards that would support RAID 5 where stupid expensive, even more so when RAID 6 came into the picture. Plus in a hardware RAID I'm getting the impression that if something goes wrong, it does so in a bad way and is a ballache to fix.
 

Here's what I'm planning to do:

For now, no backup or RAID. I just want this thing built. I'm buying a single 10TB Seagate Baracuda Pro, and that will do initially; I've coped with no backups for 7 years, a new drive will give peace of mind, and will last until I can get more. (I'm taking a few months)

In the future, I'll buy more of the same drives. I'll use storage spaces in either mirror or parity, as it seems storage spaces will involve the least pratting around to fix anything, allows for expansion, and won't cost me anything extra. The only issue I have with this plan is getting all my data off the single drive temporarily in order to create the storage space when I do get another; my old drives and a ton of externals will likely be used for this, assuming none of them are dead at that point.

Link to post
Share on other sites
Posted (edited)

just get 2 main (however much space you will need)hard drives and run raid 1 on your main machine for your games (and other important things) then make a nas with lots of storage and based around something like freenas and run plex or another media streaming program to share all your media. 

Edited by GDRRiley
changed issues

Good luck, Have fun, Build PC, and have a last gen console for use once a year. I should answer most of the time between midday and midnight.

NightHawk: i5 6600k @4.2, ASUS Z170m-plus, H105, 16gb corsair vengeance LPX, Strix GTX 970 XFX RX 580 8GB, Corsair RM750X, 500 gb 850 evo, 250gb 750 evo and 5tb Toshiba x300

My compute server (remember to add link) HP DL380G6 2xE5520 24GB ram with 4x146gb 10k drives and 4x300gb 10K drives, running NOTHING can't get anything to work :)

WIP NAS Cisco Security Multiservices Platform server e5420 12gb ram, 1x6 1tb raid 6 for plex + Need funding 16+1 2tb raid 6 for mass storage.

PSU White list:

I run the boat topic

 

Link to post
Share on other sites
Posted · Original PosterOP

@GDRRiley I'd rather avoid Plex if possible, as most of my media is in Matroska MKV format - traditionally not a nice file to deal with when it comes to streaming as traditional 'media' but it's good for what I use it for.

I just need a simple windows share, where other PCs can see the files as a folder structure. This is how i currently have it, and would rather it stay that way. This way all I need to do is point Kodi at the share and it can deal with it, as the folder structure and metadata nfo files are in a way it understands, and I'd rather not change a system that works.

 

 

Also, I'm not up to making a NAS, this PC is costing enough as it is, let alone making a second; plus I did say I don't want an always-on box, which is what I think you're angling at.

 

I have said that the games are not important (saves is on steam cloud, games themselves can be re-downloaded), it's the media I need the backup/redundancy/mirroring for.

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Sign in

Already have an account? Sign in here.



  • Recently Browsing 0 members

    No registered users viewing this page.

Last Topics

how to reset bios on laptop sell xbox 360 to gamestop drill bit for stripped screws windows 7 activation fix ubiquiti wireless controller how to get around bios password how to take out damaged screw replace xbox 360 slim hard drive xbox 360 hard drive sata how to sync a xbox 360 controller to a pc xbox 360 3.5mm adapter steam machine gamepad pvp kernel not working


×