Yes, SATA connectors suck. I'm surprised there hasn't been a modification to the design or at least materials in the last 20 years. They are fragile. I've broken many in my day, without much effort at all.
I think experimenting is fun. But when someone puts together a 40TB NVMe pool of drives without a plan, I think money could be better spent elsewhere. But my opinion is irrelevant and it's your money to spend, LOL.
Money really isn't an issue, luckily. The server is meant to accelerate most daily activities while centralizing storage. One idea I've had is keeping Steam games on it so that I don't have to install on individual devices. I did make sure TRIM was enabled, but I will have to see how well it performs.
TRIM usually has to happen on the device level, and it's still a sticky situation with most conventional RAID configurations even ZFS. I'm just curious how it holds up over time. Although 32TB usable is a lot of storage, it probably won't rear its ugly head any time soon unless you hammer it with lots of data regularly.
Thanks! This might be the issue. I have a LUKS on my drives. The LUKS has discard disabled by default. Thus fstrim isn't possible. I have now enabled discard on the LUKS and fstrim works. Let's see how many hours/days it needs to trim.
SMR drives work well with TRIM/Discard as long as it has ample powered on idle time. But a 90% full disk won't work well regardless if SMR or not if there's frequently deletes and rewrites due to fragmentation.
How full are the drives? How big are the files and how many? Do you delete files frequently and add new? Does the disk support TRIM and is it active on your PC where it's connected? Do you leave your hard drive connected but idle for a period of time? This is needed for proper cleanup of SMR zones if device supports TRIM.
That is a function of the file system/operating system, not the drive. So you'd have to set up your smart phone to use a recycle bin if it's even possible.
This literally happened to a few years ago. Tossed out some old cables and such from a few large bins that had not been used in easily a decade. One of the cables I tossed I needed two weeks later and had to buy another. I'm a border line hoarder already, and I was so proud of myself for actually tossing stuff out. Be a good 15 years before I do that again now...
Meh, for me, it's worth the peace of mind to get rid of the clutter, and spend $10 on a new cable when I need it rather than spend hours sorting through my rats nest of cables to find the one I need that I probably won't find anyhow because I become discouraged, and buy another one anyhow.
You can use Steam Wallet funds to buy a Steam Deck. It's even on its very first sale right now. Plus, it might incentivise you to play more. Seen it happen a lot with Steam Deck users that are finally clearing some backlog.
Meh, I bought a Steam Deck thinking this, but it's just been collecting dust. I never was very good with game controllers, figured this might be a bit better. But nope.
So even if incrementals are based only on the previous backup of any kind, it would be included in the C backup even though it wasn’t in the B backup since there was no change between A and B? If that’s the case, how does backup C know to include it if it wasn’t in backup B due to no change at that time?
So if let's say i have 2 backups of the same files right now but i nvr did any checksum test n saved a checksum file. I wouldnt know which is the perfect copy?
Do checksum of both sets, compare, and see if any mismatch. If there's mismatched ones, then go through them manually to see which is good and which is bad, if you can even tell.
i'm going to guess you are A. speaking from lack of experience, or B. have not used it in a LONG time, or C. Never figured out proper stripe size vs allocation size.
Yeah I doubt many resellers will happily provide you a full smartctl -x query of drive. It shows life time min/max temp and G-sense shock if the drive has it.
Probably not. But even then I don't believe it shows a histogram over time of temperatures. So you may see it hit 60C but you don't know if for one second or 1000 hours.
Shucking all the way. You should research what the manufacturer has in store at various capacities. Reason being that you really want CMR drives for NAS/raid operation and manufactures often make SMR drives up to a certain capacity limit. IIRC with most it's 8tb but you should check yourself. If that's still the case I'd shuck 10tb drives instead.
Yeah. I'm testing open media vault with snapraid and mergerfs myself right now for my collection of large unchanging files. (Guess I'll have to update my flair when I decide to go all in on it, lol)
That works too! In my opinion, with a large media collection, mergerFS or Drivepool with SnapRAID makes a lot of sense, or even UnRAID. No need to spin up all those drives all the time in a conventional RAID. Not to mention the flexibilty.
I should have mentioned that they are all external drives I am trying to pull the files from. I'm going to try to do this under windows 10 pro. Does rmlint work on Windows 10? Another goal is to make files easier to find too, thus having them on one drive instead of going through each one for a specific photoshoot.
No I bought 100 OneDrive accounts by buying bulk cheap CDkeys and I wrote a program in python to make a virtual disk that I can drag a 10 TB file into and have it be separated into 300 33GB files and get uploaded to OneDrive at 10 GB/s
That's not RAID. That's spanning or pooling. Speed is irrelevant unless you have 100 unique 100 MB/sec internet connections that are aggregated and reads from each one simultaneously to get the the 10GB/sec. Then I'd say color me impressed.
The external drives are just internal drives with a plastic shell around them. The enclosures that come with the drives usually are cheap and don't offer any active airflow, causing them to run hot with extended use. Plus, disks just die. USB also is not the most reliable connection either. Could be the USB to SATA PCB, could be the cable, could be the USB port/controller chip on the motherboard. If it was seeing heavy use any one of those things could have gone bad.
ZFS is fantastic, and I use it in my homelab.
You can't set up RAID 6 in Windows without a RAID card. It is not supported through software.
I'm writing up the whole sad affair but my home machine runs (ran) a RAID 10 array with four 8TB (7.2 effective) vintage 2019 HGST drives.
Yes, SATA connectors suck. I'm surprised there hasn't been a modification to the design or at least materials in the last 20 years. They are fragile. I've broken many in my day, without much effort at all.
I think experimenting is fun. But when someone puts together a 40TB NVMe pool of drives without a plan, I think money could be better spent elsewhere. But my opinion is irrelevant and it's your money to spend, LOL.
Money really isn't an issue, luckily. The server is meant to accelerate most daily activities while centralizing storage. One idea I've had is keeping Steam games on it so that I don't have to install on individual devices. I did make sure TRIM was enabled, but I will have to see how well it performs.
TRIM usually has to happen on the device level, and it's still a sticky situation with most conventional RAID configurations even ZFS. I'm just curious how it holds up over time. Although 32TB usable is a lot of storage, it probably won't rear its ugly head any time soon unless you hammer it with lots of data regularly.
I thought the firmware was locked so you couldn't change the smart data.
It's not supposed to be changed, but it can be with the right tools.
Toshiba 2tb I've been using for last few years us a good buy, at half the price.
That's a hard drive, not an SSD.
I have had a 2TB BX500 for a couple years now and hasn't been an issue. Then again it's just used as my download drive and temporary storage.
Thanks! This might be the issue. I have a LUKS on my drives. The LUKS has discard disabled by default. Thus fstrim isn't possible. I have now enabled discard on the LUKS and fstrim works. Let's see how many hours/days it needs to trim.
SMR drives work well with TRIM/Discard as long as it has ample powered on idle time. But a 90% full disk won't work well regardless if SMR or not if there's frequently deletes and rewrites due to fragmentation.
How full are the drives? How big are the files and how many? Do you delete files frequently and add new? Does the disk support TRIM and is it active on your PC where it's connected? Do you leave your hard drive connected but idle for a period of time? This is needed for proper cleanup of SMR zones if device supports TRIM.
That is a function of the file system/operating system, not the drive. So you'd have to set up your smart phone to use a recycle bin if it's even possible.
This literally happened to a few years ago. Tossed out some old cables and such from a few large bins that had not been used in easily a decade. One of the cables I tossed I needed two weeks later and had to buy another. I'm a border line hoarder already, and I was so proud of myself for actually tossing stuff out. Be a good 15 years before I do that again now...
Meh, for me, it's worth the peace of mind to get rid of the clutter, and spend $10 on a new cable when I need it rather than spend hours sorting through my rats nest of cables to find the one I need that I probably won't find anyhow because I become discouraged, and buy another one anyhow.
You can use Steam Wallet funds to buy a Steam Deck. It's even on its very first sale right now. Plus, it might incentivise you to play more. Seen it happen a lot with Steam Deck users that are finally clearing some backlog.
Meh, I bought a Steam Deck thinking this, but it's just been collecting dust. I never was very good with game controllers, figured this might be a bit better. But nope.
It was never insane.
So even if incrementals are based only on the previous backup of any kind, it would be included in the C backup even though it wasn’t in the B backup since there was no change between A and B? If that’s the case, how does backup C know to include it if it wasn’t in backup B due to no change at that time?
It always has to reference the initial backup.
I was kinda thinking that was what differential backups did though.
Differential always saves changes since the last full backup, and that's it.
So if let's say i have 2 backups of the same files right now but i nvr did any checksum test n saved a checksum file. I wouldnt know which is the perfect copy?
Do checksum of both sets, compare, and see if any mismatch. If there's mismatched ones, then go through them manually to see which is good and which is bad, if you can even tell.
No. It means it will burst into flames any second. Send it to me and I'll make sure to dispose of it properly.
SSDs are more reliable than hard drives, price per gb you can find cheaper SSDs than HDDs, 40 USD for a 1tb ssd is cheap in my opinion for now.
Why run RAID 1 at all? Just make sure to backup your files regularly.
I thought it was a 4 layer with 100gb
No, single and dual layer are 25GB per layer. 100 and 128GB are 33GB per layer:
i'm going to guess you are A. speaking from lack of experience, or B. have not used it in a LONG time, or C. Never figured out proper stripe size vs allocation size.
go help this bloke:
It all comes down to cost and warranty. You can buy good new quality drives on sale for $15-16/TB, most with 3-5 year warranties.
Yeah I doubt many resellers will happily provide you a full smartctl -x query of drive. It shows life time min/max temp and G-sense shock if the drive has it.
Probably not. But even then I don't believe it shows a histogram over time of temperatures. So you may see it hit 60C but you don't know if for one second or 1000 hours.
Shucking all the way. You should research what the manufacturer has in store at various capacities. Reason being that you really want CMR drives for NAS/raid operation and manufactures often make SMR drives up to a certain capacity limit. IIRC with most it's 8tb but you should check yourself. If that's still the case I'd shuck 10tb drives instead.
To date, WD make SMR drives 6TB and smaller, Seagate 8TB and smaller, Toshiba 6TB and smaller.
Is WD EasyStores better than WD Elements? 8 TB WD Elements is $40+ cheaper
They're the same thing.
Sad we haven't got more than 5tb since then.
The latest density of hard drive platters is 2.2TB per platter on 3.5" drives. Assuming you can get same density on 2.5" drives, if you do the math:
+1
Good info, thanks!
So this essentially will zip all files in a folder. No incremental backup or dedup?
Yeah. I'm testing open media vault with snapraid and mergerfs myself right now for my collection of large unchanging files. (Guess I'll have to update my flair when I decide to go all in on it, lol)
That works too! In my opinion, with a large media collection, mergerFS or Drivepool with SnapRAID makes a lot of sense, or even UnRAID. No need to spin up all those drives all the time in a conventional RAID. Not to mention the flexibilty.
Presumably the 9.08TB was the maximum size OP could make for a parity volume on the pool consisting of a 16, 14, 6, and 5TB drives.
Or just ditch Storage Spaces altogether and go with Drivepool or Drivebender. Or UnRAID even.
I should have mentioned that they are all external drives I am trying to pull the files from. I'm going to try to do this under windows 10 pro. Does rmlint work on Windows 10? Another goal is to make files easier to find too, thus having them on one drive instead of going through each one for a specific photoshoot.
I don't see any need for RAID 1. That's just added complexity for no good reason.
No I bought 100 OneDrive accounts by buying bulk cheap CDkeys and I wrote a program in python to make a virtual disk that I can drag a 10 TB file into and have it be separated into 300 33GB files and get uploaded to OneDrive at 10 GB/s
That's not RAID. That's spanning or pooling. Speed is irrelevant unless you have 100 unique 100 MB/sec internet connections that are aggregated and reads from each one simultaneously to get the the 10GB/sec. Then I'd say color me impressed.
The external drives are just internal drives with a plastic shell around them. The enclosures that come with the drives usually are cheap and don't offer any active airflow, causing them to run hot with extended use. Plus, disks just die. USB also is not the most reliable connection either. Could be the USB to SATA PCB, could be the cable, could be the USB port/controller chip on the motherboard. If it was seeing heavy use any one of those things could have gone bad.
What do you use to check/diagnose these?
If you can get any connection, check the SMART status first.