Recovering a Cru RTX800-TR RAID Array

This blog post falls into the “I hope it is useful to someone” category.

I recently had the pleasure of figuring out how to restore a Cru RTX800-TR RAID array that was suspected to be inoperable.

cru-rtx800-tr.png alt text

This thing is a beast, and worth saving.

This was a personal favor for a friend, not related to work.

This is a technical write-up of what I did.

Specs

  • Product: RTX800-TR
  • Interface: Thunderbolt up to 10 Gbps bi-directional
  • Drive Types Supported: 3.5” SATA Hard Drives
  • RAID Levels: JBOD, 0, 1, 4, 5, 6, 10, and ATTO DVRAID (!)
  • Operating System Requirements
    • Windows 7 or 8
    • Windows Server 2012 or 2008 R2 (x64 only)
    • Mac OS X 10.6.8 or later
    • Linux distributions that support the connection type used (LOL)
  • Shipping Weight
    • 25 pounds (without drives)
    • 37 pounds (with drives)

History

I don’t exactly know the full history of this product.

I can tell you that CRU was bought by DigiStore.

But that doesn’t really matter. What matters is the fact that the actual RAID controller is manufactured by ATTO Technology.

Here is what it shows up as in Linux in dmesg:

thunderbolt 0-1: ATTO Technology, Inc. ThunderStream SC 3808E

For the purposes of this RAID array, we need to focus on ATTO stuff, because they make the controller and therefore the drivers.

Plan of Attack

My plan here was:

  1. If at all possible, use native vendor tooling
  2. As a last resort, try re-assembling the raid using mdadm (software raid) by plugging in all 8 drives directly into a PC
  3. As a super last resort try file carving on individual drives to see if there was anything.

This is a 32TB RAID array, unknown RAID level, unknown volume configuration, and unknown filesystem.

First we should probably get some data about what we are working with.

Reading manuals was a great first step, and luckily those were archived online.

False Start: AttoConfig

After reading all the manuals, getting the AttoConfig RAID configuration tooling working sounded like a great first step.

I know a thing about mdadm, but reassembling a hardware RAID using software in Linux is dicey. Especially if this thing is using a custom RAID level (DVRAID). Even knowing the volume configuration level and RAID level sounded like something that would help me immensely, so I first started working on getting the configuration tool working.

Luckily, this software works in linux! You can even download it now from ATTO’s website.

Sure, you need to install Java 8, deal with some shisms, but I got it to run:

attoconfig in linux

Indeed it does see the “ThunderStream SC 3808E”, however right in the download page it does warn you:

ATTO ConfigTool requires a supported ATTO Product. The HBA drivers and firmware are not included with the utility.

And yes, it is true, there is no driver included. The AttoConfig tool can’t actually see anything about this RAID array.

Thinking About Drivers

I’m sure a Windows / Mac driver exists, or existed at some point for this thing, but good luck finding it. I doubt one existed for Linux ever.

But maybe it does exist just under a different name?

False Start 2: Trying Ancient Linux Drivers

ATTO has drivers on their website, even for discontinued products. They have Thunderbolt devices, NICs, and HBAs. Nowhere is this particular product listed.

But, I tried downloading and extracting every driver they had anyway.

I looked for the one that looked the closest, lnx_drv_esasraid2_271.tgz, and looked at it.

Right away, I kinda knew this was not going to work.

The thing is, the readme already says that this driver was upstreamed into linux 3.12 (Nov 2013!). I shouldn’t have to install this driver at all.

But, it did have a attocfg module referenced, which the AttoConfig tool needed to be loaded. Maybe I could build this module and load it?

Bringing this driver up to modern standards was beyond my capabilities. I asked Claude to do it, but in the end failed.

Again though, it didn’t make sense. Linux already has this driver, it seems like it would have worked already out of the box if it was the correct one.

Thinking About Thunderbolt

Thunderbolt is just PCIe over a fancy cable.

I incorrectly thought that I wouldn’t need any special drivers for this at all. I thought that this would be implementing some sort of standard Thunderbolt Mass Storage protocol. (What I had in my mind was Firewire’s mass storage protocol)

But really, it is just PCIe over a cable. What if I looked at the problem as if this was a normal PCIe raid device in my computer? Forget the Thunderbolt part.

Thinking About PCIe RAID Devices

So yea, this is what it shows up as in lspci:

0b:00.0 RAID bus controller [0104]: ATTO Technology, Inc. Device [117c:0067] (rev b0)
Subsystem: ATTO Technology, Inc. Device [117c:4067]

The vendor (117c) is in the PCI Database, but not this specific device (0067). grep revealed that none of my drivers were configured to understand this PCI id at all.

But, screw it, what if this RAID device just got a new number, but was actually something Linux already knew how to talk to?

What does it take to “force” linux to use the esas2r driver I was looking at to try this device ID, even if it isn’t programmed to do so?

Turns out that is WAY easier than I thought:

modprobe esas2r
echo "117c 0067" > /sys/bus/pci/drivers/esas2r/new_id

I didn’t need to recompile the driver at all! You can hot-bind any device like this, even if the modinfo doesn’t officially autoload for that particular device ID.

Incredibly, IT WORKED!

[10937.575362] esas2r 0000:0b:00.0: enabling device (0000 -> 0002)
[10949.174025] scsi host7: ATTO ExpressSAS 6GB RAID Adapter (bus 0x0B, device 0x00, 
IRQ 0x39) driver version: 1.00  firmware version: 1.15

[10949.185340] scsi 7:0:0:0: Enclosure         ATTO     Virtual SES      0001 PQ: 0 
ANSI: 6
[10949.385448] esas2r: request failure - cdb:a0 reqstatus:7 target:0
[10949.385613] scsi 7:0:1:0: Direct-Access     ATTO     Media201800      0001 PQ: 0 
ANSI: 5

There were errors, but the volumes showed up.

Even the userspace CLI tools (not AttoConfig) allowed me to access the RAID controller.

I used the provided atraidcli_x64 CLI tool to access RAID info, drive status, etc:

$ ./atraidcli_x64 -c 1 -x DumpEventLog
Channel 1: DumpEventLog


 ATTO ThunderStream SC  Event Log 

Reading all 23 entries
 0000 00:00:00 SYS  System date has changed. Current date:[01/01/2000]
 0001 00:00:00 INFO Boot/reset due to Thunderbolt link up
 0002 00:00:00 INFO Reset.  Reason: Chip Reset.
 0003 00:00:00 WARN-Target Change on 0
 0004 00:00:00 WARN-Target Change on 1
 0005 00:00:00 INFO [5001086000595c88] for disk [K7J7RGZL            ]
 0006 00:00:00 INFO  discovered
 0007 00:00:00 INFO [5001086000595c89] for disk [K7JGNB3L            ]
 0008 00:00:00 INFO  discovered
 0009 00:00:01 INFO [5001086000595c8a] for disk [K7JGW3TL            ]
 0010 00:00:01 INFO  discovered
 0011 00:00:01 INFO [5001086000595c8b] for disk [K7JH1ZRL            ]
 0012 00:00:01 INFO  discovered
 0013 00:00:01 INFO [5001086000595c8c] for disk [K7JGBWYT            ]
 0014 00:00:01 INFO  discovered
 0015 00:00:01 INFO [5001086000595c8d] for disk [K7JH1VXL            ]
 0016 00:00:01 INFO  discovered
 0017 00:00:01 INFO [5001086000595c8e] for disk [K7JESM6T            ]
 0018 00:00:01 INFO  discovered
 0019 00:00:01 INFO [5001086000595c8f] for disk [K7JGLZPL            ]
 0020 00:00:01 INFO  discovered
 0021 00:00:02 INFO RAID Group state now Online: Media2018
 0022 01:32:23 SYS  System date has changed. Current date:[11/22/2025]

Ready.

$ ./atraidcli_x64 -c 1 -x RGDisplay
Channel 1: RGDisplay

;GroupName       Type   Interleave Capacity Partitions Members Status
;----------------------------------------------------------------------------
Media2018        RAID5  128     KB  25.47TB          1       8 ONLINE

Ready.

It was a good old RAID 5 array, and amazingly NOT degraded! (Although a RAID 5 array of this size is not recommended)

To make this force load permanent, I added this file in:

# in /etc/modprobe.d/esas2r.conf
options esas2r pci_device=0x117c,0x0067

It was a hfsplus (Mac) filesystem that Linux could mount with no problem. I mounted it in ReadOnly mode and handed it back over to the owner.

Conclusion

The fix was easy, but it took me a number of hours to get there.

Luckily I didn’t have to scrounge together docks to get all 8 drives plugged in and experiment with mdadm, assembling the RAID array by hand.

In the end Linux already knew how to talk to this thing with esas2r, it just needed a little hint that this was the same class of RAID controller, and it just happened to be external over Thunderbolt with a different PCI Device ID.

Comment via email