IoT Attack, Incident Response

I missed an installment on Friday, and maybe I'm a little tired of blogging about What's Wrong With the World. So here's a taste of something else.

About a month ago, I had what I believe was a ransomware attack on my home infrastructure. I want to lay out what happened and what I have done about it since.

Preparation

One of the main elements of my computing environment here had been a QNap TS-451, running the current version of QTS. For most of the three years + that I've had this unit, it's been forbidden by firewall rules from reaching out to the Internet. This was for two main reasons: one, that its software is largely predetermined by the vendor, therefore out of my control; and two, that the company had suffered in the past from successful attacks on its own infrastructure which resulted in the compromise of customer data.

The fact that the software on the unit was out of my control puts these devices firmly in the Internet of Things category. Which I may have referred to in the past by a more derogatory term. So you understand my reluctance to let the QNap go tiptoeing through the tulips of the 'Net. What you might not understand... what I myself am still wondering about... was my decision about a year ago to relent from that restriction. After all, it required many more steps to update the firmware if I didn't let the unit go get the updates on its own. So... I did.

I took precautions, chiefly by disabling the infamous "QTS Cloud" service. What I did not realize until it was too late was, QNap essentially re-enabled part of that service under the umbrella of its cannot-be-disabled "Help Desk" package. Which is to say, the Help Desk package can nominally be disabled, but QTS re-enables it autonomously. And silently.

identification

One day in mid-June, I noticed several hundred strange files on the home dir of my QNap device. They had names like, "position-right-44-D" and each contained 50 characters of random-looking text. It immediately occurred to me that, if I wanted to obfuscate a ransomware key on the target system for support of the encryption phase of the attack, this is a pretty nice way to do it. This impression was reinforced by the reports I had been seeing online of successful ransomware attacks on QNap devices. These reports were linked to the QTS Cloud service, which I did not use. Or so I thought. The "Help" Desk service was again active on my unit, despite my having attempted to disable it.

Containment

I went straight downstairs and unplugged the network cable from the QNap. Checking all my other devices for the presence of the obfuscated-key files, I found only a handful of them on the cloud storage account where the QNap synced backups for offsite. After removing those, no more of these have appeared anywhere on my still-connected machines.

Containment is a harrowing step of incident response, because you can never be completely sure you have the intrusion contained. You can only asymptotically approach that magical zero.

Eradication

The QNap was primarily serving as a backup destination. As my backup strategy is 3-2-1, it would be logical to assume that there's no such thing in my world as "the only copy" of a given backup. My next move was to determine how redundant my backups really are, by checking around on different boxes in my environment and cloud for copies of everything I considered the QNap the primary source for. As it turned out, I passed this test rather well. Everything important that was on the QNap was also found -- in a current version -- in at least one other place.

This enabled the most straightforward and effective strategy for eradicating malware from a device, the one we know affectionately as, "Nuke from space." I placed the hardware on a workbench and plugged in a USB stick with a live-CD version of Debian on it. I booted straight to the USB device and installed Debian 10.0 on the QNap. (Fortunately, I had previously upgraded its RAM from 1GB to 8GB, an adequate amount for a modern Linux distro.) I was able to overwrite the internal QNap boot device with the /boot partition of Debian. The rest installed onto the main drives which I completely wiped and reformatted for this purpose.

Recovery

The biggest downside of "nuke from space" is that I lost all the data that had been kept on the QNap until that point. But as mentioned above, I had a copy of every item somewhere else. Scattered, but accessible. I didn't even have to go to the cloud for any of it. Restoring the data was tedious, and time-consuming, but ultimately successful.

Lessons Learned

I have taken away some good lessons from this incident.

• Internet-of-things is and always will be Internet-of-crap.
• Make rules that your IoT devices may not make outbound connections.
• If IoT devices don't work under such rules, reconsider owning them.
• If you make a rule that an IoT device may not make outbound connections, never relent from that rule.
• If a device will accept a proper general purpose OS instead of its custom software, consider installing that.

A corollary to all the above is, if you buy a device like a QNap NAS, you should consider whether the hardware alone is worth the price. Because the software aspect of these devices can be relied upon to bring more misery than benefit.

I recently needed to replace another large file server. I considered a QNap _{hah just kidding} Synology device, but it turns out they cannot be re-flashed with a general purpose OS like Debian. So, I'm having a custom tower server built which will run Debian. With Samba being compatible to Time Machine, I'm not even giving up any important functionality.