I know I’m not paranoid, but the world is out to get me. It will probably be through all the IoT and connected devices, though not if I can help it. How hard could it be?
Having worked extensively with networks, operating systems, and IoT devices, I already knew it would take a lot. Compounding the problem is that I also run half-a-dozen physical servers for my home and lab as well as more than a dozen virtual machines, many of which host various web servers for internal and external use. Definitely not what most IT pros do, but more than what the typical home owner has on hand.
My collection of IoT devices is actually pretty low and many are transient because I test a wide range of devices. These don’t always stick around and going through the exercise of tracking and fencing them isn’t easy, as you shall see. Likewise, locking things down isn’t always an easy chore as well.
Getting Started
I began this project way back in early 2018. I figured that locking down the servers would be the best place to start since I had the most control over them—how hard could it be? They run various versions of Linux and Windows Server.
The first thing was to make sure the security support was configured properly and as tight as possible. The Windows Server systems were primarily for file services, so using the defaults seemed to be enough. I have a limited number of users and the admin password is ridiculously long. The same is true for the Linux servers. I restrict SSH access to non-root for Linux servers and non-admin access for remote control of Windows Server. In theory, it takes two steps to remotely log in a root.
The next thing was to lock down most of the Fedora and CentOS servers with SELinux. I wrote “Don’t Do It: Disabling SELinux” along the way. Making sure there was SELinux support, or adding it to the file servers, was easy since that support is built in, but it was a challenge for some of the web servers. I had set many of them up with SELinux disabled or left Apache unconfined.
Though challenging, it was doable, and I learned a bit more about SELinux. It actually turns out to be easier to do once you know about all of the tools. I suspect reading a good SELinux book would have helped, but I wound up learning on the job in an incremental fashion over the years. Adding it after the fact tends to be much harder than adding new services to a running system and making sure those services are also confined properly.
In addition to SELinux support, I added AIDE (Advanced Intrusion Detection Environment) intrusion detection support. This tracks file and directory changes on Linux servers. It’s more to provide a tip that a hack was attempted or has occurred, so it’s more like antivirus software on a PC; nonetheless, it’s still useful. The more tedious part of the setup was placing read-only databases on a server and checking that periodic checks were being done.
Locking down Windows workstations and laptops was already done to the degree I could. They have separate admin accounts, some install options have been locked down and we at least run passwords at home although auto login is the norm for the PCs.
Firewall Duties
Another chore was the firewall/gateway. For that I run ipFire (Fig. 1). It’s a good compromise between functionality and simplicity. It provides VPN access, intrusion detection, and so on. The user interface tends to be more for the technically inclined, but that works for me. I also have a Verizon gateway/router that’s essentially required for my internet service and it provides similar functionality. The thing is, I don’t have control over it from an update standpoint, although I can configure it. It has a lot of port forwarding rules, as does the ipFire system, but the latter gets updates on a regular basis.
1. The ipFire firewall/gateway is a nice blend of functionality and simplicity.
Part of the ipFire work included locking down the IoT and connected devices on the network. I would have liked to set up separate VLANs for everything; however, the combination of switches and wireless access points is a chore I’ll leave till next time. On the other hand, I run everything with fixed IP addresses and set up ipFire to limit or prevent internet access by the devices. For example, my network printers don’t have internet access even if I enable those options on the printer, which they all have of course. If I need to print something at home while on the road, I will use my VPN access.
One feature that would be useful for ipFire would be the ability to allow access to the internet via IP/MAC address through the host and DHCP support, instead of having to create a firewall entry manually. Some router/gateways provide this functionality.
Connecting to the Internet
In any case, printers, network cameras, and so on have no internet access while some devices have limited access. One of those is the Honeywell RTH9580WF (Fig. 2) Wi-Fi thermostat. I replaced the Nest thermostats with these when I found out that the 5-wire connection caused havoc with the contactor on my air conditioner. It turns out that a Nest will snarf power by cycling the air conditioner on and off quickly if it doesn’t have a dedicated power line. For my configuration, I would need six wires and pulling a new cable was not an option.
2. The Honeywell RTH9580WF is a Wi-Fi thermostat that needs to access two IP addresses to provide its remote services.
Anyway, I gave the RTH9580WF a fixed IP address and tracked its operation. I set up ipFire to allow access only to the Honeywell site. That worked great except that the outside weather status wasn’t updating. I finally found a blog by David Trebacz that talked about how Honeywell connects to Microsoft Azure. I was able to get the IP address for the firewall entry by using the ipFire logs. At this point, the fixed IoT devices were taken care of. It would really be nice if Honeywell would put this information somewhere on its website or online documentation rather than requiring hackers to extract this information and provide it in a blog.
Unfortunately, this will be typical of any IoT device these days. They assume complete, unobstructed internet access.
It was time to set up centralized monitoring support. This turned out to be very useful and something I should have been doing all along as I periodically missed things like downed servers and backups that weren’t working right. I run a couple of backups including Bacula, but never set up email notification and so on. That’s all in place and I had a number of notifications throughout the year that helped streamline the operation.
Centralized monitoring was a chore to set up, but very useful in providing information about when something failed. This included tracking devices like UPSs (almost everything has a UPS around here, tends to get noisy when the power goes out) to network printers. It’s handy because it can also notify—assuming the network and gateway are not down.
The Essential Logging Support
I have used Centreon (Fig. 3) on a number of networks in the past, so I set it up on a virtual server on my network. It turned out to be very handy for tracking changes like the free disk space on my MythTV backend. I was always wondering why some recordings weren’t happening. Turns out I had set the free space setting a bit too tight and hadn’t been looking at the MythTV logs too closely.
3. Centreon is an open-source, system-monitoring system.
Setting up Centreon along with SNMP support was a simple exercise since I had done it numerous times before. I just wonder why I didn’t do it sooner. I think it was partially the slow growth in the number of servers and services. The unfortunate thing is most of the IoT devices have minimal ping tracking since they tend to provide their own monitoring support. Though it would really be nice to have something hosted on my own network, it’s unlikely to happen, which is why so many people wind up with a collection of Apple or Google products just so they can be centrally managed.
Now, most of the servers have logging support internally. I started with Fluentd to provide centralized log collection. Fluentd is one of many unified logging systems. As it turns out, a company called Treasure Data provides td-agent that provides data from a client to a Fluentd server. Treasure Data is now part of Arm and its IoT services.
I actually fell back to the rsyslogd support because I have a relatively small, static environment and it’s already part of the standard Linux setup. It was just easier to set up the clients without adding additional software to the mix. I also don’t need cloud-based services since the object was to provide local system management.
Having logs turned out to be handy diagnosing some issues with Zoneminder (Fig. 4). This is video-surveillance software that I have hooked up to four IP cameras spaced around the exterior of the house. I have run into issues with cameras dropping out and disk-space issues like I had with MythTV. These turned out to be configuration details, and I fixed the problem permanently. It’s one of the challenges of putting together your own configuration, but at least I know that it will continue to run until the hardware breaks. I have had more than a dozen pieces of “internet” hardware essentially turn into bricks because the cloud support went away.
4. Zoneminder is an open-source video security system.
This process was done over months when I had the time. In the end, it provides minimal output as it should, with the hope that a notice will pop up if something goes wrong and the information about what happened can be examined.
So, at one end of the spectrum will be those IT guys saying I should have been doing this all along. At the other end are those scratching their heads as to why I would take all this time to cobble together these services. In the end, it was definitely worth it. The setup requires minimal maintenance once it’s all in place. Likewise, adding something to the mix requires a bit more work than just turning on a device, but there’s a known combination of chores that must be done to allow for better security and tracking.
I did finish up all my changes by the end of 2018, so I could start off the new year feeling a bit more secure. I still need to finish up adding software inventory support. I use OCS Inventory NG. It’s useful for tracking software versions since many of my systems don’t run automatic updates.
Managing updates? That’s another chore for the future.
So, what did I do at the start of this year? Remove search browser malware from my Windows 10 PC. Sigh.
