What you’ll learn
- Why Log4j is an issue for embedded developers.
- Challenges with using open source.
- How Bill killed 28 TB of storage.
Unfortunately, I recently killed a pair of 14-TB hard drives on one of my servers. How this relates to open source and Log4j is not too convoluted.
The hard drives decided to kick the bucket because they got too hot. This occurred because the case fan that cooled the block of hard drives decided it was a good time to die. Of course, there were no alarms and I discovered that the drives were generating errors when I accessed some files. This was a tertiary server that I don’t access much, which is why the problem wasn’t discovered sooner. More details on this later, as well as what I wound up doing about it.
What is Log4j?
So, the first piece of this puzzle is Log4j (Fig. 1). Log4j is an open-source project with Apache Software Foundation. This Java-based system provides logging services and is incorporated into a number of other projects and products, such as many open-source projects.
Logging is a central component to the Internet of Things (IoT). Not all IoT systems use Log4j, as it’s just one of many middleware systems used by developers. As such, many other systems depend on the quality of its code when it comes to functionality and security. The latter has become a problem recently with CVE-2021-45105, CVE-2021-45046, and CVE-2021-44228 being issued.
CVE stands for Common Vulnerabilities and Exposures. The CVE Program manages definitions and cataloging these problems, with each receiving a unique number. For instance, CVE-2021-44832 is one for Log4j2 about a vulnerability to remote code execution (RCE) attacks.
Projects that use Log4j are susceptible to identified bugs like these until the Log4j version they’re using is fixed or the application mitigates the problem. Some problems are inherent in the underlying system. A version with a bug fix is all that’s required to secure an application, which is why keeping up-to-date with the latest versions of software is generally recommended.
But….
Embedded developers are an interesting lot. They tend to want to use stable development tools rather than diving on the latest technology that’s being used with smartphone apps or cloud systems. Technologies like hypervisors were uncommon a long time ago but have become ubiquitous in mid- to high-end platforms. Things like serverless computing are still in the cloud. Even keeping up-to-date with tools and support software can be an issue.
Dealing with Changes and Fixes
Changes can cause all sorts of headaches even for projects that do not require certification. Those that do often require recertification when changes are made. Changes require testing and they need to be deployed. This can be time-consuming and costly.
Embedded developers write lots of code, but most systems are built using other components from an RTOS to middleware. Open-source components within this mix are where things like Log4j come into play. Of course, proprietary solutions can reduce the dependency on open-source solutions, but the problems we’re discussing aren’t limited to open-source platforms. They also occur with closed, proprietary solutions.
The rub is who is responsible for fixing problems and distributing the fixes. Usually closed, proprietary software is the responsibility of the company sourcing this software. Sometimes the source is available whereby developers using the software can make changes. For open-source software, the source code is available, but whether any fixes are rolled back into the main development thread is up to the open-source maintainers. In general, developers don’t want to take over maintenance and support of either open or closed solutions.
Some open-source projects have corporate support. Linux is a classic example, but most open-source projects lack financial or work support. Lately there’s been more discussion and action taken regarding projects with little corporate support, but are projects used by many companies either internally or as part of their products.
Lately, one developer corrupted their own open-source project, essentially breaking software that depended on it to try to make others aware of the lack of support and the importance of the project. Those that simply used the updated source code without checking had a problem. Rolling back to a prior version allowed software that was dependent on the project to work, with future updates perhaps fixing the issue.
Of course, someone could essentially take the older software and start a new project based on it, since the software has an open-source license. There’s still the problem of support going forward.
Open-Source Support
Supporting open-source development is at the crux of this issue, as well as determining the dependencies of your application. There is actually software to do the dependency tracking. It can include tracking the different types of licenses involved, but that’s another story for some other time.
What developers need to keep in mind when choosing their tools and support software is what the real costs will be, where the money is being spent, etc. Free software, open-source software, and closed/proprietary software all have their own set of costs and issues.
So, how does all this open-source stuff blend with my dead hard drives?
Disk Doom
Well, a while back I wrote about how I used an open-source project called Centreon to monitor my various systems, including the server with the overheated hard disk-drives. It has now been modified to track the temperature of every hard drive on all my servers (Fig. 2). This took a bit of effort because the systems used different controllers and had different ways of logging the data. Log4j isn’t used by this particular software, but it’s a similar application since Centreon essentially pulls information from various sources, compares it to my settings, and reports warnings and errors.
This type of system is typical for an enterprise or server farm, but rare for a lab or home setting like mine. Also, while the system was already set up to track dozens of details like disk space, it wasn’t set up to check either the fans on the system or the temperature of anything including the disk drives.
As I have found out, there are dozens of details I could and should be checking via the system. In actuality, though, I just wait for it to send me an email about any problems. It has done pretty well overall. Still, it’s hopefully a hint for you to keep in mind what you might be looking at for your digital twin or IoT device. That’s because more pieces of information likely could be useful if you just thought about it. It might be the battery level or an intruder alert.
In any case, I had backups for most of the data on the drives and storage-wise it was only 14 TB since it was a RAID 1 configuration. That didn’t help, since the drives are adjacent and both overheated. Luckily, backups are on another server in a different room. One can never have enough backups.