Firmware flaw in recent Stirling SU780XLE -80C freezers

[This post is a follow-up to my previous post on this subject]

Wow, I never thought I’d learn so much about a freezer company, but here we are. I took a deep dive on this issue with the Stirling SU780XLE ULT freezers. It’s still second-hand (through company reps and people on social media) and I don’t know if I believe everything about the explanations I’ve received (for example, I count roughly 8 instances of freezer firmware getting stuck through various contacts, and I vaguely remember a company rep saying this has happened <= 10 times), but this is my understanding of the situation:

The issue is indeed a firmware problem, and it affects all units produced between ~ Aug 2019 and ~ Sep / Oct 2020. Aug 2019 is when they switched one of their key electronic components to a Beagle Bone (apparently a circuit-board akin to a Raspberry Pi). Part of its job is to relay messages from one part of the circuitry to another. The firmware they wrote for it had a flaw, where — in certain circumstances that the company still does not understand — one part of the relay no longer works, and the other part of the relay just keeps piling up commands that go unexecuted. So that’s the initial issue. There is also supposed to be “watchdog” code that recognizes these types of instances, but this was not working either. Thus, the freezer becomes stuck in the last state it was in before the relay broke. If it was in a “run the engine to cool down the freezer mode”, then it would have been stuck in a state that kept things cold. If it was in a “stay on but don’t do anything b/c it’s cool enough” mode, then it would have been stuck in a state where it didn’t cool the freezer at all. This is the state my freezer was stuck in**.

[** I’m actually not 100% convinced on this. My freezer stopped logging temperatures / door openings, etc at the end of August. If I look at number of freezer hours, it says ~8,000 hrs (consistent with Oct’19 through Aug’20) rather than the ~10,000 hrs for Oct’19 to Nov’20). It is definitely within the realm of possibility that my Stirling has been a zombie for the last 70+ days, and either slowly reached 5*C over time or had a second event over the last weekend that triggered the thaw in its susceptible state.]

It sounded like they had seen numerous freezers get stuck in the former format, which was the less devastating mode since it didn’t result in freezer thawing and product loss. They had seen one freezer get stuck in the catastrophic format before me, back in Aug 20th. They brought it back to their workspace, and couldn’t recreate the failure. They could artificially break the relay to reproduce the condition, allowing them to create additional firmware that actually triggers the “watchdog” (and other failsafes) to reset the system when it has sensed that things have gone wrong, event though they still don’t know what the original cause of the issue is. The reason the freezers produced after Sep / Oct 2020 are unaffected, is that these have already been programmed with the new firmware. The firmware I had when it had encountered the problem was 1.2.2, while it became 1.2.7 after it got updated.

Freezers made / distributed(?) within the last month were pre-programmed with the updated firmware, and are supposedly not susceptible to the GUI freezes. Apparently they’re having trouble updating the firmware in the units b/c the update requires a special 4-pin programming unit that is in short supply due to the pandemic.

I won’t get into the details of my experience with Stirling (it apparently even includes a local rep who contracted COVID). They completely dropped the ball in responding, and they know that (and I’m sure they regret it). What will remain a major stain on this situation is that THEY HAVE KNOWN ABOUT THIS FLAW FOR MONTHS AND DID NOT WARN ANY OF THEIR CUSTOMERS. I received an email ~ 8 days ago saying they were going to schedule firmware updates to “improve engine performance at warmer set points, enhance inverter performance and augment existing functionality to autonomously monitor and maintain freezer operation”. Other customers with susceptible units did not even receive this vague and rather misleading email. My guess is that they chose to try to maintain an untarnished public perception of their company over the well-being of the samples stored by their customers. My suspicion is that their decisions may have been exacerbated by the current demand for -80*C freezers for the SARS-CoV-2 mRNA vaccine cold chain distribution (Stirling has a major deal with UPS, for example), though there is no way I will ever confirm that.

After my catastrophic experience, they bungled their response, and only jumped to action after I tweeted about my experience. I really wanted to like this company, as they are local and not one of the science supply mega-companies (eg. ThermoFisher). My fledgling lab is still out almost $3k in commercial reagents, and many of my non-commercial reagents and samples were compromised. They did make a special effort to update my firmware today and answer my questions, but I still can’t help to feel like a victim of poor manufacturing and service. All of the effort I’ve put in the last few days was to get to some answers and help others avoid the same situation I was put in.

I’ll post any updates to this page if I learn any more, but I’m now satisfied with my understanding of what happened. Now back to some actual science.