On this particular characteristic, aerospace engineer Chris Lewicki shares the second he realised he’d made a large mistake whereas working at NASA’s Jet Propulsion Laboratory — and the way he managed the results for the Mars Rover machine and his staff.
Some errors really feel worse than loss of life.
A February night in 2003 began out routine at NASA’s Jet Propulsion Laboratory (JPL) in Pasadena, CA. I gowned up in cleanroom garb and handed into the Excessive Bay 1 airlock in Constructing 179 the place practically all of NASA’s historic interplanetary spacecraft have been constructed for the reason that Moon-bound Ranger collection within the Nineteen Sixties.
After years of labor by hundreds of engineers, technicians, and scientists, there have been solely two weeks remaining earlier than the Spirit Mars Rover could be transported to Cape Canaveral in Florida for launch forward of its sibling, Alternative.
I used to be into my unofficial second shift having already logged 12 hours that Wednesday. Lengthy workdays are a nominal state of affairs for the meeting and check part. Each system of a spacecraft is totally examined and confirmed to be in good working order earlier than it’s buttoned up for the final time on Earth.
Spirit and Alternative, a part of a now-historic twin mission, had been among the many most advanced spacecraft ever constructed at the moment and represented practically a billion {dollars} invested by NASA. No stress.
The rovers, between them, had 62 brushed-type motors to drive and steer the wheels, management the robotic arm, goal the cameras, level the antenna to Earth, and the assorted robotic origami unfolding and deployments following touchdown. The rover had undergone in depth testing to simulate the cruel circumstances it could face on Mars as a subject geologist.
Particularly vital are occasions involving pyrotechnics, because the explosive shockwaves can harm brittle carbon elements inside these motors.
That night time, whereas my colleagues centered on testing the rover itself, I used to be tasked with verifying the integrity of the motors within the Rock Abrasion Device (RAT) hooked up to the tip of Spirit’s robotic arm.
Disassembling and inspecting motor elements after every spherical of environmental testing is just not sensible. Nonetheless, we are able to verify their inside situation by analyzing their electrical efficiency. To do that, utilizing a tool known as a break-out-box, we disconnect the motor from the spacecraft and hook it as much as an exterior energy provide and strip chart recorder. A functioning motor will present a clean, exponential lower in electrical present throughout spin-up, whereas any issues present up as blips within the sign.
It was a check I had carried out quite a few occasions. My numerous roles on the challenge had given me the expertise to decipher the maze of diagrams mapping the ten,000 pin-to-pin connections that made the whole lot on the spacecraft work, and my accountability in writing the directions on tips on how to join and management all of the motors on the rovers made me the apparent selection for this check marketing campaign.
The second of realisation
Contained in the cleanroom, John, {the electrical} chief in cost there, helped me discover the gear I wanted. Then Mary, our cabling knowledgeable, did the cautious work of unplugging connectors and inserting check gear on the interface I requested for. We ran our pre-test affirmation routine.
The connection interface was operational, the ability provide settings and strip chart setup had been right, and a fast check pulse to a reference motor validated the configuration. With the whole lot so as, the reference motor was eliminated and we jumpered-in Spirit’s RAT-Revolve motor, chargeable for rotating the grinder and brush on a Mars rock.
The testing steps had been confirmed one final time, and we had a inexperienced mild for pulsing the ready motor with vitality.
To get the clearest sign and reveal the smallest of imperfections from the motor, the usual process is to present it as a lot energy because it desires. This makes it vitally necessary to ship the inrush of electrons to the precise place. A incorrect connection may do blue-smoke-releasing catastrophic harm.
Our pre-test routine was an necessary precaution to confirm that this potentially-dangerous configuration was right.
The heartbeat was despatched to the motor. As at all times, the consequence was speedy, however this time, alarmingly unfamiliar. The strip chart didn’t appear to be something we had seen earlier than. It didn’t even appear to be a damaged motor. It was decidedly — one thing else.
My thoughts raced for explanations and in what appeared like an on the spot, arrived on the more than likely clarification. My eyes adopted the wires from our breakout field on the check cart to the spacecraft, and the rationale for the unfamiliar sign landed like a dagger via my coronary heart. All that energy we simply launched didn’t go into the RAT-Revolve motor.
Because of a mistake I had made with the break-out-box, it went the opposite route on the connector interface, sending a surge of electrical energy straight into the spacecraft, as a substitute of the motor.
The implications
The potential penalties rolled over me in nauseating waves. I’ll have simply created a $500 million piece of scrap. With solely two weeks till the spacecraft was delivered for launch operations, there was no time to recuperate from a giant downside. I used to be immediately conscious that there could also be just one rover launched to Mars on this synodic cycle. And my palms had been holding the still-warm rover homicide weapon.
I had realized from numerous experiences on this and different initiatives that dangerous information doesn’t get higher with age so I instantly keyed the mic on my headset and informed Leo, the check conductor operating the opposite testing in parallel, what had simply occurred.
His response twisted the knife in my chest. “Yeah, we appear to have misplaced all spacecraft telemetry only a bit in the past.” NOT signal.
Everybody in my neighborhood was listening in on the voice loop on their headsets, and off-mic, John unleashed a string of profanities about me that might function a sophisticated tutorial for even essentially the most seasoned sailors. The staff instantly ran the spacecraft’s emergency shutdown process and we had been instructed to depart the cleanroom for what would in all probability be a harm evaluation briefing.
I had turned 28 lower than a month prior, appeared and felt a lot youthful, and was a couple of years into my first huge job after school. This primary vital step in my dream profession as an interplanetary spacecraft engineer, which I had aspired to since junior excessive, was maybe additionally going to be my final.
Others within the system check space moved away from me as darkish actuality descended. Matt, the Meeting Take a look at and Launch Operations supervisor, firmly instructed me to jot down down the whole lot I may keep in mind about what had simply transpired. I’m unsure when the tears began, however they had been in all probability flowing as I recorded these particulars alone in a convention room.
With my notes in hand, Leo and my colleagues meticulously examined the night’s occasions. There have been two apparent issues that had occurred. One, a big pulse of electrical energy had gone someplace apart from supposed, and two, telemetry had stopped coming from the spacecraft. Ominously, however maybe with a ray of hope, there was not an apparent hyperlink between these two issues.
Because the staff reasoned via the issue, it appeared the surge of electrical energy probably ended up within the H-Bridge motor driver circuit, primarily a wise site visitors controller for electrical energy. What I did was not good, however fortunately due to one thing known as back-EMF, this was one a part of the rover truly designed to deal with additional vitality.
We determined that the errant pulse had in some way glitched the system sufficient to interrupt the info movement with out completely disabling it. With the spacecraft already powered off, we’d do what you do with your individual client electronics: we’d flip it again on to see if the ability cycle had cleared the issue.
It was near midnight and notifications in regards to the incident had made it up the chain of administration to Pete, the Undertaking Supervisor. Replanning throughout all the challenge of a thousand folks was at stake.
The staff, now with plenty of additional consideration and oversight, re-grouped and ran the usual spacecraft power-on process. When booting up the spacecraft, it takes a bit for the electronics to return on-line, then for the software program as well up and begin producing telemetry. There’s a circuit that produces a pulse each clock cycle (8 occasions a second), turning a purple mild on the bottom help instrument rack right into a robotic heartbeat indicator.
The spacecraft energy provide went via its acquainted development of voltage steps and currents, however after an excessive amount of time, the heartbeat remained darkish, and the telemetry by no means got here.
The aftermath
I don’t actually keep in mind what occurred subsequent. In all probability one thing about conferences within the morning to determine what the hell can we do now?! What I do keep in mind is the sensation of emotional devastation that adopted me house the place I recounted the story to my spouse. I used to be satisfied I’d lose my job within the morning and house exploration historical past would connect my identify to a specific chapter of infamy.
Again at JPL within the morning, in a gathering with a recent shift and maintain outs from the prior night time of catastrophe, we as soon as once more labored via the detailed sequence of reconstructed occasions in search of clues or potential restoration, which felt an increasing number of fleeting till one essential piece of the puzzle was recognised.
The Fluke 87III digital multimeter is a ubiquitous device within the labs of JPL. After I entered the cleanroom the earlier night time, I wanted one and requested John, the sailor linguist, the place I may get one for my check. All had been in use, so he pointed close to the spacecraft to at least one apparently monitoring bus voltage however not concerned in any testing. I fastidiously eliminated the leads and proceeded on to my date with future in testing the RAT motors.
The monitoring multimeter I disconnected was truly finishing the circuit that powered the spacecraft’s floor check telemetry. I inadvertently disabled the connection the moment I eliminated the leads.
We instantly realised that the subsequent factor to do was to revive the multimeter to this obligation and energy up the spacecraft.
We did simply that. It labored. There was a collective gasp because the telemetry flickered again to life — Spirit was not useless in any case!
The staff resumed testing, having misplaced only some hours, and I exhaled essentially the most monumental sigh of reduction in my lifetime, reassured that I won’t have truly doomed the mission to a single-rover endeavour.
The remainder of that morning was a blur. Weeks of analyses adopted on the RAT-Revolve motor H-bridge channel resulting in detailed discussions of potential thin-film demetallisation. Finally the challenge gained the arrogance to disposition the {hardware}: Use As Is.
The lengthy days continued. I moved to Cape Canaveral to start the ultimate preparations earlier than launching the rovers to Mars, and extra thrilling stress-filled moments punctuated the times and weeks. Then Spirit was on Mars, and after a yr of latent stress, it turned out the RAT-Revolve motor labored simply wonderful, and the entire expertise turned a life lesson.
The lesson
As I’ve recounted this story over time, it has not solely enriched my understanding but additionally impressed others to discover and share their very own brushes with failure.
The act of sharing transforms these experiences into worthwhile classes, each for the storytellers and their viewers. Later in my profession, at my asteroid mining startup Planetary Sources, we recognised the ability of those narratives in our hiring course of and staff tradition. We intentionally requested job candidates to share a failure story of theirs, inviting them to acknowledge and study from their previous challenges, whereas recognising that failure is a pure means of studying.
The core lesson I’ve drawn from my rover ordeal is finest expressed in these phrases: “Let your scars serve you; they’re a useful studying expertise and funding in your functionality and resilience.”
Within the depths of the disaster, when the tears had been flowing and everybody else within the system check centre was shifting away, one individual walked towards me. Ernie, a sensible and sort man who had come out of retirement to assist with the round the clock spacecraft shift work approached me and put his hand round my shoulder, and in a mild grandfatherly voice quietly reassured me.
He then uttered the clear phrases that I’ll always remember: ‘Bear in mind this sense the subsequent time you must sign-off that one thing is OK.’
I went on to turn into Flight Director for Spirit and Alternative as they explored the floor of Mars, incomes NASA’s Distinctive Achievement Medal for my efforts, so clearly I didn’t get fired for this incident. However that wasn’t clear till a couple of days later in one of many extra pivotal conferences of my life.
Within the tense interval following the mishap, with definitive evaluation nonetheless pending, passionate and polarised debates ensued in regards to the assessments’ hazards, and plenty of argued for stopping them altogether. The talk concluded, and the criticality of those assessments — ensuring our motors would perform flawlessly on Mars — was nonetheless paramount. The assessments wanted to proceed.
And I nonetheless keep in mind the shock when Undertaking Supervisor Pete delivered the choice and the follow-on information: ‘These assessments will proceed. And Chris will proceed to steer them as we’ve paid for his training. He’s the final individual on Earth who would make this error once more.’
I discovered myself returning to the ‘scene of the crime’ for a lot of extra assessments, after I had fastidiously revised the procedures to eradicate the prospect of repeating the identical mistake. Every time I performed this check once more, Pete’s vote of confidence mixed with Ernie’s phrases of knowledge introduced with it a second of nausea, a stark reminder of the previous incident, but additionally the readiness and confidence to proceed. The belief administration confirmed in me, regardless of the preliminary error, marked a key second in my profession, highlighting progress and the flexibility to beat challenges.
Now, every time I’m known as upon to present my approval or endorsement for one thing vital, I’m immediately transported again to that second — the room, the lighting, the chair I used to be in, the desk, the pit in my abdomen, the extreme mixture of concern, nervousness and remorse for an oversight that almost led to disaster.
Ernie’s knowledge that day, mixed along with his compassionate method throughout my second of vulnerability, left an indelible mark on me. Now, when confronted with vital selections, I not solely recall that have but additionally attempt to help others in navigating their very own difficult moments. And like Pete did for me, my goal is to help in remodeling these experiences into catalysts for progress and resilience, reinforcing the notion that our responses to adversity can outline our path ahead.
These tales of close to misses, studying curves, and eventual triumphs are usually not simply mine, however are shared by many who construct issues. In house exploration, failure is just not an possibility — it comes pre-installed.
Each misstep is a stepping stone in the direction of larger success, and collectively, our collective knowledge can pave the way in which for future improvements, achievements and breakthroughs in creating and rising our presence in and profit from house.
This text was initially revealed on Chris Lewicki’s web site and is repurposed right here with permission.
