DO NO HARM
                               by JOHN G. HEMRY

It’s possible to do some jobs too well!
                                       ****
      Sandra’s acting weird, the geeks can’t figure out why, and the boss is
spinning like a pulsar.”
       Kevlin pulled his attention out from the immersive medical simulation long
enough to give Yasmina a questioning look. “I thought Sandra was supposed to
leave this morning.”
      “Right. She won’t go. Come on. The director’s called an all-principals
meeting.”
       “I’m a doctor,” Kevlin objected. “I’m supposed to keep the people working
for the corporation on this station healthy. Why do I care about Sandra’s
problems?”
       Yasmina smiled back at him in a mocking way. “I’m a doctor, too. If I have
to go, so do you.”
       “They need you to analyze the project director’s mind just in case he gets
really dangerous this time,” Kevlin suggested. “I’m just a simple country doctor with
a low-gravity, space illness specialty.”
       “Sure. Then you’ll come in handy if the director bursts a vein while he’s
yelling at everyone.” Yasmina beckoned. “Come on.”
      Grumbling just loud enough for her to hear, Kevlin paused the sim and
followed her down the hallway. “I could always monitor the director’s health from
my office,” he suggested.
      “Nice try. Didn’t your teachers at med school ever tell you not to try to con a
shrink?”
       Sandra was still at loading dock four alpha. Yasmina led the way onboard the
ship, then along a passageway that ended in Sandra’s control room. The limited area
was already full of exasperated engineers of various types and persuasions, some
looking dejected, some angry, and some staring into space as they tried to think.
“Why can’t we do a virtual meeting?” one complained as Kevlin and Yasmina
wedged their way in.
      Another engineer answered in an accusing voice. “Because the director found
out you guys had been hacking the meeting code so you could have avatars sitting in
for you while you did other stuff. Now we all have to crowd in here in person so he
can be sure we’re all actually getting yelled at.”
      “People have been hacking virtual meeting code since the Stone Age,” the first
engineer protested, then hastily stopped speaking as a short man with a lofty attitude
and an ugly frown strode in, the crowd somehow contracting away from him so he
had free room.
      “Report,” the director stated, glowering at the chief designer.
       The chief designer, who had been arguing with Sandra’s captain, made a
helpless gesture. “Sandra won’t work. Something’s shorting out her central control
functions.”
       The director’s glower deepened. “The Spaceship Autonomous Network
Developmental Research Application is the most expensive project in the history of
this company. I expect more from you than vague reports that it just doesn’t work!
Are you saying the control network isn’t receiving the commands?”
      “No,” the chief designer responded in a tight voice. “I’m saying that the
control network isn’t responding to external signals. It’s in some kind of weird loop,
with only a few apparently random signals going out to minor subsystems. We give a
command and nothing happens.”
     “Nothing happens? Something has to happen! If nothing is happening that
means something is happening!”
      Kevlin gave a glance at Yasmina, who was watching the director with a
fascinated expression. He just knew she would love to get the director into a
controlled environment so she could analyze his mental processes.
      One of the other engineers tapped the air in front of him, activating a virtual
display. “This is what Sandra’s central processing activity is like.”
       Yasmina looked suddenly startled as an image appeared overhead. “That
looks like an EEG of an epileptic seizure.”
       Eyes swung to focus on the doctor. “An epileptic seizure?” the director asked
in a deceptively mild voice.
       Though it was obvious she regretted speaking, Kevlin wasn’t surprised that
Yasmina refused to back down. “Yes,” she insisted. “That’s what that looks like. If
I saw that representation of signal activity in a human, I’d say it was a seizure.”
      “This is a ship,” the designer protested.
         “Yes,” Yasmina agreed. “A ship you constantly refer to as if it were human,
as if it were alive, talking about the complexity of an internal and external sensing
network that mimics that of a living creature. I’ve read the specs on the central
command system. You modeled it on basic brain functions. Well, maybe that means
it’s subject to the sort of problems living brains develop.”
      Kevlin waited for an outburst of laughter or scorn, but it didn’t come. A third
engineer nodded with a wondering expression. “The operating system is incredibly
complex, full of learning routines and development loops. It could’ve developed
problems like that.”
      “How do we cure it?” the director demanded. “In people?”
       This time Yasmina grimaced in the way of a doctor trying to explain complex
things in layperson’s terms. “Short term, we use medications that raise the seizure
threshold. Long term, we go in and fix whatever is causing the brain to short-circuit.”
      The chief designer’s eyes narrowed in thought. “Short-circuit? What could
have caused that to happen? Sandra’s central command functions were working fine
yesterday. We haven’t modified them since then.”
      “Stray signals?” another engineer suggested.
      “The central command area is shielded.”
      “Maybe some other part of the, uh, neural network on Sandra?” Yasmina
offered.
      This time everyone’s attention turned toward a senior engineer, who looked
defensive. “The test monitoring equipment couldn’t—”
      “It’s wireless!” the director snapped.
        Sandra’s captain and the chief designer were studying something. “Stray
signals. That would do it. They must be filtering in through the sensing network. Oh,
hell. I bet they’re reflecting down these access trunks and into the command circuit
sub-junctions.”
       The director’s glower deepened as he barked another order at the hapless
senior engineer. “Turn it off!”
       The senior engineer punched some commands, and a moment later the
depiction of Sandra’s control system activity cleared. A muted cheer sounded,
choked off as the director stabbed a finger at Sandra’s captain. “We’ve already lost
two hours. Get this thing underway and get the tests done. Everybody else who isn’t
part of the crew get off this ship now!”
      Yasmina turned to go, but stopped when the director called out again. “Not
you, Dr. Finshal. In light of the fact that we needed your assistance to correct this
problem with Sandra,” he added with a scowl at the chief designer, “I think it would
be wise if you go along on the test voyage.”
      “I hope you enjoy the trip,” Kevlin whispered, taking a step away.
      “Dr. Shan!” Kevlin barely avoided wincing as he turned to face the director.
“You, too. Since one type of doctor was able to diagnose a problem with Sandra,
having a physician along too might be a good idea.”
      “Um, but I need to—”
      The director had already vanished down the passageway. Most of the
engineers vanished in his wake, leaving only the ten members of Sandra’s crew and
the two doctors.
      “This is all your fault,” Kevlin grumbled to Yasmina. They were strapped into
acceleration seats at the back of Sandra’s main control room.
      “Think of it as an adventure if that helps you cope,” she replied.
      “An adventure? We’re just going outside lunar orbit and coming back. Some
adventure.” Kevlin “tapped” the virtual screen before his seat, bringing up different
images, pausing briefly when he reached one showing an outside view of Sandra still
docked to the station, Earth’s globe floating serenely in the background. Someone
had positioned that shot with the skill of a public relations expert capturing an
important moment. Finally he settled on the crew status display, providing real-time
updates on important activity within the crew’s bodies. “Some of the crew members
were up all night,” he observed out loud.
      “Really?” Yasmina frowned. “I’ve recommended against that sort of thing.”
       “They’ve been doped. Looks like pentastamine. Yup. As good as a full
night’s sleep.”
      “I don’t care,” Yasmina grumbled. “There’s no substitute for natural sleep.”
      Kevlin shrugged. “The stuff’s been tested—”
      “I know! I also know there’s a lot we still don’t understand. The human body
and brain are incredibly complex.”
      His reply was cut off when the captain raised her voice. “Sandra. Separate
from the station and proceed along preplanned track Alpha One.”
      A cool female voice replied. “Command understood. Complying.”
       Yasmina’s scowl deepened. “I told them to have her repeat back the
command so they could be sure she actually had heard them right. But they
complained that was inefficient since they know everything about Sandra and how
she’ll respond. You’d think their experience this morning would have suggested they
don’t know everything about her.”
      “Her?” Kevlin asked. “You’re talking about Sandra as if she’s alive, too.”
      “So? Look.” She reached across and brought up a different display for him.
“This monitors all systems. What does that look like?”
      Before him, an image of Sandra loomed, the ghostly exterior allowing a clear
view of representations of subsystems depicted with visual cues for performance.
The power system’s branches pulsed green, the filaments of the command network
glowed golden throughout the ship, life support flared blue. “I hadn’t seen this
before,” Kevlin admitted. “It does look like a living thing. Is that just how the display
works?”
       “Not entirely. The ship integrates the latest tech using living models. There’s a
host of macro and nano-based devices swarming through the hull to keep all
subsystems working right and in repair. It’s all networked under the central control
system, linked into one entity.” Yasmina shrugged. “I’ve got a subspecialty in
psychocybernetics so I was involved in some of the design discussions. Not that
Sandra has consciousness or can develop it. But her functions run along lines
suggested by things like the human brain stem.”
       Kevlin saw commands racing through the depiction of Sandra’s ‘nervous
system,’ then the ship lurched as it detached from the station, pushing clear of the
rotating structure and swinging around. The main drive cut in and slammed him
against the back of his seat. A black fringe wavered around the edge of his vision as
the acceleration grew.
    “Sandra!” The captain called in a voice tortured by pressure. “Keep ship’s
movement within crew comfort parameters.”
       “Command understood. Complying.” Sandra’s voice, of course, wasn’t
stressed at all.
      The acceleration slacked off. Kevlin took a grateful breath and shook his head
carefully. “Why did she have to be reminded of that?”
     Yasmina was watching the crew in the command seats arguing among
themselves. “I imagine that question is being debated right now.”
       After that, very little happened of interest to Kevlin. Sandra bored a hole
through empty space on a trajectory avoiding normal space traffic, while the
engineers put her through various tests. Kevlin monitored the crew’s physical states,
spotting the reactions that told him when Sandra had performed particularly well and
the other reactions that indicated something Sandra had done had generated concern.
That got old, too, until on a whim he brought back up the display showing Sandra’s
inner workings and compared it to the human crew’s as the ship went through her
paces.
      “What’s so fascinating?” Yasmina asked.
      Kevlin blinked at her, taking a moment to refocus. “I was just watching the
behavior of the ship’s subsystems. If I didn’t know better, I’d think I was seeing
autonomous physical reactions.”
      “I told you it was modeled on that.”
       “No, I don’t just mean actions in response to commands. It looks like
reactions to the commands, to how well the ship performs. See?”
      Yasmina peered across, her face intent. “That’s weird. I haven’t seen that
reported. No, wait. There’s been some reports of transient system behaviors. The
geeks thought they were caused by learning routines and would damp out as the
system matured. Are you seeing that?”
      “No. They’re getting stronger and more obvious.” Kevlin took a look at the
crew, who seemed calm enough, then checked their physical states. Stress was
obvious there in a lot of cases. Something was bothering them. “Has Sandra failed
any tests so far?”
      “Not as far as I can tell. Results keep showing her exceeding expectations.”
      Sandra’s voice sounded again. “Cooling subsystem module seven suffering
from degraded performance.”
      Kevlin focused on that component, seeing the images marking nano- and
macro-scale automated maintenance drones hastening to the site. Nothing seemed to
happen for a while as the devices clustered on the ailing component, then a second
wave of repair drones appeared, bulling past the first wave. Within moments, the
module’s performance markers improved. “Did Sandra just create a new repair
capability?”
      One of the crew heard him. “We call it evolving. The system learns what new
capabilities are needed and modifies existing equipment.”
      “Where does it get the resources? Does it cannibalize existing equipment?”
       “It can.” The engineer grinned and highlighted a display showing the first wave
of now-obsolete repair drones being disassembled by some of their successors.
“There’s also a small supply stockpile onboard for her to draw on during the test
voyage. We didn’t top her off since we didn’t know how she’d work until we put
her through her paces.”
      “Good,” Kevlin muttered. He saw Yasmina eyeing him.
      “What’s bothering you?” she asked.
      “Nothing.”
      “Right. And I can’t judge thoughts from exterior cues. What’s wrong?”
      He frowned, not wanting to admit it. “Evolves. I don’t like that.”
      “There’s limits. Sandra can’t evolve into sentience.”
      “They’re sure?”
       Yasmina nodded. “I was in on that design work. Sandra’s control system is
roughly analogous to the more primitive parts of the human mind, the stuff that
handles basic functions. There’s nothing that can evolve into a higher brain function
because the space is tightly constrained and the resources are fenced off. In order to
modify itself enough to achieve a simulation of sentience, Sandra would have to be
sentient to begin with.”
      “You’re telling me she’s just operating on instinct?”
      “Pretty much.”
       Kevlin tried to relax his frown. “So all we have to worry about is instinctive
level behavior. I hope they didn’t forget about the id.”
        “Oh, no!” Yasmina declared in a dramatic voice. “They forgot about the id!”
She chuckled. “It’s been ages since I’ve seen that ancient video, but I still remember
that line. The id doesn’t really exist, you know.”
      “Something does that we can call the id,” Kevlin replied.
      “Sandra doesn’t have ancestral behavior patterns inherited from a long line of
evolution,” Yasmina noted sharply.
      Kevlin subsided, gazing morosely at the displays again.
                                            ****
      They watched, they ate the boxed meals provided, they watched some more.
The crew made things happen, made things go wrong, made things break, and
watched Sandra take corrective action. As minor incidents were properly handled
and Sandra’s abilities evolved, the tests continually got harder, stressing the ship’s
responses. Most of the action was apparent only on the displays, though, as events
unspooled inside the ship away from the control room.
       Kevlin switched displays restlessly, even taking a while to watch the view from
the chase ship that was following Sandra as a safety precaution. But watching a ship
whose motion wasn’t apparent against an unchanging backdrop was like staring at a
painting. A little of that went a long way. Kevlin finally dozed off, waking to see it
was late at night on the human clock, though of course the star-studded darkness
outside Sandra hadn’t changed.
       Voices were raised in the forward part of the control room. The argument
there was probably what had awakened him. The captain noticed Kevlin was awake
and directed a question his way. “What do you think of this?”
      Kevlin’s virtual display lit with an image of Sandra. He studied it, baffled as to
why the crew would be asking his opinion about anything to do with the ship, then
spotted a section about two-thirds of the way back on the ship. Something had
bulged out into the passageway there, pushing into adjacent areas as well and even
cutting off a couple of subsystem circuits. Frowning, Kevlin zoomed in the display,
seeing that the mass consisted of hundreds of identical components, fused together.
“What’s going on?”
       “We have no idea,” the captain barked. “Sandra doesn’t seem to understand
it and claims she can’t stop it. Our best guess is that one of the repair segments
locked somehow and keeps replicating the same component.”
      “Out-of-control replication?” Kevlin couldn’t hide his reaction. “Like a
cancer?”
      “Cancer?” The captain looked baffled, then appalled.
      One of the other engineers nodded quickly. “Her repair systems have been
evolving rapidly under the pressure of the tests. One of them must have evolved in a
way that bypasses Sandra’s control functions.”
      “How do we stop it?” the captain demanded.
       “Uh...” Kevlin scratched his head, noticing that Yasmina had also woken up
and was watching them with a captivated expression. The woman got her kicks out
of the strangest things. “Starve it? Can you kill power to it? Or prevent whatever’s
building the components out of control from getting access to more resources?”
       The crewmembers went into a huddle. The captain called out several orders to
Sandra, looking steadily more unhappy as each command failed to choke off the
equipment tumor still growing into the passageway. Finally she turned to two of the
crewmembers. “Chen and Ragosa. You two go down there and manually cut
through these circuits and feeders. See? I’m downloading the diagram to your
personals. The repair drones can operate without external power for a limited period,
but then they’ll shut down in that area. That’ll at least stop that thing from growing
any more while we identify the bad components.” She frowned again as the two
unbuckled and floated free of their seats. “Take your suits.”
       “Ah, hell,” Chen protested. “We’ll be carrying enough as it is with the laser
cutters and manual tools.”
      “Wear your suits! I won’t get nailed for violating safety precautions on a
shakedown voyage!”
      Kevlin watched them go, then gazed at the display again.
      “Now what?” Yasmina asked softly.
      “I don’t know. Something is nagging at me. I can’t figure out what it is.”
      “Do you think those two are in danger?”
      “No.” Kevlin shook his head. “I don’t think so. How could they be? There’s
safeguards built into the system, right?”
      “Sandra’s full of them,” Yasmina agreed. “She can’t try to harm people, or let
people come to harm. You’re still worried about something serious going wrong?”
       “They’ve mimicked the operation of a living organism, Yaz. Unpredictable
and living go together.”
         “How could it bypass the safeguards?”
       “I don’t know!” Kevlin made a frustrated gesture. “They designed this thing’s
internal functions, especially its self-operating and repair functions, to ‘evolve.’
What are the limits on that?”
         “I told you. Sandra can’t get sentient.”
       Kevlin glowered at his display. “There’s a whole lot of things that go wrong in
living organisms that operate below the level of sentience.”
         She frowned at him, but said nothing more, apparently thinking.
                                       ****
       Chen and Ragosa reached what even the crew had begun calling a cancer and
started cutting.
      “Alert,” Sandra’s voice declared dispassionately. “Interior damage in port aft
main passageway between frames sixty-five and sixty-six.”
         “Acknowledged,” the captain replied. “Authorized repair work is underway.”
       A moment passed, then Sandra spoke again. “Alert. Interior damage in port
aft main passageway between frames sixty-five and sixty-six continues and is
intensifying.”
         “Acknowledged,” the captain repeated. “This is authorized repair work.”
      “The ship is suffering internal damage,” Sandra repeated, her voice somehow
sounding insistent to Kevlin.
       The captain frowned and went into another huddle with the other engineers,
only to be interrupted by a call from Chen. “Hey, what’s Sandra doing? There’s all
sorts of repair drones showing up and milling around. Ah, damn. Some of them are
repairing the cuts we’re making!”
       “Sandra!” the captain shouted without waiting to reply to Chen over the
circuit. “Cease repair activity in port main passageway.”
         “Command understood. Complying.”
         Chen came back on, sounding aggrieved. “Why aren’t you telling Sandra to
stop?”
      One of the crew spoke up. “Here it is. Sandra’s acting on our commands but
doing almost immediate resets in response to the stimuli from her internal
damage-sensing network.”
      For some reason, the captain swung and gave Kevlin an accusing stare. “Can
you explain that?”
        Kevlin swallowed before answering. “If Sandra were human, I’d say it was
like telling someone not to scratch when they keep feeling an itch. Just how closely
do your damage and repair network feeds to the central control system resemble the
stimulus-response process to discomfort or pain in a living creature?”
      “Sandra doesn’t feel pain,” someone insisted.
      “She feels something that prompts her to action, doesn’t she?”
      The captain gave his crew another glance, then they began talking rapidly again
in low voices that didn’t carry well.
      Yasmina spoke to Kevlin in a whisper. “Are you wondering at what point a
stimulus-response system evolves into a pain network?”
      “Yeah. They can say it’s not pain, but if it triggers the same defensive
response in the organism, then what’s the difference?”
       Sandra spoke again, her voice definitely more urgent. “Damage spreading in
port aft main passageway. Require immediate response.”
      “Hey!” Chen roared over the communications circuit. “We’re being swarmed
by those damned drones! They’re fixing the cuts faster than we can make them!”
      “Go to full power on the laser cutters,” the captain ordered him. “Get those
cuts done fast so Sandra will calm down.”
      Silence fell for a moment as the engineers in the control room tapped rapidly
through screens. “Making good progress now,” Chen reported. “The cutter is frying
some of the repair drones that get in the way, though.”
      “Damage spreading rapidly in port aft main passageway!” Sandra sounded
very urgent now. “Immediate action required.”
      “Sandra, repairs are underway,” the captain repeated in a frustrated tone.
“Take no action.”
      Kevlin felt something, and glanced back to see that the hatch leading onto the
bridge had opened. No one was there waiting to enter, though.
      One of the crew noticed, too. “Sandra, reseal the hatch to the bridge.”
      “Command understood. Complying,” Sandra replied, her voice its usual dead
calm again.
      The hatch didn’t close. “Sandra, reseal the hatch to the bridge,” the captain
ordered this time.
      “Command understood. Complying.”
      “Captain?” Another engineer was staring at something on his display. “She’s
doing instant resets again after acknowledging our orders. Every interior hatch and
door on the ship is open.”
      The captain stared at him, then spoke in a powerful voice. “Sandra. Close all
doors. Command override Sigma Sigma Sigma.”
      “Command understood. Complying.”
      “It’s not closing,” Yasmina observed.
      “Damn!” one of the crew exclaimed. “We had to give Sandra a reset
capability so she could function autonomously, but she’s started using that to get
around our commands.”
      “She’s not using it,” Yasmina objected. “There’s no conscious thought
involved. I’m sure it must be a defensive response operating below the level of
consciousness. Her subsystems are telling her something has to be done so she’s
working around obstacles to action.”
      “Captain! Airlock doors are opening! Interior and exterior!”
      The captain hesitated the barest fraction of a second before yelling and
springing into action. “Into suits! Everyone into your suits and seal them! Chen!
Ragosa! Seal your suits!”
      The emergency suits were stored next to the seats, fortunately. Kevlin’s hands
were shaking as he pulled on the suit, fumbling with fittings that should have been
second nature after countless emergency drills on the station. A growing breeze was
tugging at him as he struggled to get the chest seal in place.
      “Strap in!” the captain was shouting. “As soon as you get the suit on, strap in
and then get your helmet sealed!”
        Kevlin dropped into his seat and pulled the harness across, clicking it into
place just as the breeze grew to a gale of wind trying to suck him out through the
hatch and ultimately out through the airlock. Wondering if he was really gasping for
air already, Kevlin got the helmet down, trying not to panic as the suit automatically
pressurized. Cool air flowed from the recirc unit and Kevlin slowly got his trembling
under control. Shocked by a sudden realization, he looked over and saw Yasmina
also strapped in, her own suit just finishing pressurizing. Ashamed that he had
forgotten about her, forgotten about anything but his own fears, Kevlin looked away
again.
       “It’s okay,” he heard Yasmina over the suit’s circuit. “Perfectly natural
reaction.”
     Kevlin mumbled a reply, wishing she hadn’t been able to understand his
embarrassment.
      Other voices came over the circuit, the captain’s finally overriding them all. “I
need an estimate as to why that happened. Anybody? Any ideas?”
       It suddenly seemed so obvious. “Sandra is trying to get rid of us,” Kevlin
stated.
      Momentary silence followed that declaration, then the captain came on again in
a deathly calm voice. “Explain that. There’s numerous safeguards built into the
operating system that put human safety at a premium. Sandra can’t attack humans.”
       “She’s not attacking humans,” Kevlin explained, feeling more and more
certain. “Her repair subsystem is attacking an infection. Don’t you see what you’ve
been doing? You’ve been deliberately causing damage to her, on an escalating level.
Her repair system has dealt with it at every stage, evolving the whole time. Well, it’s
a simple leap from being reactive to the damage to reacting to what’s causing the
damage. The cutting back there was the last straw. To Sandra’s repair system, we’re
parasites at best and harmful infections at the worst. Sandra can’t override the
actions of her repair system any more than we can without the help of targeted
medications.”
      “She tried to expel the parasites?” the captain asked. “What happens if that
doesn’t work?”
        “I’d imagine her repair system will go after the parasites directly. Her repair
system is rapidly developing an immune component. I should have seen that coming.
It’s a logical progression for any such system.”
      The captain’s voice rang through the circuit. “Chen! Ragosa! Stop cutting and
get back here!”
         “But we’re almost through—”
       “Stop cutting! We can see a new wave of repair drones headed your way! Get
out of there fast!”
       The wait for the two engineers to return seemed interminable. Chen and
Ragosa were pulling themselves through the hatch when it started closing. They
barely cleared it before it sealed. “We’ve lost all internal control,” someone reported
in a desperate voice.
         Kevlin saw the captain gazing around as if thinking through her next action.
“All right,” the captain announced. “I’m declaring an emergency. All nonessential
personnel are to leave the ship. Get to the boat and stand off in it. We’ve lost
comms to the chase ship, so bring them up to date.”
      The captain and three other crewmembers remained seated, but six of the
engineers unstrapped and began hauling themselves to the hatch, beckoning to
Kevlin and Yasmina to follow them. Kevlin unstrapped as well, making a point of
waiting until Yasmina had done the same and started after the engineers. Scared as
he was, he wouldn’t race ahead of her.
      Two of the engineers had braced themselves and were tugging at something.
The hatch swung open reluctantly under the pressure of the emergency release.
       The journey through the ship to the boat dock was strange. The passageway
was deserted, yet to Kevlin it felt haunted. He couldn’t look at a bulkhead without
thinking of Sandra’s pseudo-life functions pulsing behind them.
       The engineers reached the access panel to the boat dock and wrestled its
manual control until that opened reluctantly as well. The first one who started to
enter the dock stopped and stared. “It’s gone.”
      Kevlin shifted so he could just see over the engineer’s shoulder. The boat,
which should have filled the dock, simply wasn’t there even though the outer hatch
remained sealed. On the deck, a swarm of repair drones were picking at a
diminishing pile of something.
      One of the engineers laughed in a slightly hysterical way. “I was wondering
where Sandra was getting the resources to build so much. She ate the boat.”
       “Oh, God,” another engineer responded. He tried calling the captain, to no
avail. “Back to the control room. Let’s go before those things try to recycle us.”
       The captain gave them a startled glance when they returned, her face setting
into grim lines as her engineers reported what they had seen. “That does it. I’m
pulling Sandra’s plug. Once she’s off, we’ll get aft and shut down the main power
supply.” Unstrapping, the captain went to the aft bulkhead and lifted a cover to
expose a large manual switch.
       Kevlin gave Yasmina a questioning glance as the captain pulled the switch
down. All of the lights went out and Kevlin’s virtual display vanished, leaving only
the lights on the suits to illuminate the control room.
        “The Frankenstein switch,” Yasmina answered Kevlin’s unspoken question.
“Some people also call it the wooden stake or the silver bullet. Every artificial
intelligence system has one built-in that manually cuts all power. Just in case the AI
starts singing ‘A Bicycle Built for Two.’”
      “How many artificial intelligence systems have built-in autonomous repair
capability that can operate without power for a while?” Kevlin asked.
      The captain heard, stared toward Kevlin, then placed one palm over the
bulkhead next to the manual cut-off switch. “I can feel activity behind the bulkhead.”
       “They’ve identified the cause of Sandra’s problem.” The lights came back on.
“And they’ve fixed it. Captain, you’ve got a wonderfully effective simulation of a
living organism here in terms of identifying injuries and taking corrective action. And
it knows what keeps trying to hurt it, and that we just tried to shut down its brain.”
      “Sandra can’t be sentient!”
       “She’s not! It’s all happening at a level way below sentience!” Kevlin yelled.
“Why should that be a surprise? Out of all the threats to human life, how many are
sentient and how many are essentially mindless, like bacteria?”
     A momentary silence fell. “Can we stop the drones if they try to take us out?”
someone wondered.
       “What about the nanos? Sandra’s subsystems have been modifying them right
and left. The rate of evolution seems to be on a exponential curve.”
       “If it’s like the evolution of living organisms, most of the modifications will be
harmful or useless and die out,” Kevlin suggested. “Some of them might even
threaten Sandra.”
      “Most will die out? Or some might further harm the ship? That’s not all that
reassuring, doctor. The seals on our suits are supposed to keep out nanos, but
nothing’s perfect.” The captain gestured. “We’re abandoning ship. Everybody out.
Back to the boat dock.”
       Repair drones of various kinds were visible in the passageways this time as
they pulled themselves through the ship. Kevlin stared as he saw several drones
attack another and disable it. They had to veer to one side as a bulkhead bulged
perilously toward them. In another area, drones were busy dismantling what Kevlin
recognized as a cooling unit. “She needs that!” one of the crew protested. “Why
would Sandra take apart an essential component?”
      “Sandra isn’t,” Kevlin insisted. “Her subsystems are doing it. Just like when
humans run short of calcium and the body robs it from bones to keep the teeth
strong. Part of the repair subsystem thinks some other part of Sandra needs those
components more.”
       The last remnants of the boat had vanished along with the drones that had
digested it for Sandra. The captain and another engineer tugged at the emergency
release on the outer hatch with no results. “I’ll have to blow it using the explosive
bolts.” She yanked open a panel, pulled out a battery, connected leads to two
attachments behind another panel, then pushed a button.
       Faint echoes of the explosions reached Kevlin through his handhold on the
ship as the hatch swung out. The captain turned to face them. “Push yourselves
clear of the ship. We don’t dare wait here for rescue from the chase ship. Go!”
       They went. Kevlin shoved off, looking back to see Sandra’s shape
diminishing behind him, the captain’s suited figure going last out of the hatch. He
heard her calling the chase ship on the distress frequency. “SOS. We need
emergency pick up. Full macro and nano-scale decontamination required. Remain
clear of Sandra. Repeat, remain clear of Sandra.”
        Kevlin wondered how long the recirc unit would keep him alive. Full-scale
decontamination took a while. But then, he couldn’t argue with the captain’s order,
either.
                                            ****
        Yasmina joined him at the display, looking like she’d been vigorously
scrubbed with sandpaper over every part of her body, every hair shaved clean.
Kevlin knew he looked the same, and knew she also felt like her insides had been
similarly sandpapered. He would probably shudder for the rest of his life whenever
someone mentioned a full macro and nano decontamination.
      She gestured at the image of Sandra. “What’s happening? Any guesses?”
Sandra’s clean lines had been distorted by random bulges. Remote readouts showed
system failures cascading through the ship.
       “She’s dying,” Kevlin stated. “Pure and simple. Some of her repair functions
evolved into harmful out-of-control infections. Other parts of her are attacking her.
See this stuff? Any immune system risks getting too efficient. At that point it starts
attacking itself. You can see where all the control system filaments in this part of
Sandra are dead. I’ll bet her own repair system is destroying them.”
      “Autoimmune diseases,” Yasmina observed in a shocked voice.
       “Yeah. The testing process matched with learning routines and an ability to
improve repair capabilities inevitably pushed Sandra into becoming better and better
at identifying and fixing damage. Unfortunately, living organisms are obvious lessons
that there’s no optimum point at which that stuff stops. It keeps trying to get better
even after it gets so good at its job that it turns harmful.”
      The captain had come to stand with them, face sober. “It shouldn’t have
happened. We knew everything there is to know about every one of the components
on that ship.”
      Another engineer shook his shaved head. “It’s a scientific principle that you
can know everything there is to know about something, and still not be able to
predict an outcome. We just proved it again.”
     “Assuming you did know everything,” Kevlin snapped. “You tried to make a
machine work like a living creature, with self-direction and self-repair capabilities.
What made you think you could tell how it would act? Humans are the mature result
of millions of years of evolution and we only function halfway well because of an
enormous investment in cultural, organizational and medical systems designed to
control our actions and compensate for our faults!”
      “What’ll happen to Sandra?” Yasmina wondered.
       The captain glanced at Kevlin. “Do you think she’ll be safe once the power
dies and everything goes dark?”
      “The macro stuff, probably. I don’t know about the nanos. It all depends if
they evolved in the direction of viruses that can remain dormant for almost indefinite
periods while awaiting conditions to reactivate.”
       This time the captain grimaced. “We’ll have to junk her. There’s no telling
how some of her internal components have evolved, so we’ll probably use an
automated drone to shove her onto a trajectory into the sun. We’ll have to severely
limit or block evolution of nanos on the next model. Maybe not even use them.
They’re too hard to track if they do start changing. But we can put limits on the
macro drones, too. We’ll do better next time,” the captain vowed.
      “That statement probably could’ve been carved on a substantial number of
tombstones throughout human history.”
      “Next time will be different,” the captain insisted.
      “You’re right about that,” Kevlin agreed. “Next time I won’t be aboard.”
      “Yes, you will.”
      “No, I won’t. My contract clearly limits the duties to which I can be
assigned.”
       The captain smiled. “If you’re right, these ships will need medical expertise to
identify, diagnose, and treat problems. One of the potential duties listed in your
contract is ship’s doctor. So congratulations. That’s what you’ll be. The ship’s
doctor.”
      Copyright (c) 2007 John G. Hemry