Could replacing a bash script with a new version cause a running instance of the script to fail

Question

I am running bash scripts from java programs on a server. I just uploaded a new version of the script intending the next run of the script to use the version. I did not mean to interrupt the existing, running instances of the script. However, I just received over 100 notifications of crashes from my 300 servers. I'm guessing replacing the running bash script with a new version caused this. However, this would require that the running bash script is reading from the disk as it gets to each new step. Is this how it works?

The running versions of the bash script run some ray tracing software. Each run takes 2 hours. Sub-steps take between 5 minutes and 1.5 hours. The script always reports crashing after finishing a step in the script. It never reports crashing an already running sub-step. Some crashes report not finding commands that I cannot find in the script. Different crashes report different places.

help!

EDIT: I copied the script to all 300 servers using scp. The file was replaced on the file system. This is not a shared-file.

Answer 1

SiegeX is half right - bash will load an entire script into memory, so a script can continue to run even if it's source file is deleted while the process is running. But bash will also check whether the source file is updated while the script is running. If it has been, bash will reload it and continue running it from the current position reopen the file, seek to the current position of the script, and continue running the script from that point.

Here's a proof-of-concept script:

# If you modify a script, will it change the behavior of
# processes that are currently running that script?
# Does this script print "Foo" or "Bar"?

cat >foo.sh <<EOF
sleep 5
echo Foo
EOF

bash foo.sh &
sleep 2

cat >foo.sh <<EOF
sleep 5
echo Bar
EOF

wait

So the upshot is don't modify the source files of bash scripts if you care about the processes that are currently running that script.

---

(This script, however, displays "Foo". The "current position" of the bash script is always at the beginning or end of a line.)

echo "sleep 5 ; echo Foo" > foo.sh
bash foo.sh &
sleep 2
echo "sleep 5 ; echo Bar" > foo.sh
wait

Answer 2

Don't update a running system if you can avoid it

---

Deleting the script is one thing, but modifying it may produce more "interesting" results.

Also, changing a file that is replicated and/or network-mounted introduces behavior specific to the filesystem and deployment protocols. These are not going to be modelled accurately by a simple test on a local hard mount or one where a network mount is modified on the same system that is reading the file.

Furthermore, "uploading" this file to 300 servers introduces all kinds of wonderful complexity that us overflowians probably don't have nearly enough information to analyze.

ISTM that your issues probably are related to the update. I think the mystery commands may come from bash reading part of a script from the old version and part from the new version. I do know that you should probably shut down the subsystem, if possible, while updating.

Answer 3

I'm not sure if things have changed since mob's answer, which has been accepted, but in bash 4.3.46 (such as comes with ubuntu 16.04), it is true that bash monitors the script file for changes, but this is broken if the file is deleted.

So a slight modification to his script does the 'right' thing:

# If you modify a script, will it change the behavior of
# processes that are currently running that script?
# Does this script print "Foo" or "Bar"?

cat >foo.sh <<EOF
sleep 5
echo Foo
EOF

bash foo.sh &
sleep 2

rm foo.sh

cat >foo.sh <<EOF
sleep 5
echo Bar
EOF

wait

This now prints Foo.