Running Python well on LINCOMM: scripts vs. the interactive shell

Why a saved Python script is the right way to run real work on LINCOMM, when the interactive shell and notebooks still help, and how to keep results reproducible.

There are two ways to run Python on LINCOMM (Linux Community Servers): typing into the interactive shell, or running a saved script. Both are useful, but for the real work you came to LINCOMM to do, a script is almost always the better choice. This page explains why, and how to keep your results reproducible.

The two ways to work

The interactive shell (you start it by typing python) runs one statement at a time and shows the result immediately. Together with notebooks, it's ideal for exploring: inspecting a dataframe, trying a chart, checking what a function returns.

A script is a plain .py file holding the whole analysis, which you run start to finish:

python analysis.py

Why a script wins for real jobs

Once you know what you want to compute, a script beats the interactive shell for three reasons:

Reproducibility. A script is the exact, complete record of what you did. Run it again and you get the same result. Lines typed into a shell are easy to lose and hard to trust.
It runs unattended. LINCOMM is for long jobs. A script started inside tmux runs to completion on its own, even if your connection drops — you don't have to feed it statements. See Keep work running with tmux and How to run and manage jobs on LINCOMM.
It travels. A script with relative paths runs unchanged on your computer and on LINCOMM, and you can share it with a colleague. See How to use relative paths in Python.

Notebooks are for exploring, not for the heavy run

Notebooks are excellent for working through an idea, because you can run a cell, look at the output, and adjust. But a notebook run out of order can hide bugs and quietly produce results you can't reproduce. When the analysis is settled, move the code that matters into a plain .py script and run that for the full job. Keep the notebook for exploration; let the script be the thing of record.

Keep the environment reproducible

A script only reproduces a result if it runs against the same packages. Differences between the Python on your computer and the Python on LINCOMM are a common reason a script behaves differently in the two places. Recording the package versions your script needs — for example in a requirements.txt file kept beside it — makes it far easier to get the same setup in both. This pairs naturally with developing locally first; see Test your script locally, then run it on LINCOMM.

A good working pattern

The two modes work best together. Explore in the shell or a notebook until you know what you want. As soon as a piece of code works, move it into your .py script. When the script is complete, run it with python under tmux for the full job. You get the speed of exploration and the reliability of a script.

Limitations and trade-offs

A script is less convenient for open-ended exploration, where you don't yet know your next step. That's what the shell and notebooks are for.
Reproducibility depends on the environment, not just the code. Matching package versions across machines takes a little discipline, but it prevents surprises.