How the service works: from ECC to Azure and back

1. The whole path

The Azure machines cannot see ECC's file system. That one fact shapes everything else: a job on the Azure side runs against a copy of your files, and its results are a copy that you bring back. So every job has the same four steps, and all four are typed on an ECC login node, the way you already work.

1. azsync push copies your ECC home to your Azure home. A folder keeps its path: ~/work on ECC is ~/work on the Azure side, so your job script needs no new paths.
2. sbatch submits the job, with the queue and the account named in the script. The scheduler creates a machine in Azure for it, which takes three to five minutes the first time, and the job's output lands in your Azure home.
3. azsync pull <folder> brings back the folder you name, and only that folder, to the same path on ECC.
4. Remove from the Azure home what you have brought back and what the job no longer needs. Nothing is fetched for you and nothing is deleted for you.

You never log into an Azure machine yourself and you never touch Azure's own tools. The machine shuts itself down five minutes after the last job on it finishes, and you pay for nothing in between. Batch jobs work; interactive jobs (srun --pty) do not on the Azure queues yet.

2. Where your files are

Two homes, one path. Your ECC home is on ECC's file system. Your Azure home is on a file server in Azure, not on ECC; on an Azure machine it is /home/<you>, exactly the path your ECC home has on ECC. azsync push fills the one from the other, and that copy is what your job runs against.

Anything a job opens file by file must be on the Azure side: a Python or conda environment, a git checkout. The link between the two sides carries about 85 small file operations a second, so a job that reached back to ECC for every file would crawl. That is why the copy exists at all.

The Azure home is working space. There are no backups and no snapshots on the Azure side or on ECC. Keep your own copies of anything you cannot afford to lose.

3. Moving data

Small files: azsync push and azsync pull

azsync push copies your whole ECC home to your Azure home and sends only what changed since the last push: the first push of a large environment takes a while, the next one takes seconds. azsync pull <folder> copies one folder of your Azure home back to the same path on ECC. The folder is named relative to your Azure home, so azsync pull test/mycase/output lands in ~/test/mycase/output.

• It never deletes anything, on either side.
• It never replaces a file that is newer on the side it is copying to. Edit a file on ECC and push: the Azure copy is replaced. Let a job write a result on the Azure side and push again: the result is left alone, because it is the newer file. --force switches that rule off and is off by default, on purpose.
• --dry-run counts and estimates without copying. --dated puts a pulled folder in <folder>/azure-<today's date>/, so two runs are never mixed up. --to puts it somewhere else under your home or a project directory.
• Inside a job script, or anywhere azsync is not run from a terminal, add --yes: without a terminal it cannot ask about a copy of more than a minute, so it prints the estimate and copies nothing.
• .ssh, .cache, sockets and editor junk are skipped. Scratch and datasets are never copied.
• A copy is not a job: it uses no machine and no core-minutes are billed for it.

The link moves about 80 MB a second in one stream and about 85 small files a second; whichever of the two is larger is what you wait for. A gigabyte is about 15 seconds, 10,000 files about 2 minutes, 100,000 files about 20 minutes. azsync status shows the Azure file server and when you last pushed; azsync help lists everything it can do.

Large data: Globus

Something very large and very numerous, a dataset, does not belong in azsync. Scratch and datasets are never copied by it. Large data moves with Globus, between the collections "EduceLab Compute Cluster (ECC)" and "CCS Azure home". Both keep modification times, so Globus and azsync agree about which copy of a file is newer. Symlinks and conda environments do not go through Globus: push those with azsync.

A dataset that many jobs read can be loaded once on the Azure side, read only, where it stays; ask for that with a support ticket.

The worked example is 00-transfer-data in /share/examples/azure/ on ECC: a tiny tree to push, a job to run against it and a pull of the result, with the copy log explained line by line.

4. Submitting: which queue

One queue is open today: eus-test, the Azure test queue in East US, billed to the account azr_test. Its machines have two cores and 8 GB (7 GB for jobs), a job may run for 30 minutes, and two machines exist, so two jobs can run at once. Ask for one machine: they have no fast network between them, and a job spread across two would spend its time waiting. Production queues on demand and on spot, a GPU queue and a Central US queue are coming; none of them can be submitted to yet.

Two lines are all that is new in a job script: the queue and the account. Three short headers, from the examples on ECC:

A first test: one core, ten minutes

#SBATCH --partition=eus-test          # the Azure test queue
#SBATCH --account=azr_test            # the account the core-minutes are billed to
#SBATCH --ntasks=1                    # one core; a machine has 2 cores and 8 GB
#SBATCH --time=00:10:00               # ten minutes; the queue allows 30 today

Both cores of one machine for one program (OpenMP)

#SBATCH --partition=eus-test
#SBATCH --account=azr_test
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2             # both cores of the machine for one program
#SBATCH --time=00:30:00               # the queue's limit today

Two MPI ranks inside one machine

#SBATCH --partition=eus-test
#SBATCH --account=azr_test
#SBATCH --nodes=1                     # one machine: no fast network between them
#SBATCH --ntasks=2                    # two ranks, one per core
#SBATCH --time=00:30:00               # the queue's limit today

Longer than 30 minutes, a GPU, or more than one machine is not on the test queue today. Ask for a longer limit with a support ticket; the production and GPU queues will have their own limits when they open. The job script builder writes a complete script for the queue you pick and says what the run will cost.

Time is charged in core-minutes for the cores the job held, from the moment it starts until it ends, whether or not it uses them, rounded up to the whole minute: two cores for five minutes is 10 core-minutes. Ask for the time you need; you are billed for the time the job runs, not for the limit you asked for. On the test queue 1250 core-minutes is approximately one dollar. Dollar figures are estimates from Microsoft's public list prices and can change; the core-minutes are the exact quantity billed to your allocation.

5. Spot and on demand

On demand is the ordinary price: the machine is yours until the job ends. Spot is the same machine at a discount, which Azure can take back at any moment, with about 30 seconds' notice. When that happens the job is killed, and the spot price itself moves. Spot is for work that can be put back in the queue and restarted (#SBATCH --requeue) or that checkpoints, never for a job that must run to the end in one piece.

The test queue runs on demand only; spot is not offered on it yet. A production queue on spot is coming, alongside one on demand. The price page shows both prices for every machine type, and the builder shows what a run costs both ways.

6. When the job ends

azjob <jobid> on an ECC login node prints the job's report: its state and exit code, the machine it ran on, the elapsed time, the core-minutes billed and what is left of the credit. The output path it prints is on the Azure side.

1. Pull the folders you want to keep, by name. azsync pull test/mycase/output. Nothing comes back by itself: output you never name stays in your Azure home, not deleted, and nobody fetches it for you. A pull never overwrites a newer file on ECC, and --dated keeps two runs apart.
2. Check the copy arrived on ECC. ECC is now where your copy lives. There are no backups and no snapshots on the Azure side.
3. Then remove from the Azure home what you have brought back and what the job no longer needs. The Azure home is a shared volume that the service pays for at the size it is provisioned, full or empty, and it grows in steps as it fills. A copy of finished results left there costs the service and protects nothing, because it is not backed up. azsync never deletes, so the removing is done where the files are: the last lines of a job script run on the Azure side, in your Azure home, so a job can remove its own intermediate files before it ends, and a finished case's folder can be cleared the same way, by a short job that does nothing else.

A job's own --output file lands next to the script on the Azure side; azsync pull <the case folder> brings it back with everything else in that folder. Once a job is old enough, the scheduler no longer keeps the output path and azjob prints the working directory instead.

7. Test small first

Run every new job on the test queue at a small scale before you scale anything up: one core or one machine, a short time limit, a small input. The first job of the day waits three to five minutes, occasionally ten, in the state CF while the machine is created; that is normal, so do not cancel and resubmit. A job that arrives while a machine is already up starts within a second. When it finishes, azjob should say COMPLETED (exit code 0:0), and its report tells you what that size cost. Only then make it bigger.

• Two things behave differently from ECC, and both are handled in the examples: MPI uses Intel MPI or OpenMPI, because the MPICH on the software tree needs InfiniBand and these machines have none; and Apptainer needs --no-mount bind-paths, because ECC's configuration binds paths that do not exist on an Azure machine.
• Write results into your Azure home, never into /tmp on the machine.
• GPU work: the GPU queue is coming and is waiting on quota. When it opens, the same rule applies: a short job on one GPU first, read the report, then scale.
• 01-serial-job in /share/examples/azure/ is the smallest complete job, one core and one file, and proves the whole path end to end. Start there.

8. Help

Need help? Open a support ticket. Say which queue and, if a job is involved, the job id: azjob <jobid> prints everything we would otherwise ask for. If a copy is involved, paste the line azsync printed.

No PHI, FERPA-protected, export-controlled or CUI data may be placed on this service. There are no backups and no snapshots on the Azure side or on ECC. Keep your own copies of anything you cannot afford to lose.