For a stretch of this year I worked on the image side of a harmful-content detection system at a nonprofit. Incoming posts were scored by models, and anything that scored high enough was flagged for a human reviewer. The image model was an Ultralytics YOLO26 object detector. I trained it, built the pipeline that retrains and registers it, and kept its inference path running in production.

This post is about the gap between "the training job ran" and "there is a model you can deploy". In my runs, most failures happened in that gap.

From EC2 scripts to a pipeline

The first training runs were plain EC2: a setup script, a data validator and a thin training wrapper around Ultralytics. One early yolo26s run used these settings:

model: yolo26s.pt
epochs: 100
batch: 32
imgsz: 640
cache: ram
patience: 50
cos_lr: true
save_period: 10

That run logged 28 epochs. Validation mAP@50 went from 0.622 after the first epoch to 0.944 at epoch 28, with precision around 0.92 and recall around 0.90. Each epoch took about 620 seconds on that box. The results were good enough to justify building real infrastructure around the model.

Next I built the training pipeline as a Step Functions state machine. It starts a SageMaker training job and registers the output in the SageMaker Model Registry as PendingManualApproval. An engineer reviews mAP50, precision and recall and approves. The approval event goes through EventBridge to a small ECS deploy task, which updates the inference service's task definition with the new artifact path and does a rolling deploy. The execution input looks roughly like this:

{
  "modelGroupName": "<model-group>",
  "imageTag": "<training-image-tag>",
  "datasetS3Uri": "s3://<models-bucket>/yolo/<dataset>/",
  "instanceType": "ml.g5.2xlarge",
  "volumeSizeGb": 200,
  "maxRuntimeSeconds": 86400,
  "hyperParameters": {
    "model_size": "l",
    "epochs": "40",
    "batch": "-1",
    "imgsz": "640",
    "patience": "12",
    "cache": "disk"
  }
}

The epoch cap, patience, cache mode and volume size in that block all came out of the failures below.

Failure 1: the runtime cap

The config for the hands-off training trigger I was building asked for yolo26l, 100 epochs, on one ml.g5.2xlarge (a single A10G), with maxRuntimeSeconds at 86400. SageMaker caps a job at that runtime and stops it when the time runs out.

The job was killed around epoch 58. That is roughly 25 minutes per epoch, so 100 epochs would have needed about 40 hours. The training entrypoint only copies artifacts to the model directory after Ultralytics finishes. So nothing reached S3: no final weights, not even a partial checkpoint.

The early-stopping setting could not rescue this. With patience: 50 in a 100-epoch run, the model would have had to stall for half the run before stopping.

My first fix was to switch the default to yolo26s with patience: 12. The same day I went back to yolo26l but sized the run to the cap instead of the other way round. I used 28 minutes per epoch for the estimate, which leaves some room for slower epochs:

EPOCH_MINUTES = 28   # yolo26l on one A10G, padded estimate
CAP_HOURS = 24

def max_safe_epochs(margin_hours: float = 5) -> int:
    return int((CAP_HOURS - margin_hours) * 60 // EPOCH_MINUTES)

# -> 40 epochs, about 19h at that epoch time; patience 12 stops earlier only if val mAP stalls

My notes estimated that a 4-GPU instance would finish 100 epochs in about 18 hours, and managed spot looked like a way to cut cost. I documented both as untested options and kept them out of the default. The account's quota was one ml.g5.2xlarge for training, which also meant a YOLO job and a text-classifier job could not run at the same time.

The deeper problem is still there: artifacts only leave the container at the end. Nothing in the pipeline synced checkpoints out during the run, so any job that dies late loses everything.

Failure 2: cache='ram' and a 32 GB host

The next long run used yolo26s on the full dataset on the same instance. It died at epoch 56 of 100. The SageMaker failure reason was a variant of "use an instance type with more memory".

With cache='ram', Ultralytics keeps decoded training images in host memory. A g5.2xlarge has 32 GB, and on the full dataset the process ran out of it partway through. I attributed that to the RAM cache and did not profile memory to confirm what grew over the run. As with the runtime cap, the job was killed before the entrypoint saved anything.

What made this annoying was that I had validated the pipeline about a week earlier with a 10-epoch run, and that run passed. My change was one line with a comment explaining it:

hyperparameters = {
    # Host OOM at epoch 56/100 on the full dataset with cache='ram'.
    # Cache decoded images on the 200 GB EBS volume instead.
    "cache": "disk",
}

The 200 GB volume in the execution input is there for this reason. If you validate a training pipeline with a short run, it is worth one run at production length and full dataset size before you trust the defaults.

Failure 3: a pipeline that reported success when training failed

While getting the pipeline working end to end, I found two bugs in the state machine that hid failures.

The first was in the S3 URI handling. The state machine appends training/ and evaluation/ to datasetS3Uri. If someone passed a .zip path, the job received <uri>.ziptraining/, a prefix that does not exist. I added a guard that rejects anything that is not a prefix ending in /. I also wrote a staging script that copies a dataset into the layout the pipeline expects and prints the correct URI.

The second was worse. The failure branch sent a notification and then ended. Its notify state had End: true, so Step Functions marked the execution SUCCEEDED even though the training job had failed. The fix was to send the notify state on to a real Fail state. The shape, simplified:

"NotifyFailure": {
  "Type": "Task",
  "Resource": "arn:aws:states:::sns:publish",
  "Parameters": { "TopicArn": "${TrainingNotifications}", "Message.$": "States.JsonToString($.error)" },
  "Next": "TrainingFailed"
},
"TrainingFailed": { "Type": "Fail", "Error": "TrainingJobFailed" }

The inference side: scaling a worker to zero

Deployment had its own lesson. Later I moved YOLO inference onto the GPU worker that already served the text classifiers, to share the hardware. A follow-up change then scaled the old CPU YOLO worker to zero whenever the colocated path was on, on the assumption that the GPU worker had replaced it.

It had not. They were two different pipelines. The CPU worker polled a Temporal task queue fed by a schedule, and that path wrote detections to Postgres. The GPU worker handled on-demand requests from a different queue and never touched the scheduled one. With the flag on in production, the scheduled workflow had no poller, and detections stopped reaching Postgres. The service scaled on CPU target tracking. With zero tasks running there is no CPU metric to act on, so it never scaled back up. The fix restored a minimum of one task unconditionally and removed the plumbing that made zero possible.

Before you scale a worker to zero, check which queues it polls. Its replacement may not poll the same ones.

Gating and the auto-trigger

Near the end of my time there I ported the text-classifier baseline comparison to YOLO. It compares a candidate's mAP50, mAP50-95, precision and recall against a stored baseline, with the right direction for each metric. When gating is enabled for an execution, a regression skips registration. I also added a CI job that builds the training image, so a broken image build fails in CI instead of hours into a SageMaker job.

The same change added a "drop a zip, get a model" trigger: S3 event, EventBridge, an ECS task that stages the dataset, then StartExecution. That part did not survive. Adding a bucket notification to a bucket that another stack already managed made the production infrastructure deploy fail, so I reverted the trigger the same day and kept the gating. When I left, the two-script path (stage, then trigger) was still the documented way to train.

What I would do differently

Two of these failures had the same shape: the job ran for many hours and left nothing in S3. The fix I did not get to, syncing checkpoints out during training, would have left a usable checkpoint from each of those runs. After that, I would make one full-length, full-dataset run the acceptance test for any change to training defaults, instead of a short smoke run.