How to monitor cron jobs and catch silent failures in production

To monitor a cron job, have the job report its own success: after the work finishes, it pings an external heartbeat monitor, and the monitor alerts when the ping does not arrive within the expected interval plus a grace period. That catches jobs that never start, jobs that fail, and servers that go offline.

Key takeaways

• ▸HTTP uptime checks cannot see scheduled work. A job has to report its own execution status.
• ▸Send the heartbeat with &&, never ;, so it only fires when the job exits successfully.
• ▸Set the grace period from the job's real run time and scheduling delays, not a guess.
• ▸A heartbeat should mean the work succeeded, so validate the result (file size, record count) before you ping.
• ▸Test every monitor with a controlled failure. A monitor that has never alerted is an untested dependency.

Your cron job is running. Or is it?

Cron is one of the simplest and most reliable ways to schedule work on a server. Need to run a database backup every night? Cron can handle it. Need to sync data with an external API every hour? Cron can handle that, too. Need to generate reports, process files, clean up expired sessions, or execute a scheduled task in your Django application? Cron is probably somewhere in the solution.

But there's a problem. Cron can tell your server to run a command. It cannot guarantee that the command completed successfully. A scheduled job can fail in several ways:

• ▸The server goes offline.
• ▸The cron daemon stops running.
• ▸A crontab entry is accidentally removed.
• ▸A script exits with an error.
• ▸An external API becomes unavailable.
• ▸A database connection times out.
• ▸A job starts but never finishes.
• ▸A job completes successfully but produces incorrect results.

In many of these situations, your application won't generate an obvious alert. Your nightly backup might stop running. Your data synchronization might silently fail. Your background worker might stop processing jobs. And you might not discover the problem until a customer reports it.

This is why monitoring cron jobs and background processes is an essential part of production infrastructure monitoring. This guide covers how to monitor scheduled tasks, implement heartbeat monitoring, detect failed executions, and build a more reliable monitoring strategy for production systems.

Why isn't traditional uptime monitoring enough?

Most developers start with HTTP uptime monitoring. You configure a monitoring service to request an endpoint every few minutes:

GET https://example.com/health

If the endpoint responds successfully, the service considers your application healthy. This works well for monitoring websites and APIs. However, it doesn't tell you whether a scheduled task is working. Consider a nightly backup process:

0 2 * * * /usr/local/bin/backup.sh

Your website might be responding perfectly while the backup job hasn't run in three days. Your API might return HTTP 200 while a background worker has stopped processing messages. Your application might be healthy while its data synchronization process is failing.

The important distinction is that scheduled jobs need to report their own execution status. This is where heartbeat monitoring becomes useful.

What is heartbeat monitoring?

Heartbeat monitoring is a technique in which a process periodically sends a signal to an external monitoring service. The signal indicates that the process is alive or has completed its expected work. For a scheduled task, the process might send a heartbeat after successfully completing its execution.

The monitoring service tracks when the heartbeat should arrive. If the expected signal doesn't arrive within a configured time window, the service generates an alert. This is sometimes called a dead man's switch. The concept is straightforward:

SCHEDULED JOB
      |
      v
EXECUTES WORK
      |
      v
COMPLETES SUCCESSFULLY
      |
      v
SEND HEARTBEAT
      |
      v
EXTERNAL MONITOR
      |
      v
EXPECTED SIGNAL RECEIVED?
      |
      +---- YES ----> HEALTHY
      |
      +---- NO -----> ALERT

The important advantage is that the monitoring system doesn't depend on the job successfully running in order to detect a failure. If the server goes offline or the scheduler stops working, the heartbeat stops arriving, and the monitoring system can detect the missing signal.

In Faultline, a heartbeat is a monitor type like HTTP or ECS. You give it an interval and a grace period, and your job calls a unique ping URL when it succeeds. If no ping arrives within interval plus grace, the check fails and goes through the same incident, escalation, and runbook pipeline as every other monitor. The token in the URL is the only credential, so the server running the job needs no API key.

How do you add a heartbeat to a cron job?

Let's start with a simple example. Suppose you have a backup script:

#!/bin/bash

/usr/local/bin/database-backup

You want to execute it every night at 2 AM. Your crontab might look like this:

0 2 * * * /usr/local/bin/backup.sh

Now let's add a heartbeat. After creating a heartbeat monitor, you get a unique endpoint that the job can contact. With Faultline, one CLI command creates the check and prints its ping URL:

faultline check create nightly-backup --type heartbeat \
  --interval 86400 --grace 1800

# Ping URL, call this at the end of your cron or worker run:
#   curl -fsS https://api.fltln.io/hb/hb_AbC123...

Your cron entry might look like this:

0 2 * * * /usr/local/bin/backup.sh && curl -fsS -m 10 -o /dev/null https://api.fltln.io/hb/hb_YOUR_TOKEN

The && operator is important. It ensures that the heartbeat is sent only if the backup script exits successfully. If the backup fails, the heartbeat isn't sent, and the monitoring service can detect that the expected completion signal never arrived.

Why not use a semicolon?

This would be incorrect:

/usr/local/bin/backup.sh; curl https://api.fltln.io/hb/hb_YOUR_TOKEN

A semicolon executes the second command regardless of whether the first command succeeds. Your monitoring service could receive a successful heartbeat even though the backup failed. That defeats the purpose of monitoring.

A more robust implementation

For production systems, I'd recommend handling logging and execution status explicitly.

#!/bin/bash

set -euo pipefail

LOG_FILE="/var/log/backup.log"
HEARTBEAT_URL="https://api.fltln.io/hb/hb_YOUR_TOKEN"

echo "Starting backup: $(date -Is)" >> "$LOG_FILE"

/usr/local/bin/database-backup >> "$LOG_FILE" 2>&1

echo "Backup completed: $(date -Is)" >> "$LOG_FILE"

curl \
  --fail \
  --silent \
  --show-error \
  --max-time 10 \
  --retry 3 \
  --output /dev/null \
  "$HEARTBEAT_URL"

This example:

• ▸Stops execution when the backup fails.
• ▸Records the execution in a log.
• ▸Sends a heartbeat only after successful completion.
• ▸Retries transient heartbeat failures.
• ▸Limits the time spent waiting for the monitoring endpoint.

The exact implementation will depend on your monitoring service and the behavior of your job. For example, if the backup can partially succeed, you'll want to validate the resulting backup file before reporting success. A successful exit code alone doesn't always mean the work produced the intended result.

How do you detect a job that never starts?

Heartbeat monitoring is particularly useful for detecting jobs that never execute. Consider a scheduled data synchronization task that should run every 15 minutes:

*/15 * * * * /usr/local/bin/sync-data.sh

If it stops running, the external monitoring service eventually notices that the expected heartbeat is missing. This can detect problems such as:

• ▸A server becoming unavailable.
• ▸A cron daemon stopping.
• ▸A scheduled task being removed.
• ▸An invalid cron configuration.
• ▸A process failing before it reaches the heartbeat step.

The monitoring system should be configured with an expected schedule and an appropriate grace period. For example:

The grace period prevents alerts when a job is slightly delayed. For jobs with complex schedules, use a schedule-aware monitor rather than assuming a fixed interval. A job that runs every weekday at 9 AM needs different scheduling logic from one that runs every 15 minutes.

Faultline heartbeats use a fixed interval plus grace today; there is no cron-expression schedule yet. For the 15-minute sync above, that maps directly: --interval 900 --grace 300. For a weekday-only job, the interval has to cover the longest gap, the weekend, which means a missed weekday run is caught later than it would be with a schedule-aware monitor.

How do you detect a job that starts but never finishes?

A basic heartbeat tells you whether a job completed. It doesn't necessarily tell you whether the job started and then became stuck. For that, you can use a start signal and a completion signal:

JOB STARTS
    |
    v
SEND START SIGNAL
    |
    v
PROCESSING
    |
    v
JOB COMPLETES
    |
    v
SEND SUCCESS SIGNAL

If the start signal arrives but the completion signal doesn't arrive within the expected duration, the monitoring system can identify a potentially stuck execution. This is useful for:

• ▸Long-running data imports.
• ▸Database migrations.
• ▸Background workers.
• ▸Report generation.
• ▸Large file processing.
• ▸Scheduled API synchronization.

For example, a job normally takes between 5 and 10 minutes. You might configure a maximum expected duration of 20 minutes. If the job starts but doesn't report completion within that window, the monitoring service can generate an alert. This is more useful than simply checking whether the job ran at some point during the day.

Faultline does not track start signals or a maximum run duration yet; a heartbeat records successful completion only. A stuck job is still caught, just later: it never sends its completion ping, so the check fails once interval plus grace passes.

Why monitor the result, not just the process?

One of the biggest mistakes in production monitoring is assuming that a successful process exit means the work succeeded. Consider a database backup. The backup script might exit with code zero while producing an empty file. Or a data synchronization task might complete successfully while importing zero records because an upstream API returned an unexpected response.

Your monitoring should verify the actual outcome. For a backup, that might mean:

• ▸Confirming the file exists.
• ▸Checking that the file size is reasonable.
• ▸Verifying the archive.
• ▸Confirming that the backup completed within the expected time.
• ▸Periodically testing restoration.

For a data synchronization task, it might mean:

• ▸Confirming the expected records were processed.
• ▸Validating the response from the external API.
• ▸Checking that the resulting data meets application requirements.
• ▸Recording the number of successful and failed operations.

A heartbeat should represent meaningful success. It should not simply mean that a process reached its final line of code. With Faultline, put those checks in the script before the ping line, so a bad backup simply never pings and the missed heartbeat opens the incident.

How do you monitor background workers?

Cron jobs aren't the only processes that need monitoring. Background workers can fail in ways that are just as difficult to detect. For example, consider a Django application using Celery. Your application might enqueue a task:

from celery import shared_task

@shared_task
def process_invoice(invoice_id):
    invoice = Invoice.objects.get(id=invoice_id)

    # Process invoice
    invoice.process()

The task might execute successfully. But what happens if the worker stops running? What happens if the queue grows faster than the workers can process it? What happens if a worker becomes stuck? Monitoring the worker process alone won't necessarily tell you whether the application is keeping up with its workload.

For background workers, consider monitoring:

• ▸Worker availability.
• ▸Queue depth.
• ▸Oldest pending task age.
• ▸Task execution duration.
• ▸Failed task count.
• ▸Retry count.
• ▸Last successful task execution.

A worker that is technically alive but hasn't processed a task in several hours may be unhealthy. The same principle applies to Docker containers, ECS services, and other infrastructure components. A process being alive is not the same as a service being healthy.

Faultline covers the worker side of that list. The CLI and heartbeats docs include Python and Node.js workers that ping from the consume loop, even when the queue is empty, so a wedged worker stops pinging. Docker's own healthcheck can send the ping for a container, and ECS checks compare a service's running tasks to its desired count. Faultline does not read queue depth, task age, or retry counts; keep those in your queue's own metrics.

What are the common cron monitoring mistakes?

Monitoring only the server

A server can be healthy while a scheduled job is failing. Monitor the job itself.

Sending a heartbeat before the work completes

If the heartbeat is sent before the task finishes, the monitoring service may report success even if the task subsequently fails. Send the success signal after the work and its required validation complete.

Using an unrealistic grace period

If a job normally takes 20 minutes, a 2-minute grace period will create unnecessary alerts. Use historical execution duration and expected scheduling delays to determine an appropriate threshold.

Ignoring time zones

Cron schedules commonly use the server's local time zone, and your monitoring configuration must account for that. Daylight saving time can also affect scheduled execution. For important jobs, explicitly document the expected time zone and verify the behavior around clock changes. Faultline heartbeats measure elapsed time since the last ping, not a wall-clock schedule, so on a daylight saving change a daily job's real gap becomes 23 or 25 hours. Give daily jobs at least an hour of grace.

Never testing the alert

A monitor that has never generated an alert is an untested dependency. Test the monitoring configuration by intentionally causing a controlled failure. Verify that:

• ▸The job failure is detected.
• ▸The expected incident is created.
• ▸The notification reaches the correct person.
• ▸The alert contains enough information to investigate.
• ▸Recovery is detected correctly.

You should also test what happens when the monitoring endpoint itself is unavailable. The Faultline quickstart ends with firing a real test incident end to end, and faultline push gives up after 10 seconds, so a monitoring outage never hangs your job (with curl, -m 10 does the same).

How do you build a more reliable monitoring strategy?

A good production monitoring strategy should cover multiple layers.

Each layer answers a different question. Infrastructure monitoring tells you whether the underlying systems are healthy. Application monitoring tells you whether the application is responding correctly. Heartbeat monitoring tells you whether scheduled work is completing when expected. Incident management helps ensure that detected problems reach the right people.

Automated remediation can take this a step further by executing approved operational procedures when a failure occurs. The goal isn't to generate more alerts. The goal is to detect meaningful failures, understand their impact, and respond effectively.

Where does Faultline fit?

Monitoring a production system involves more than checking whether a website responds. Your infrastructure may include APIs, Docker containers, ECS services, background workers, scheduled jobs, queues, internal services, and external dependencies.

Faultline brings infrastructure monitoring and incident response together. Instead of relying on a server's own logs or a developer noticing that a scheduled job failed, you monitor expected behavior and detect failures when they occur. Faultline helps teams:

• ▸Monitor HTTP, TCP, SSL, and DNS endpoints, Docker containers, ECS services, and cron jobs and workers through heartbeats.
• ▸Detect failures and missed check-ins, and open an incident when something goes wrong.
• ▸Escalate to whoever is on call through Slack, PagerDuty, email, or a webhook.
• ▸Investigate with the incident timeline, recent checks, and an AI summary.
• ▸Run runbooks that wait for your approval by default, then verify recovery 2 minutes later and page on-call if the fix did not hold.

For teams that want to bring AI into their operational workflows, Faultline also has an MCP server, so Claude can check which jobs are failing, read the incident, and, after you approve, run the runbook. The objective is straightforward: detect problems before they become customer-facing incidents, and make the response process more reliable.

Conclusion

Cron is an excellent scheduling tool. But scheduling a task and monitoring its successful completion are two different responsibilities. A reliable production monitoring strategy should account for jobs that never start, jobs that fail, jobs that become stuck, and jobs that complete without producing the expected result.

Heartbeat monitoring provides a simple and effective way to detect missing execution signals. Combine it with execution tracking, realistic grace periods, outcome validation, and tested alerting, and you'll have a much stronger foundation for reliable background processing.

Because the most dangerous production failure isn't always the one that throws an exception. Sometimes, it's the job that quietly stops running. Know before it breaks.