Packages, Services, Cron and systemd

On your laptop you start an app by typing node server.js in a terminal. When you close the terminal, the app dies. On a server that is not good enough. The app must start when the machine boots, restart after a crash, write logs somewhere you can read, and run as a limited user. Something has to be in charge of that. On modern Linux that something is systemd. For work at certain times, like a nightly backup, there is cron. And the software itself, from nginx to git, comes from the package manager.

This is lesson 6 of nine. In lesson 5 we reached a remote server over SSH. Now we put it to work. The next lesson, disks, memory, logs and troubleshooting, is about what to do when the server is slow or full. If you use containers, remember that a container usually has no systemd at all; see Docker lesson 0 for how a container runs one main process instead.

Words used in this lesson

• Package is a bundle of files (a program, its docs, its settings) plus a description of what it needs.
• Package manager is the tool that installs, upgrades and removes packages: apt on Debian and Ubuntu, dnf on Fedora and Red Hat, apk on Alpine.
• Repository (repo) is a server with a list of packages, signed so you can trust them.
• Dependency is another package that this one needs to work.
• Service is a program that runs in the background for a long time, like a web server.
• systemd is the first program the kernel starts on most Linux systems (PID 1). It starts and watches services.
• Unit is systemd's word for a thing it manages. A service unit runs a program. A timer unit runs another unit on a schedule.
• Unit file is a small text file (INI style) that describes a unit.
• systemctl is the command to control systemd. journalctl reads the logs systemd collects.
• Journal is systemd's log store.
• Cron is an older scheduler. A crontab is the list of scheduled jobs for one user. A cron job is one line in it.
• Environment is the set of variables (like PATH) a program starts with.
• Idempotent means running something twice gives the same result as running it once.

Packages: how software gets onto the machine

You could download a program from a website, unpack it, and copy it into /usr/local/bin. That works once. Then you must update it by hand, remember what else it needs, and find every file when you want to remove it. A package manager does all of that.

On Ubuntu and Debian there are two layers you should know:

• dpkg works on individual .deb files on this machine. It knows what is installed. It does not fetch from the internet or solve dependencies.
• apt sits on top. It downloads from repositories, works out dependencies, and calls dpkg to do the install.

Looking at what is installed

Let us look at jq, a small JSON tool, which was already installed:

$ dpkg -l jq | tail -1ii  jq             1.7.1-3ubuntu0.24.04.2 amd64        lightweight and flexible command-line JSON processor

The first two letters ii mean: wanted state "install", actual state "installed". Next, what files did the package put on disk, and what package owns a given file?

$ dpkg -L jq | head -6/./usr/usr/bin/usr/bin/jq/usr/share/usr/share/doc$ dpkg -S /usr/bin/jqjq: /usr/bin/jq

dpkg -L pkg lists files of a package. dpkg -S file answers "which package owns this file?" This second one is very handy when you find a strange file and wonder where it came from.

What apt knows about a package

$ apt-cache policy jqjq:  Installed: 1.7.1-3ubuntu0.24.04.2  Candidate: 1.7.1-3ubuntu0.24.04.2  Version table: *** 1.7.1-3ubuntu0.24.04.2 500        500 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages$ apt-cache depends jq | head -4jq  Depends: libc6  Depends: libjq1

Installed is what you have. Candidate is what apt install would give you. The number 500 is the repository's priority. jq depends on a shared library package, libjq1; this is the "dependency" that apt installs for you without being asked.

Installing: always look first

apt-get install -s (-s means simulate) shows the plan without doing it. Here is a package I did not have yet, tree:

$ apt-get install -s treeThe following NEW packages will be installed:  tree0 upgraded, 1 newly installed, 0 to remove and 42 not upgraded.Inst tree (2.1.1-2ubuntu3.24.04.2 Ubuntu:24.04/noble-updates [amd64])Conf tree (2.1.1-2ubuntu3.24.04.2 Ubuntu:24.04/noble-updates [amd64])

Inst is unpack, Conf is configure. Then I really installed it (apt-get install -y tree) and ran tree --version, which printed tree v2.1.1 .... The -y flag answers "yes" to the prompt, which is needed in scripts and Dockerfiles. Be careful: -y also says yes to things you did not read. Do a dry run first on a machine that matters.

Remove versus purge

$ apt-get remove -s treeThe following packages will be REMOVED:  treeRemv tree [2.1.1-2ubuntu3.24.04.2]$ apt-get purge -s treeThe following packages will be REMOVED:Purg tree [2.1.1-2ubuntu3.24.04.2]

The two simulations look similar, but note the marker: Remv versus Purg. remove deletes the program but keeps your config files (in case you reinstall later). purge deletes the config files too. For a service like a database, this difference matters: remove keeps the settings; purge deletes them. I did not test the config-file difference with a package that has config in /etc (the tree package has none), so that sentence comes from the way the two commands are documented, not from this run.

update is not upgrade

This is a classic beginner confusion.

• apt update only refreshes the list of packages and versions from the repositories. It installs nothing.
• apt upgrade installs newer versions of what you have.

Without apt update first, apt works from an old list, and may say a package "has no installation candidate" or install an old version. The lists are stored as files; on this lab machine they live in /var/lib/apt/lists. In a Dockerfile you write apt-get update && apt-get install -y ... in one line (see Docker lesson 1) so a cached old list is never reused.

A simulated upgrade shows what would change:

$ apt list --upgradable 2>/dev/null | head -3Listing...containerd.io/noble 2.4.1-2~ubuntu.24.04~noble amd64 [upgradable from: 2.3.6-1~ubuntu.24.04~noble]docker-buildx-plugin/noble 0.38.0-1~ubuntu.24.04~noble amd64 [upgradable from: 0.37.1-1~ubuntu.24.04~noble]$ apt-get -s upgrade | tail -2Conf librsvg2-2 (2.58.0+dfsg-1ubuntu0.1 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64])Conf libxml2-utils (2.9.14+dfsg-1.3ubuntu3.10 Ubuntu:24.04/noble-updates, Ubuntu:24.04/noble-security [amd64])

Manual versus automatic packages

apt remembers which packages you asked for (manual) and which were pulled in only as dependencies (auto). apt-mark showmanual lists the first kind (842 on this VM); apt-mark showauto | head -3 shows names like bind9-dnsutils. When you remove a package, its automatic dependencies stay behind until you run apt autoremove. Use apt-get autoremove -s first to see the plan.

Where do packages come from?

The list of repositories is in /etc/apt/sources.list.d/. On this Ubuntu 24.04 machine there are two files: ubuntu.sources (the Ubuntu archive) and docker.list (Docker's own repo that someone added). Every repo has a signing key, and apt checks the signature. Adding a third-party repo gives its owner the ability to put software on your server, so add only repos you trust, and prefer to install from the distribution when you can.

Other families (reviewed, not run)

The ideas are the same everywhere. Language package managers (npm, pip) are separate from the system one; see the Node.js series for npm.

Services: why a plain node server.js is not enough

Say you start your API with node server.js & over SSH. When your SSH session ends, the shell sends a hang-up signal, and your app may stop. Even with nohup, nothing restarts it when it crashes, nothing starts it after a reboot, and its logs go who-knows-where. A service manager solves that. On most Linux systems, that is systemd.

Is systemd even running here?

Check what PID 1 is:

$ ps -p 1 -o comm=process_api

On a normal Ubuntu server this prints systemd. In my sandbox it did not, so the commands below that need a running systemd failed with:

System has not been booted with systemd as init system (PID 1). Can't operate.Failed to connect to bus: Host is downNo journal files were found.

That is the same message you would see inside many Docker containers and some WSL setups. If you ever type systemctl in a container and get this, it is not broken: there is just no systemd there. Keep that in mind during interviews: "container" and "systemd" usually do not go together.

A unit file, line by line

A service is described by a unit file, usually in /etc/systemd/system/ (your own units) or /usr/lib/systemd/system/ (units shipped by packages). Here is the file I wrote for a small Node API called myapp:

[Unit]Description=My demo Node APIAfter=network-online.targetWants=network-online.target[Service]Type=simpleUser=deployWorkingDirectory=/opt/myappEnvironment=NODE_ENV=productionEnvironmentFile=-/etc/myapp.envExecStart=/opt/node22/bin/node /opt/myapp/server.jsRestart=on-failureRestartSec=3NoNewPrivileges=yes[Install]WantedBy=multi-user.target

It has three sections.

[Unit] says what this thing is and how it relates to other units.

• Description is the text you see in systemctl status.
• After=network-online.target means "start me after the network is up". After is only about order.
• Wants=network-online.target means "also try to start the network-online target". Wants is about pulling in another unit, and it is soft: if that fails, I still start. (Requires= is the strict version.) The two are separate on purpose: order and dependency are different questions.

[Service] says how to run the program.

• Type=simple means "the process I start with ExecStart is the service". This is the default for a foreground program like node server.js. Other types exist; for example oneshot is for a command that runs, finishes and exits (good for scripts), and forking is for old daemons that detach themselves. If a program forks into the background but you chose simple, systemd thinks it exited.
• User=deploy runs it as that limited user, not as root (lesson 1).
• WorkingDirectory sets the folder it starts in.
• Environment= sets a variable. EnvironmentFile=-/etc/myapp.env reads variables from a file; the - before the path means "do not fail if the file is missing".
• ExecStart is the command. Use the full path to the program. (When I wrote /usr/bin/node the verifier correctly told me the file did not exist on this machine, see below.)
• Restart=on-failure restarts the service if it exits with an error, or is killed by a signal, or times out. Other common values are always and no. RestartSec=3 waits 3 seconds first, so a crash loop does not spin the CPU.
• NoNewPrivileges=yes is a small safety setting: the service and its children cannot gain more privileges, for example through a setuid program (lesson 1).

[Install] matters only for systemctl enable. WantedBy=multi-user.target means "when the machine boots into normal multi-user mode, start this too". enable makes a link so that happens. start runs it now. enable and start are two separate ideas: systemctl enable --now myapp does both.

Checking a unit file without systemd running

systemd-analyze verify reads unit files and reports problems. It worked here even though systemd is not PID 1.

The first version, with the wrong path to node:

$ systemd-analyze verify myapp.servicemyapp.service: Command /usr/bin/node is not executable: No such file or directory

After I changed the path to the real one (/opt/node22/bin/node on this VM), the command printed nothing, which in this tool means "no problems found".

A unit with a typo in a key name:

$ systemd-analyze verify bad3.servicebad3.service:9: Unknown key name 'Typo' in section 'Service', ignoring.

And a bad value, Restart=sometimes:

$ systemd-analyze verify bad.servicebad.service:6: Failed to parse service restart specifier, ignoring: sometimes

The verifier has limits. I put User=nosuchuser in bad3.service and it did not complain, because that check needs the real system. So "verify is clean" means the syntax is fine, not that the service will start. I also tried ExecStart=node server.js (no full path) and verify did not complain either. I did not read why; systemd can find a bare command name in its own search path in newer versions, but I still recommend a full path so behaviour does not depend on PATH.

You can also ask for a rough security score:

$ systemd-analyze security --offline=true myapp.service | tail -1→ Overall exposure level for myapp.service: 8.8 EXPOSED :-(

The scale runs from 0 (locked down) to 10 (wide open). Our tiny unit scored 8.8 because it uses almost none of the sandboxing options (ProtectSystem=, PrivateTmp=, SystemCallFilter= and more). The list printed shows what is missing, line by line. You do not need a perfect score, but it is a cheap checklist.

The commands you will use (reviewed, not run)

These need a running systemd, so I could not run them. This is the common set:

sudo cp myapp.service /etc/systemd/system/sudo systemctl daemon-reload        # tell systemd to re-read unit filessudo systemctl enable --now myapp   # start now AND at every bootsystemctl status myapp              # state, recent logs, main PIDsudo systemctl restart myapp        # stop and startsudo systemctl reload myapp         # ask it to re-read config, if it supports thatsystemctl is-active myapp           # prints "active" or "inactive" (good for scripts)systemctl list-units --type=service --state=failed

The step people forget is daemon-reload: after you edit a unit file, systemd still has the old copy in memory until you run it. Another good habit is to use sudo systemctl edit myapp, which creates an override file for a packaged unit instead of changing the package's own file (which a package upgrade could overwrite).

Reading logs with journalctl (reviewed, not run)

systemd collects the standard output and standard error of every service into the journal. So your Node app only needs to console.log; you do not need log files. Commands:

journalctl -u myapp                  # all logs of the myapp unitjournalctl -u myapp -f               # follow live, like tail -fjournalctl -u myapp --since "1 hour ago"journalctl -u myapp -p err           # only errors and worsejournalctl -b                        # this boot onlyjournalctl -u myapp -n 50 --no-pager

In this sandbox the only output I got was No journal files were found. For logs without systemd, and how to search them, see lesson 7. If you want the bigger picture of structured logs, see Observability lesson 1.

Why Restart= needs care

A service that crashes immediately and restarts every 3 seconds is a crash loop. systemd has a start-rate limit, so it will eventually stop trying and mark the unit failed. This protects the machine, but it also means your service stays down until someone looks. For a service that must stay up, combine Restart= with monitoring that tells you about it. I did not trigger a crash loop here because systemd was not running; this is from the systemd.service and systemd.unit manual sources I read, which describe restart delays and start limits.

Cron: a job at a time

Cron is a program (a daemon named cron) that wakes up every minute, reads everyone's crontab files, and starts the jobs whose time has come. You edit your own list with crontab -e, show it with crontab -l.

The five fields

A cron line is five time fields and then a command:

┌───────── minute        (0-59)│ ┌─────── hour          (0-23)│ │ ┌───── day of month  (1-31)│ │ │ ┌─── month         (1-12)│ │ │ │ ┌─ day of week   (0-7, 0 and 7 are Sunday)│ │ │ │ │* * * * *  command

Some examples, and I checked each with the real Node library cron-parser 4.9.0, starting from a fixed time (12 October 2026, 03:50 UTC):

*/15 * * * *     2026-10-12T04:00  2026-10-12T04:15  2026-10-12T04:3030 2 * * *       2026-10-13T02:30  2026-10-14T02:30  2026-10-15T02:300 9 * * 1-5      2026-10-12T09:00  2026-10-13T09:00  2026-10-14T09:0030 4 1,15 * 5    2026-10-15T04:30  2026-10-16T04:30  2026-10-23T04:300 0 31 2 *       ERROR: Invalid explicit day of month definition61 * * * *       ERROR: Constraint error, got value 61 expected range 0-59

Read them as: every 15 minutes; at 02:30 every day; at 09:00 Monday to Friday; and the surprising one.

The last two lines show invalid schedules being rejected: February 31 does not exist, and minute 61 is out of range. (I wanted to check syntax with crontab -T, which the cronie manual mentions, but the Ubuntu cron program said invalid option -- 'T'. Different cron programs have different options, so check man crontab on your machine. A Node parser is a handy extra check, but it might not agree with every cron program on edge cases.)

There are also shortcuts like @daily, @hourly, @reboot that some cron programs accept.

What cron really does: a real run

I started the real cron daemon in the sandbox and installed a crontab for the deploy user with two jobs that write what they see into a file:

* * * * * echo "PATH=$PATH SHELL=$SHELL pwd=$(pwd) user=$(whoami)" >> /tmp/lab-linux-b/pkg/cron.out 2>&1* * * * * which node jq >> /tmp/lab-linux-b/pkg/cron.out 2>&1

About a minute later the file contained:

PATH=/usr/bin:/bin SHELL=/bin/sh pwd=/home/deploy user=deploy/usr/bin/jq

This is the single most important thing to know about cron. A cron job does not run in your normal login environment:

• PATH was only /usr/bin:/bin. My node (in /opt/node22/bin, which my shell has on its PATH) was not found: which node jq printed only jq. This is the number-one reason "my script works in the terminal but not from cron".
• SHELL was /bin/sh, not bash. Bash-only features may not work.
• The working directory was the user's home folder, not where your script lives.
• No .bashrc, no nvm, no exported variables from your session.

The fix: use full paths for commands (/opt/node22/bin/node /opt/app/job.js), or set PATH= or SHELL= at the top of the crontab, and cd to the right folder inside the job.

The crontab file itself is stored here, with strict permissions:

$ ls -la /var/spool/cron/crontabs/-rw------- 1 deploy crontab  378 Oct 12 03:45 deploy

You do not edit that file directly; use crontab -e so the syntax is checked on save.

The percent sign trap (real run)

In a crontab line, the character % is special: the cron program turns an unescaped % into a newline, and the text after the first one is passed to the command as standard input. I tested three jobs:

* * * * * echo "bad: $(date +%H)" >> /tmp/lab-linux-b/pkg/cron.out 2>&1* * * * * echo "ok: $(date +\%H)" >> /tmp/lab-linux-b/pkg/cron.out 2>&1* * * * * date +%H >> /tmp/lab-linux-b/pkg/cron.out

The output file got exactly one line:

ok: 03

Only the line that escaped the percent (\%H) worked. The other two produced nothing in the file: the first became a broken shell command, the second lost its redirect because everything after % was cut off and became input. Their error output had nowhere to go. Which shows another cron fact: cron emails output and errors to the user, and on a server with no mail program the message just disappears. I saw /var/mail was empty. The lesson: redirect output yourself (>> /var/log/job.log 2>&1) and escape every % as \%, or put the command in a script file.

Other cron habits

• Time zone: cron runs in the server's local time zone (often UTC on cloud servers). A job at "02:30" can be missed or repeated around a daylight-saving change; the manual notes that times that do not exist (the "missing hour") never match. I did not test DST here.
• Overlap: if a job takes longer than its interval, cron starts another copy. Use flock -n /tmp/job.lock command so a second copy does not start (reviewed, not run).
• Silent failure: log to a file, and check the exit code. Add a "heartbeat" check so someone is told if the job stops running.
• Idempotent jobs: if a job can be run twice by mistake, it should be safe to do so.

systemd timers: cron's modern cousin

A timer unit runs a service unit on a schedule. It has two files that share a name: backup.timer and backup.service. The service describes what to run. The timer says when.

# backup.service[Unit]Description=Nightly backup[Service]Type=oneshotExecStart=/bin/true

# backup.timer[Unit]Description=Run backup every night[Timer]OnCalendar=*-*-* 02:30:00Persistent=trueRandomizedDelaySec=5min[Install]WantedBy=timers.target

systemd-analyze verify backup.timer backup.service printed no problems. (I replaced the real backup command by /bin/true because this is only a demo of the file shapes.) You enable the timer, not the service: systemctl enable --now backup.timer (reviewed, not run).

Why use timers instead of cron?

• Persistent=true remembers the last run on disk. If the machine was off at 02:30, the job runs when the machine comes back. Cron simply misses it. This only works for OnCalendar= timers, according to the manual.
• RandomizedDelaySec=5min adds a random delay so a hundred servers do not all hit the same backup server at 02:30. The related setting AccuracySec defaults to one minute, which lets systemd group wake-ups to save power; set it small if you need exact timing.
• Output goes to the journal, so journalctl -u backup.service shows each run. No lost mail.
• You can see what is planned: systemctl list-timers (reviewed, not run).

The calendar syntax, explained by a tool

The time format is different from cron. Instead of guessing, ask systemd-analyze calendar. It works without a running systemd:

$ systemd-analyze calendar '*-*-* 02:30:00'Normalized form: *-*-* 02:30:00    Next elapse: Tue 2026-10-13 02:30:00 UTC       From now: 22h left$ systemd-analyze calendar 'Mon..Fri *-*-* 09:00' --iterations=3 | tail -7Normalized form: Mon..Fri *-*-* 09:00:00    Next elapse: Mon 2026-10-12 09:00:00 UTC       From now: 5h 15min left   Iteration #2: Tue 2026-10-13 09:00:00 UTC       From now: 1 day 5h left   Iteration #3: Wed 2026-10-14 09:00:00 UTC       From now: 2 days left

And the shortcut words expand like this:

A bad expression is rejected clearly:

$ systemd-analyze verify bad4.timerbad4.timer:2: Failed to parse calendar specification, ignoring: every nightbad4.timer: Timer unit lacks value setting. Refusing.

Note both lines: the unit is not just "ignored", it is refused because a timer with no valid schedule is useless.

For "run N seconds after boot" or "every 10 minutes after the last run finished", timers have OnBootSec= and OnUnitActiveSec=. Check systemd.timer(5) for details; I read the page but did not run these.

Cron or timer?

My rule: on a normal Linux server with systemd, use timers for anything important. Use cron for something quick, for compatibility, or where systemd is not there.

A short checklist before you ship a service

1. Does the unit use a full path and a non-root User=?
2. Did systemd-analyze verify say nothing?
3. After daemon-reload and enable --now, does systemctl status say active (running)?
4. If you kill the process with kill -9, does Restart= bring it back? (reviewed, not run)
5. Can you read its logs with journalctl -u?
6. For scheduled jobs: are all paths full, all % escaped, output redirected, and overlaps prevented?

Try it

You need a Linux machine. Use a VM or a Linux box where you may install things.

1. Run dpkg -S $(command -v ls) (or rpm -qf on Fedora). Which package owns ls?
2. Run apt-get install -s htop (or any package you do not have). What does Inst tell you? Did it add dependencies?
3. Write a small hello.service with Type=oneshot and ExecStart=/bin/echo hello. Run systemd-analyze verify hello.service. Now misspell ExecStart as ExecStrat. What does verify say?
4. Run systemd-analyze calendar 'Mon..Fri *-*-* 09:00' --iterations=3. Why is the first answer not always today?
5. Add this to crontab -e: * * * * * date +%s >> /tmp/c.out. After two minutes, what is in the file? Then fix the line.
6. Predict, then check with a cron parser or by reasoning: when does 0 9 1 * 1 run?

Answers

1. On most Debian systems it is coreutils. (ls belongs to a package of basic tools.)
2. Inst means "would be installed". If the package depends on libraries you lack, those appear as extra Inst lines.
3. I ran it. The misspelled key gave Unknown key name 'ExecStrat' in section 'Service', ignoring. and then Service has no ExecStart=, ExecStop=, or SuccessAction=. Refusing. A service needs a command to run, so this typo is not just ignored; the whole unit is refused.
4. If it is already past 09:00 today or it is a weekend, the next time is the next weekday morning.
5. The file will be empty or wrong: %s is cut at the percent sign. The fix is date +\%s.
6. At 09:00 on the 1st of every month and at 09:00 every Monday (both day fields restricted, so either matches).

Interview questions

Key takeaways

• dpkg is the low-level package tool; apt adds repositories and dependencies. apt update refreshes lists; apt upgrade installs. Use -s to look before you act.
• remove keeps config files; purge deletes them too.
• A service needs a supervisor. A systemd service unit has [Unit], [Service], [Install]. enable is for boot, start is for now, and daemon-reload is needed after edits.
• systemd ignores bad lines silently, so run systemd-analyze verify before you trust a unit. It checks syntax, not whether the user exists or the service will start.
• Cron runs with a tiny environment (PATH=/usr/bin:/bin, sh, home directory). Use full paths, redirect output, escape %.
• If both cron day fields are set, the job runs when either matches.
• Timers add logs in the journal, catch-up after downtime, and load spreading.
• Containers often have no systemd. In this sandbox systemctl and journalctl -u could not run, so those commands in this lesson are from the manuals, not from my output.

What's next

In lesson 7 we handle the bad day: the server is slow, the disk is full, or memory is gone. You will see df say "100%" for two different reasons, a deleted file that still takes space, a runaway process, and a step-by-step method to find the problem. Practise now: write a one-line cron job for yourself that logs the date to a file, with full paths and an escaped percent sign, and confirm it writes after one minute.