A Service That Restarted 480,000 Times While Dead: Reviving a Ghost systemd Unit

I went in to fix one feature and met a systemd service whose restart counter read 483,142. No binary, no run-as user, no data — just a leftover unit trying to come alive and dying every three seconds. This is the story of the ghost service I found while chasing a "connects fine but search fails" bug, and how I properly brought it back.

1. Trigger: only the search fails

I connected a knowledge base (local RAG) that embeds and searches past conversations to an AI bot. The connection itself was perfect — the tool list appeared, initialization succeeded. But actually running a search threw:

Error: Failed to connect to Ollama.
Please check that Ollama is downloaded, running and accessible.

This RAG depends on a local inference engine for embeddings (turning text into vectors). That engine wasn't responding.

2. Diagnosis: a zombie with only the unit alive

The service status kept spinning at activating. The logs named the culprit precisely.

service: Failed to determine user credentials: No such process
service: Main process exited, code=exited, status=217/USER
service: Scheduled restart job, restart counter is at 483142.

status=217/USER means "the run-as user specified in the unit does not exist." Checking one by one:

  • run-as user → missing (no such user)
  • the binary → missing (no file at the path)
  • model/data directories → missing

In other words, the service had lost its entire substance, leaving only the unit file. Because of Restart=always, RestartSec=3, it was repeating start → 217 failure → restart-in-3s hundreds of thousands of times — with no one noticing.

Why it got that way wasn't clear — maybe it was installed once and, during cleanup, only the binary and user were removed while the unit remained; or a migration leftover. What mattered: "this unit is not something to revive, it's something to clean up."

3. Recovery: wipe cleanly, then install properly

With no substance, you don't fix it — you install fresh. The order I used:

  1. Back up, then remove the ghost unit: stop the crash loop first. Back up the dead unit and delete it → daemon-reload.
  2. Install via the official script: create the binary, run-as user, and systemd unit the right way.
  3. Verify binding: confirm the inference engine listens on 127.0.0.1 only. It must not be open externally.
    4. ss -tlnp | grep 11434
LISTEN 127.0.0.1:11434   # local only ✅ (no external exposure)
  4. Pull the model: first confirm the exact embedding model the RAG references in its config, then pull precisely that one. (Pull the wrong model and vector dimensions won't match, breaking search.)

Even though the binding defaults to local, I pinned 127.0.0.1 explicitly via a drop-in override — so a future default change can't leak it externally.

4. Verify: does it really work, and does it not storm?

  • Embedding API: confirm the vector dimension comes out as expected (768).
  • Real search: search past conversations via the RAG and get results — both through the bot path and a direct call.
  • No recurrence: with the user now present, 217/USER won't happen. Still, watch the restart counter for 5 minutes to confirm the crash loop doesn't return.
  • Resources: idle memory footprint is small (tens of MB); the model loads only during inference and unloads after idle. Confirm numerically that it's harmless within the server's headroom.

5. Lessons

  • Restart=always can create a silent zombie. A unit left behind after its substance is gone will loop restarts forever with no alert. An abnormally large restart counter is itself an alarm.
  • "Connected" is not "working." A tool attaching doesn't mean the feature works. If a backend dependency (here, the inference engine) is dead, the connection is green while only the actual calls are red.
  • Check for substance before reviving a dead service. If user, binary, and data are all gone, it's not "recovery" but "reinstall." Diagnosis changes the direction of the fix.
  • Always verify local binding for a local inference engine. If an embedding/LLM engine opens on an external interface, that's an unauthenticated API exposed as-is.

FAQ

Q. How did you not notice the restart counter at 480k sooner?

That's exactly the point. Unless you dig through logs, a service that "fails but immediately restarts" is quiet on the dashboard. You catch these zombies only by habitually scanning systemctl --failed or restart counters.

Q. Doesn't a local inference engine eat a lot of memory?

Depends on the model. A small embedding-only model sits at tens of MB idle, a few hundred MB during inference, and unloads after idle. Keeping a large generative LLM resident is a different story — plan resources by "what you load."

Q. Why not just use an external embedding API?

For a RAG handling sensitive data like past conversations, local embeddings have the advantage that data never leaves the server. It's a privacy vs. cost/ops trade-off, and here I chose local.

This post generalizes a service-recovery task on a real, in-production personal server. Specific hosts, paths, and credentials have been deliberately omitted.