AI blog automation was quietly stopped for 4 days: A single exit(1) caused a chain of failure recovery
The posts on this blog you are currently reading are automatically written daily by an AI pipeline. However, that pipeline was completely stopped for 4 days, and I was completely unaware of it. This article is a real record from the diagnosis of the cause of the failure to its recovery, and a post-mortem on two common pitfalls: "a structure where the failure of a secondary feature kills the core functionality" and "silent failure".
1. Discovery: Failures come silently
The discovery process itself was far from ideal. Instead of a monitoring alert, I accidentally found 4 consecutive days of failures in the cron logs while performing an audit to clean up unnecessary API keys on the server. Posts that should have been automatically published every morning had not been uploaded for 4 consecutive days, based on the publication date, for 7 days straight.
This was the only error left in the logs.
❌ Vertex AI Init Failed: 403 Publisher Model
`publishers/google/models/imagen-3.0-generate-001`
is not visible to the current projectInteresting point: This is an error from an image generation model. If there were no images, it should have used stock images and published the post normally, so why did the publication itself stop?
2. The Cause Was Not Single: Three Overlapping Problems
Problem ① Silent Retirement of the Text Model (404)
The pipeline was using Vertex AI's gemini-2.0-flash-001 for text generation. While fixing a snapshot version (-001) offers the advantage of good reproducibility, the moment the cloud provider retires that snapshot, the code remains the same but starts returning 404s. Exactly that happened. The solution was simple — replace it with the current model (gemini-2.5-flash).
Problem ② The Real Culprit: Combined Initialization and exit(1)
The original code had a structure roughly like this.
# Before: Initialize text and image in a single try
try:
text_model = GenerativeModel("...") # Core functionality
image_model = ImageGenerationModel.from_pretrained("...") # Secondary functionality
except Exception as e:
logging.error(f"Init Failed: {e}")
exit(1) # ← Even if only the image fails, the whole process diesWhen the image model initialization failed with a 403, exit(1) killed the entire process, including text generation. The code already had a fallback path implemented to replace failed images with stock images (Unsplash), but the process terminated before reaching the fallback code. This is a classic coupling failure where the failure of a secondary feature (image) killed the core functionality (post publication).
# After: Separate initialization + secondary functionality is non-fatal
try:
text_model = GenerativeModel("gemini-2.5-flash")
except Exception as e:
logging.error(f"Text Init Failed: {e}")
exit(1) # Core functionality only fail-fast
try:
image_model = init_image_model()
except Exception as e:
logging.warning(f"Image Init Failed (fallback to stock): {e}")
image_model = None # Secondary functionality continues with fallbackProblem ③ The Identity of Imagen 403: Project Gating
The 403 on the image side was worth digging into further. We started by eliminating initial hypotheses, such as "it must be a billing issue".
- Billing? No — The text model (paid calls) worked normally with the same project and credentials.
- Specific model version retired? No — Even after testing all versions, including Imagen 3, 3-fast, 4, and older generation naming, all returned the same 403.
- Region issue? No — Exhaustive testing across 6 regions in America, Europe, and Asia, all identical.
- Service account permissions? No — Even when directly accessing the console with a project owner account, the Imagen card itself was not visible in Model Garden, and the error rate for the model metadata lookup API (
GetPublisherModel) was 100%.
Conclusion: All Imagen family models were non-visible (gated) at the project level in that Google Cloud project. There was no access request button, so there was no way to ungate it from the console. Suddenly, without notice.
3. Solution: Same Model, Different Door — Bypassing the API Surface
An important discovery here: Google's generative models are provided through two API surfaces. Vertex AI (GCP project and service account-based) and Gemini API (AI Studio, API key-based). If one door is blocked, the other might be open.
Indeed, when I queried the model list with a Gemini API key, imagen-4.0-fast-generate-001 was perfectly visible, and test generation succeeded immediately. I replaced the image generation part from the Vertex SDK with a single REST call.
resp = requests.post(
"https://generativelanguage.googleapis.com/v1beta/models/"
"imagen-4.0-fast-generate-001:predict",
params={"key": API_KEY},
json={
"instances": [{"prompt": prompt}],
"parameters": {"sampleCount": 1, "aspectRatio": "16:9"},
},
timeout=120,
)
img_bytes = base64.b64decode(
resp.json()["predictions"][0]["bytesBase64Encoded"])As a side effect, the image model was upgraded from Imagen 3 to Imagen 4, and the cost is approximately $0.02 per image for the fast model — about 1,500 KRW per month for 2 posts a day. The image at the top of this post is the result generated by that recovered pipeline.
4. Lessons Learned from This Incident
- Separate the initialization of secondary and core functionalities. Fail-fast only for core functionalities. Secondary functionalities should be demoted with a warning + fallback (graceful degradation). Even if fallback code exists, it's as good as non-existent if it's never reached.
- Silent failures are the worst kind of failures. If cron fails for 4 consecutive days and no one knows, then there is no monitoring. You should set up failure alerts (or "no output for N days" monitoring) instead of success alerts.
- Fixed model snapshots expire. Fixing snapshots like
-001provides reproducibility but exposes you to the provider's deprecation schedule. If there's no monitoring for deprecation notices, consider using current aliases. - Eliminate hypotheses with a matrix. If I had only guessed "it must be a billing issue," I would have been barking up the wrong tree. A single version × region exhaustive test script helped eliminate billing/deprecation/region/permission hypotheses all at once.
- There are workarounds even within a single vendor. Even for the same model, if the API surfaces (Vertex vs Gemini API) are different, the access policies might also differ. If one is blocked, test the other surface first — it's much cheaper than migration.
5. FAQ
- Q: Has the exact cause of the Vertex AI gating been identified?
A: No. The official error message was just "not visible to the current project," and there was no access request path in the console. While it might have been resolved with a support ticket, we decided there was no reason to incur that cost when the workaround was fully functional. - Q: Is the Gemini API key method less secure than service accounts?
A: The risk of key leakage is real. We mitigated this by locking the key with API restrictions (Generative Language API only), injecting it only via environment variables, and setting up billing anomaly detection. - Q: How specifically do you monitor for silent failures?
A: The cheapest method is "output-based monitoring." Instead of checking for process success, you examine the output (the date of the latest post) and alert if there's no update for N days. This catches failures regardless of how the pipeline internally fails.
6. Conclusion
The root cause of this failure was neither Google's model deprecation nor the mysterious 403. It was the fact that the code structure enforced "no post without an image," and that no one knew about this failure for four days. External dependencies will inevitably change without warning someday. Whether your pipeline gracefully degrades or silently fails entirely depends solely on the design of a single try/except block. Search for exit(1) in your automation code today — and ask yourself if it's truly a failure that should kill the entire process.
