Monitoring and Troubleshooting Pipelines

# CHAPTER 9

Monitoring and Troubleshooting Pipelines #

1. Introduction #

2. Learning Objectives #

• Navigate the GitLab Pipeline interface to isolate failing jobs.

• Read and interpret raw Runner execution logs.

• Identify common failure modes (Syntax vs. Execution errors).

• Utilize the allow_failure keyword for non-critical jobs.

• Configure pipeline failure notifications (Email/Slack).

3. Beginner Explanation #

• A package (Your Code) leaves the warehouse on a truck (The Pipeline).

• The Red Alert: The GPS tracker turns red. The truck didn't reach the destination.

• Troubleshooting: You don't just stare at the red dot and panic. You call the driver (Read the Logs). The driver says, "The bridge is closed due to a storm." (Network Timeout Error). Or the driver says, "You loaded a package that is too heavy for this truck." (Out of Memory Error).

Troubleshooting is simply reading the receipts provided by the robotic workers to determine exactly which step in the instruction manual caused the machine to break.

4. Reading the Runner Logs #

You do not need to read all 5,000 lines of the log. Scroll to the very bottom. Look for the last red text before ERROR: Job failed.

Common Log Errors:

• yaml: line 10: mapping values are not allowed in this context: You made a syntax error in your .gitlab-ci.yml. You probably used a Tab instead of Spaces.

• command not found: npm: You are using a Docker image that doesn't have Node.js installed.

• ssh: connect to host ... port 22: Connection timed out: Your Runner cannot reach your deployment server. The server's firewall is likely blocking the connection.

5. Managing Flaky Tests (allow_failure) #

You can tell GitLab to flag the job with a yellow warning ! instead of a red X, and let the pipeline continue:

flaky_api_test:
  stage: test
  script:
    - ./run_unstable_api_tests.sh
  allow_failure: true # If this fails, warn me, but keep going!

6. Mini Project: Troubleshoot Failed Pipeline #

Step-by-Step Walkthrough:

1. 1. Open your .gitlab-ci.yml.

1. 2. Create a job that deliberately fails by typing an invalid Linux command:

1. 4. Go to the Pipelines dashboard. It will turn Red. Click the failed pipeline. Click the break_it_job bubble.

1. 5. The Analysis: Read the black terminal log. You will clearly see the line: /bin/sh: eval: line 140: makemedinner: not found.

1. 6. The Fix: Now you know exactly what is wrong. The robot doesn't know the command makemedinner. You open your local code, delete the bad command, push the fix, and watch the pipeline turn Green.

Now, whenever a pipeline fails, an automated message is blasted into the #engineering Slack channel, ensuring the entire team is instantly aware of the deployment blockage.

• Retry Mechanisms: Network blips happen. If a job fails specifically because it tried to download an NPM package and the network timed out, you shouldn't have to restart the entire pipeline manually. You can use the retry: keyword to tell the Runner to automatically try the job one more time if it fails before officially turning the pipeline red.

• Ignoring Yellow Warnings: Developers often get used to seeing yellow allow_failure warnings in their pipeline and start ignoring them because "the pipeline still passes." This is dangerous. If a security scan is allowed to fail, and nobody reads the warning, you might accidentally deploy vulnerable code. Always investigate warnings.

1. 2. In what specific scenario would a DevOps engineer utilize the allow_failure: true configuration for a CI job?

13. Next Chapter Recommendation #