8 Commits

Author SHA1 Message Date
k3nny b21a7d60dc feat(resolver,graph): fetch and resolve include: remote: HTTPS URLs
release / Build and publish release (push) Successful in 1m14s
Remote includes (include: remote: https://...) were previously skipped
silently in the resolver and rendered as unexpanded leaf nodes in the
graph.

Changes:
- fetcher.FetchURL: new shared unauthenticated HTTP GET helper
- resolver: resolveRemoteInclude fetches the URL, parses YAML, sets job
  origin to the URL string, recursively resolves sub-includes, and emits
  a warning on failure (lint continues on the rest of the pipeline)
- graph: recurseRemote fetches the URL, captures direct job names, and
  recurses into sub-includes so remote nodes expand like local ones

Adds testdata/includes_remote.yml fixture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 21:42:15 +02:00
k3nny d34c39927d fix(linter): downgrade needs optional:true missing-job to warning
release / Build and publish release (push) Successful in 1m12s
parseNeedJobNames is replaced by parseNeedEntries which preserves the
optional flag from each needs: entry. When a referenced job does not
exist and optional:true is set, the finding is now WARNING instead of
ERROR, matching GitLab CI runtime behavior (the dependency is silently
skipped when the job is absent from a conditional include).

Optional missing deps are also excluded from the cycle-detection graph
since there is no real dependency edge to trace.

Adds a fixture case in testdata/needs.yml to prevent regression.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 21:27:16 +02:00
k3nny a303f63a5e feat(linter): add file/line to findings; downgrade extends missing-script to warning
Every finding now carries the source file and exact line number of the job
key in its YAML file. Format: [ERROR] job "name" (file.yml:12): message.

Pipeline-level findings (workflow rules, no stages) reference p.SourceFile.
Cross-file include jobs (local, project, component) carry the include source
as their File, set via Pipeline.SetJobOrigin after each ParseBytes call in
the resolver.

Line numbers come from the yaml.Node key node (exact job-name line) in a
new document-level first pass in ParseBytes, replacing the previous
map[string]yaml.Node approach which only gave value-node lines.

Also: jobs that declare extends: but have no script after resolution now
emit WARNING instead of ERROR. The script may come from a base in a remote
include that was not fetched (no token, offline), making the error a false
positive in common project setups.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 21:24:18 +02:00
k3nny a962c996c1 feat(graph): show jobs per file in include dependency graph
Each node in 'glint graph includes' now lists the jobs defined directly
in that file. Jobs appear as rounded Mermaid nodes with a distinct
light-purple style, connected with dashed arrows (-.->). This visual
distinction separates ownership (file -.-> job) from the include
hierarchy (file --> included-file).

The root file's jobs are collected by re-parsing it without include
resolution; local and fetched project/component nodes populate their
job list in the existing recurse* methods.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 21:03:50 +02:00
k3nny e5f926b55f docs: 📝 add logo
release / Build and publish release (push) Successful in 1m47s
2026-06-11 20:40:39 +02:00
k3nny 8c3ce050f5 Merge pull request 'fix(model): handle YAML map forms that caused unmarshall errors' (#3) from fix/unmarshall_errors into main
Reviewed-on: #3
2026-06-11 20:31:30 +02:00
k3nny c4ab64391d fix(model): handle YAML map forms that caused unmarshall errors
Variables with value/description/options sub-keys, default.image in map
form, default.before_script / default.after_script as block scalars, and
rules.changes / rules.exists in {paths, compare_to} map form all caused
"yaml: cannot unmarshal !!map into string" because the struct fields were
typed too narrowly.

Changed types in model.Pipeline, model.DefaultConfig, and model.Rule to
accept any to match GitLab CI spec flexibility (13.7+ variable declarations,
15.3+ rules.changes map form, image map form in default block).

Adds testdata/script_multiline.yml covering all these patterns.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 20:25:53 +02:00
k3nny 4cc50afb5f fix(build): replace actions/checkout@v4 with plain git clone
release / Build and publish release (push) Successful in 1m3s
actions/checkout@v4 requires Node.js which is absent in golang:alpine.
Clone the repo directly with git using an oauth2 token in the URL,
removing the Node.js dependency entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-11 01:04:27 +02:00
16 changed files with 430 additions and 53 deletions
+9 -1
View File
@@ -16,7 +16,15 @@ jobs:
- name: Install tools
run: apk add --no-cache curl git jq
- uses: actions/checkout@v4
- name: Checkout
env:
TOKEN: ${{ secrets.GITHUB_TOKEN }}
SERVER_URL: ${{ github.server_url }}
REPO: ${{ github.repository }}
REF: ${{ github.ref_name }}
run: |
git clone --depth 1 --branch "$REF" \
"$(echo "$SERVER_URL" | sed "s|https://|https://oauth2:${TOKEN}@|")/${REPO}.git" .
- name: Build Linux (amd64)
env:
+19
View File
@@ -7,6 +7,25 @@ This project uses [Semantic Versioning](https://semver.org).
## [Unreleased]
### Added
- **File and line numbers on findings** — every finding now includes the source file and line where the job is defined, e.g. `[ERROR] job "deploy" (src/deploy.yml:14): …`. For jobs that come from local or fetched includes the file reflects the include source. Pipeline-level findings (workflow rules, missing stages) reference the root pipeline file.
- **`glint graph includes` shows jobs per file** — each node in the Mermaid include dependency graph now shows the jobs defined directly in that file. Jobs are rendered as rounded nodes (`(name)`) in a distinct light-purple style, connected with dashed arrows (`-.->`) to distinguish ownership from the include hierarchy (solid `-->` arrows). The root pipeline file always shows its direct jobs; local and fetched project/component nodes show theirs when the file can be read.
### Fixed
- **`include: remote:` URL includes are now fetched and merged** — glint fetches plain HTTPS URLs in `include: remote:` entries (no authentication), parses the resulting YAML, merges its jobs into the pipeline, and recursively resolves any sub-includes the remote file itself declares. Unreachable or unparseable URLs emit a `[WARNING]` and lint continues on the rest of the pipeline. The `glint graph includes` command now expands remote nodes with their jobs and sub-include tree, matching the behaviour of local and project includes.
- **`needs: optional: true` downgraded to warning** — a `needs:` entry that carries `optional: true` and references a job not present in the pipeline now emits `[WARNING]` instead of `[ERROR]`. GitLab CI silently skips such dependencies at runtime (the job is absent when its include was not triggered), so the finding was a false positive. Non-optional missing needs remain errors. Optional missing deps are also excluded from the cycle-detection graph.
- **`extends:` jobs with missing script downgraded to warning** — a job that declares `extends:` but has no `script` after resolution now emits `[WARNING]` instead of `[ERROR]`. The script may legitimately come from a base job in a remote include that could not be fetched at lint time (e.g. no token configured).
- **Variable map form now parses correctly** — `variables:` entries that use the extended `{value, description, options}` form (GitLab CI 13.7+) no longer cause `yaml: cannot unmarshal !!map into string`. Both `Pipeline.Variables` and per-job `Variables` now accept either plain strings or map-form declarations.
- **`default.image` map form now parses correctly** — `default: image: {name: ..., pull_policy: ...}` used to cause `yaml: cannot unmarshal !!map into string`; `DefaultConfig.Image` is now typed as `any` to match `Job.Image`.
- **`default.before_script` / `default.after_script` now accept both list and scalar forms** — previously `DefaultConfig.BeforeScript` and `DefaultConfig.AfterScript` were `[]string`, causing a parse error when the field was written as a block scalar string. They are now typed as `any` to match the corresponding `Job` fields.
- **`rules.changes` / `rules.exists` map form now parses correctly** — extended `changes: {paths: [...], compare_to: "..."}` syntax (GitLab CI 15.3+) used to cause `yaml: cannot unmarshal !!map into []string`.
## [0.2.0] - 2026-06-11
### Added
+10 -3
View File
@@ -1,7 +1,13 @@
# glint
<p align="center">
<img src="assets/glint-logo.png" alt="glint logo" width="220" />
</p>
[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
[![Release](https://img.shields.io/badge/release-v0.2.0-blue.svg)](CHANGELOG.md)
<h1 align="center">glint</h1>
<p align="center">
<a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache%202.0-blue.svg" alt="License"></a>
<a href="CHANGELOG.md"><img src="https://img.shields.io/badge/release-v0.2.0-blue.svg" alt="Release"></a>
</p>
> **Disclaimer:** This tool was built through iterative AI-assisted development with [Claude](https://claude.ai). It is experimental, incomplete, and not intended for production use. Coverage of GitLab CI keywords is best-effort and may lag behind GitLab's evolving spec. Use it at your own discretion — no correctness guarantees are made. Contributions and bug reports are welcome.
@@ -19,6 +25,7 @@ A local tool to validate and lint `.gitlab-ci.yml` pipelines without needing a G
- **CI/CD catalog components** — resolves `include: component:` references from the GitLab CI/CD Catalog; public components work without a token
- **Deprecation warnings** — flags `only`/`except` usage in favour of `rules`
- **Local include resolution** — `include: local:` entries are read from disk and recursively merged before linting, so multi-file pipelines are fully validated
- **Extended variable declarations** — `variables:` entries may use the `{value, description, options}` map form (GitLab CI 13.7+); `default.image` accepts both string and map form; `rules.changes`/`rules.exists` accept both list and `{paths, compare_to}` map form
- **Graph output** — `glint graph` prints a job tree (stages → jobs) to the terminal; `glint graph includes` emits a Mermaid include dependency diagram; `glint graph pipeline` renders a GitLab CI-style PNG/SVG
- **Context simulation** — pass `--branch`, `--tag`, or `--source` to `glint check` or `glint graph` to see which jobs would be active, manual, or skipped for a specific pipeline event; evaluates `rules:if:` expressions and `only`/`except` filters
+4
View File
@@ -41,10 +41,14 @@ tasks:
ignore_error: true
- cmd: ./{{.BINARY}} check testdata/keywords_invalid.yml
ignore_error: true
- cmd: ./{{.BINARY}} check testdata/includes_remote.yml
ignore_error: false
- cmd: ./{{.BINARY}} check testdata/includes_project.yml
ignore_error: false
- cmd: ./{{.BINARY}} check testdata/includes_component.yml
ignore_error: false
- cmd: ./{{.BINARY}} check testdata/script_multiline.yml
ignore_error: false
- cmd: ./{{.BINARY}} check testdata/context_rules.yml
ignore_error: false
- cmd: ./{{.BINARY}} check --branch main testdata/context_rules.yml
Binary file not shown.

After

Width:  |  Height:  |  Size: 106 KiB

+18
View File
@@ -142,6 +142,24 @@ func (cfg GitLabConfig) FetchFile(project, filePath, ref string) ([]byte, error)
return body, nil
}
// FetchURL downloads the content at a plain HTTPS URL without authentication.
// Used for include: remote: entries which are public by definition.
func FetchURL(rawURL string) ([]byte, error) {
resp, err := http.Get(rawURL) //nolint:noctx
if err != nil {
return nil, fmt.Errorf("GET %s: %w", rawURL, err)
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("reading body: %w", err)
}
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("GET %s: status %d", rawURL, resp.StatusCode)
}
return body, nil
}
func firstNonEmpty(values ...string) string {
for _, v := range values {
if v != "" {
+81 -5
View File
@@ -4,6 +4,7 @@ import (
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"git.k3nny.fr/glint/internal/fetcher"
@@ -11,6 +12,8 @@ import (
)
// Includes returns a Mermaid flowchart of the full include dependency tree.
// Each node shows the jobs defined directly in that file, connected with
// dashed arrows (solid arrows represent the include hierarchy itself).
// It recurses into project:, component:, and local: includes to expose
// transitive dependencies. Includes that cannot be fetched are shown but not expanded.
func Includes(sourcePath string, rawIncludes []any, cfg fetcher.GitLabConfig) string {
@@ -19,7 +22,12 @@ func Includes(sourcePath string, rawIncludes []any, cfg fetcher.GitLabConfig) st
cfg: cfg,
baseDir: filepath.Dir(sourcePath),
}
root := &treeNode{id: "root", label: mermaidLabel(sourcePath), class: "main"}
root := &treeNode{
id: "root",
label: mermaidLabel(sourcePath),
class: "main",
jobs: directJobs(sourcePath),
}
b.buildChildren(root, rawIncludes)
return renderTree(root)
}
@@ -29,6 +37,7 @@ type treeNode struct {
label string
class string
children []*treeNode
jobs []string // job names defined directly in this file
}
// treeBuilder accumulates state while recursively traversing include entries.
@@ -105,7 +114,9 @@ func (b *treeBuilder) parseMap(m map[string]any) []*treeNode {
return []*treeNode{node}
}
if remote, ok := m["remote"].(string); ok {
return []*treeNode{{id: b.nextID(), label: "remote:<br>" + mermaidLabel(remote), class: "remote"}}
node := &treeNode{id: b.nextID(), label: "remote:<br>" + mermaidLabel(remote), class: "remote"}
b.recurseRemote(node, remote)
return []*treeNode{node}
}
if tmpl, ok := m["template"].(string); ok {
return []*treeNode{{id: b.nextID(), label: "template:<br>" + mermaidLabel(tmpl), class: "template"}}
@@ -126,7 +137,11 @@ func (b *treeBuilder) recurseLocal(node *treeNode, path string) {
return
}
p, err := model.ParseBytes(data)
if err != nil || len(p.Include) == 0 {
if err != nil {
return
}
node.jobs = jobNames(p)
if len(p.Include) == 0 {
return
}
orig := b.baseDir
@@ -147,7 +162,33 @@ func (b *treeBuilder) recurseProject(node *treeNode, project, filePath, ref stri
return
}
p, err := model.ParseBytes(data)
if err != nil || len(p.Include) == 0 {
if err != nil {
return
}
node.jobs = jobNames(p)
if len(p.Include) == 0 {
return
}
b.buildChildren(node, p.Include)
}
func (b *treeBuilder) recurseRemote(node *treeNode, rawURL string) {
key := "remote:" + rawURL
if b.visited[key] {
return
}
b.visited[key] = true
data, err := fetcher.FetchURL(rawURL)
if err != nil {
return
}
p, err := model.ParseBytes(data)
if err != nil {
return
}
node.jobs = jobNames(p)
if len(p.Include) == 0 {
return
}
b.buildChildren(node, p.Include)
@@ -172,7 +213,11 @@ func (b *treeBuilder) recurseComponent(node *treeNode, ref string) {
return
}
p, err := model.ParseBytes(data)
if err != nil || len(p.Include) == 0 {
if err != nil {
return
}
node.jobs = jobNames(p)
if len(p.Include) == 0 {
return
}
b.buildChildren(node, p.Include)
@@ -193,11 +238,19 @@ func renderTree(root *treeNode) string {
w(" classDef local fill:#428fdc,stroke:#1068bf,color:#fff")
w(" classDef remote fill:#868686,stroke:#686868,color:#fff")
w(" classDef template fill:#fc6d26,stroke:#e56b1f,color:#fff")
w(" classDef job fill:#f5f5ff,stroke:#7175a0,color:#333")
w("")
jobCounter := 0
var emit func(n *treeNode)
emit = func(n *treeNode) {
wf(" %s[\"%s\"]:::%s", n.id, n.label, n.class)
for _, jobName := range n.jobs {
jobCounter++
jobID := fmt.Sprintf("job%d", jobCounter)
wf(" %s(\"%s\"):::job", jobID, mermaidLabel(jobName))
wf(" %s -.-> %s", n.id, jobID)
}
for _, child := range n.children {
emit(child)
wf(" %s --> %s", n.id, child.id)
@@ -208,6 +261,29 @@ func renderTree(root *treeNode) string {
return sb.String()
}
// directJobs parses a single pipeline file (without include resolution) and
// returns the sorted list of job names defined directly in it.
func directJobs(path string) []string {
p, err := model.Parse(path)
if err != nil {
return nil
}
return jobNames(p)
}
// jobNames returns a sorted slice of all job names defined in p.
func jobNames(p *model.Pipeline) []string {
if len(p.Jobs) == 0 {
return nil
}
names := make([]string, 0, len(p.Jobs))
for name := range p.Jobs {
names = append(names, name)
}
sort.Strings(names)
return names
}
// parseComponentRef parses a CI/CD component reference of the form
// <host>/<project-path>/<component-name>@<version>.
func parseComponentRef(ref string) (host, project, component, version string, err error) {
+4
View File
@@ -24,6 +24,8 @@ func checkDependencies(p *model.Pipeline) []Finding {
findings = append(findings, Finding{
Severity: Error,
Job: name,
File: job.File,
Line: job.Line,
Message: fmt.Sprintf("'dependencies' references unknown job %q", dep),
})
continue
@@ -34,6 +36,8 @@ func checkDependencies(p *model.Pipeline) []Finding {
findings = append(findings, Finding{
Severity: Error,
Job: name,
File: job.File,
Line: job.Line,
Message: fmt.Sprintf("'dependencies' job %q must be in an earlier stage (in %q, current job is in %q)", dep, depJob.Stage, job.Stage),
})
}
+30 -2
View File
@@ -17,12 +17,25 @@ const (
type Finding struct {
Severity Severity
Job string // empty for pipeline-level findings
File string // source file where the finding originates
Line int // line number in File (0 = unknown)
Message string
}
func (f Finding) String() string {
loc := ""
if f.File != "" {
if f.Line > 0 {
loc = fmt.Sprintf(" (%s:%d)", f.File, f.Line)
} else {
loc = fmt.Sprintf(" (%s)", f.File)
}
}
if f.Job != "" {
return fmt.Sprintf("[%s] job %q: %s", f.Severity, f.Job, f.Message)
return fmt.Sprintf("[%s] job %q%s: %s", f.Severity, f.Job, loc, f.Message)
}
if loc != "" {
return fmt.Sprintf("[%s]%s: %s", f.Severity, loc, f.Message)
}
return fmt.Sprintf("[%s] %s", f.Severity, f.Message)
}
@@ -43,6 +56,7 @@ func checkStages(p *model.Pipeline) []Finding {
if len(p.Stages) == 0 {
findings = append(findings, Finding{
Severity: Warning,
File: p.SourceFile,
Message: "no stages defined; GitLab will use default stages (build, test, deploy)",
})
}
@@ -58,6 +72,7 @@ func checkWorkflow(p *model.Pipeline) []Finding {
if rule.When != "" && !validWorkflowRuleWhen[rule.When] {
findings = append(findings, Finding{
Severity: Error,
File: p.SourceFile,
Message: fmt.Sprintf("workflow.rules[%d].when has invalid value %q; valid: always, never", i, rule.When),
})
}
@@ -88,10 +103,16 @@ func checkJob(name string, job model.Job, stageSet map[string]bool) []Finding {
// After extends resolution, a job with no script/run is an error.
// Exceptions: trigger jobs, pages jobs (use pages: keyword), and template jobs.
// When the job has extends:, the script may come from a base that couldn't be
// fetched (e.g. a remote include without a token), so downgrade to warning.
hasScript := scriptNonEmpty(job.Script) || job.Run != nil
if !isTemplate && !isTrigger && job.Pages == nil && !hasScript {
sev := Error
if job.Extends != nil {
sev = Warning
}
findings = append(findings, Finding{
Severity: Error,
Severity: sev,
Job: name,
Message: "missing required field 'script' (or 'run')",
})
@@ -137,6 +158,13 @@ func checkJob(name string, job model.Job, stageSet map[string]bool) []Finding {
findings = append(findings, checkJobKeywords(name, job)...)
// Attach source location to every job-scoped finding collected above.
for i := range findings {
if findings[i].Job != "" && findings[i].File == "" {
findings[i].File = job.File
findings[i].Line = job.Line
}
}
return findings
}
+42 -19
View File
@@ -6,6 +6,12 @@ import (
"git.k3nny.fr/glint/internal/model"
)
// needEntry is a parsed element from a job's needs: list.
type needEntry struct {
job string
optional bool // true when the needs entry carries optional: true
}
func checkNeeds(p *model.Pipeline) []Finding {
var findings []Finding
@@ -15,7 +21,7 @@ func checkNeeds(p *model.Pipeline) []Finding {
stageIndex[s] = i
}
// needsGraph maps each job to the list of jobs it depends on.
// needsGraph maps each job to the jobs it depends on (existing jobs only).
// Used for cycle detection after individual checks.
needsGraph := make(map[string][]string)
@@ -24,22 +30,33 @@ func checkNeeds(p *model.Pipeline) []Finding {
continue
}
neededNames := parseNeedJobNames(job.Needs)
needsGraph[name] = neededNames
entries := parseNeedEntries(job.Needs)
jobStageIdx, jobHasStage := stageIndex[job.Stage]
for _, needed := range neededNames {
neededJob, exists := p.Jobs[needed]
for _, entry := range entries {
neededJob, exists := p.Jobs[entry.job]
if !exists {
// optional: true means GitLab CI will silently skip the
// dependency when the job is absent (e.g. from a conditional
// include). Downgrade to warning so users are informed without
// failing the lint.
sev := Error
if entry.optional {
sev = Warning
}
findings = append(findings, Finding{
Severity: Error,
Severity: sev,
Job: name,
Message: fmt.Sprintf("needs unknown job %q", needed),
File: job.File,
Line: job.Line,
Message: fmt.Sprintf("needs unknown job %q", entry.job),
})
continue
}
// Add to the cycle-detection graph only when the dep exists.
needsGraph[name] = append(needsGraph[name], entry.job)
// A job cannot need a job in a later stage.
if len(p.Stages) > 0 && jobHasStage && neededJob.Stage != "" {
neededStageIdx, neededHasStage := stageIndex[neededJob.Stage]
@@ -47,9 +64,11 @@ func checkNeeds(p *model.Pipeline) []Finding {
findings = append(findings, Finding{
Severity: Error,
Job: name,
File: job.File,
Line: job.Line,
Message: fmt.Sprintf(
"needs %q which is in a later stage (%q after %q)",
needed, neededJob.Stage, job.Stage,
entry.job, neededJob.Stage, job.Stage,
),
})
}
@@ -57,32 +76,33 @@ func checkNeeds(p *model.Pipeline) []Finding {
}
}
findings = append(findings, detectNeedsCycles(needsGraph)...)
findings = append(findings, detectNeedsCycles(needsGraph, p.Jobs)...)
return findings
}
// parseNeedJobNames extracts job names from a needs: list.
// Each element is either a plain string or a map with a "job" key.
// Cross-pipeline needs (maps with a "pipeline" key) are skipped.
func parseNeedJobNames(needs []any) []string {
var names []string
// parseNeedEntries extracts needs entries from a needs: list, preserving the
// optional flag. Each element is a plain string (job name) or a map with a
// "job" key. Cross-pipeline needs (maps with a "pipeline" key) are skipped.
func parseNeedEntries(needs []any) []needEntry {
var entries []needEntry
for _, n := range needs {
switch v := n.(type) {
case string:
names = append(names, v)
entries = append(entries, needEntry{job: v})
case map[string]any:
if _, crossPipeline := v["pipeline"]; crossPipeline {
continue
}
if job, ok := v["job"].(string); ok {
names = append(names, job)
optional, _ := v["optional"].(bool)
entries = append(entries, needEntry{job: job, optional: optional})
}
}
}
return names
return entries
}
func detectNeedsCycles(graph map[string][]string) []Finding {
func detectNeedsCycles(graph map[string][]string, jobs map[string]model.Job) []Finding {
const (
unvisited = 0
visiting = 1
@@ -101,9 +121,12 @@ func detectNeedsCycles(graph map[string][]string) []Finding {
case visiting:
if !reported[name] {
reported[name] = true
j := jobs[name]
findings = append(findings, Finding{
Severity: Error,
Job: name,
File: j.File,
Line: j.Line,
Message: fmt.Sprintf("circular dependency in needs: %v → %s", path, name),
})
}
+30 -7
View File
@@ -13,14 +13,22 @@ func Parse(path string) (*Pipeline, error) {
if err != nil {
return nil, fmt.Errorf("reading file: %w", err)
}
return ParseBytes(data)
p, err := ParseBytes(data)
if err != nil {
return nil, err
}
p.SourceFile = path
p.SetJobOrigin(path)
return p, nil
}
// ParseBytes parses YAML from an in-memory byte slice.
func ParseBytes(data []byte) (*Pipeline, error) {
// First pass: decode into a raw map to extract job keys.
var raw map[string]yaml.Node
if err := yaml.Unmarshal(data, &raw); err != nil {
// First pass: parse into a yaml.Node document to extract job keys with
// their exact source line numbers (key nodes carry the line, value nodes
// carry the body we decode into Job / map[string]any).
var doc yaml.Node
if err := yaml.Unmarshal(data, &doc); err != nil {
return nil, fmt.Errorf("parsing YAML: %w", err)
}
@@ -32,21 +40,36 @@ func ParseBytes(data []byte) (*Pipeline, error) {
p.Jobs = make(map[string]Job)
p.RawJobs = make(map[string]map[string]any)
for key, node := range raw {
if doc.Kind != yaml.DocumentNode || len(doc.Content) == 0 {
return p, nil
}
root := doc.Content[0]
if root.Kind != yaml.MappingNode {
return p, nil
}
// Walk root mapping in key/value pairs.
for i := 0; i+1 < len(root.Content); i += 2 {
keyNode := root.Content[i]
valNode := root.Content[i+1]
key := keyNode.Value
if ReservedKeys[key] {
continue
}
var rawMap map[string]any
if err := node.Decode(&rawMap); err != nil {
if err := valNode.Decode(&rawMap); err != nil {
return nil, fmt.Errorf("parsing raw job %q: %w", key, err)
}
p.RawJobs[key] = rawMap
var j Job
if err := node.Decode(&j); err != nil {
if err := valNode.Decode(&j); err != nil {
return nil, fmt.Errorf("parsing job %q: %w", key, err)
}
j.Name = key
j.Line = keyNode.Line // exact line of the job name key
p.Jobs[key] = j
}
+29 -15
View File
@@ -3,20 +3,32 @@ package model
// Pipeline represents the top-level structure of a .gitlab-ci.yml file.
// Unknown top-level keys are collected into Jobs.
type Pipeline struct {
Stages []string `yaml:"stages"`
Variables map[string]string `yaml:"variables"`
Default *DefaultConfig `yaml:"default"`
Include []any `yaml:"include"`
Workflow *Workflow `yaml:"workflow"`
SourceFile string // path of the root pipeline file; set by Parse
Stages []string `yaml:"stages"`
Variables map[string]any `yaml:"variables"` // string or {value,description,options} map
Default *DefaultConfig `yaml:"default"`
Include []any `yaml:"include"`
Workflow *Workflow `yaml:"workflow"`
// Jobs holds every non-reserved top-level key (i.e. job definitions).
Jobs map[string]Job `yaml:"-"`
RawJobs map[string]map[string]any `yaml:"-"` // pre-resolution raw maps, used by the resolver
Jobs map[string]Job `yaml:"-"`
RawJobs map[string]map[string]any `yaml:"-"` // pre-resolution raw maps, used by the resolver
}
// SetJobOrigin sets the File field on all jobs that don't already have one.
// Called after ParseBytes to record which file each job came from.
func (p *Pipeline) SetJobOrigin(file string) {
for name, j := range p.Jobs {
if j.File == "" {
j.File = file
}
p.Jobs[name] = j
}
}
type DefaultConfig struct {
Image string `yaml:"image"`
BeforeScript []string `yaml:"before_script"`
AfterScript []string `yaml:"after_script"`
Image any `yaml:"image"` // string or {name,pull_policy,...} map
BeforeScript any `yaml:"before_script"` // []string or string (block scalar)
AfterScript any `yaml:"after_script"` // []string or string
Cache any `yaml:"cache"`
Artifacts any `yaml:"artifacts"`
Retry any `yaml:"retry"`
@@ -29,7 +41,9 @@ type Workflow struct {
}
type Job struct {
Name string // set by parser, not from YAML
Name string // set by parser, not from YAML
File string // source file; set by Parse / resolver
Line int // line of the job key in its source file; set by parser
Stage string `yaml:"stage"`
Script any `yaml:"script"` // []string or string (block scalar)
Run any `yaml:"run"` // alternative to script (CI steps)
@@ -37,8 +51,8 @@ type Job struct {
AfterScript any `yaml:"after_script"` // []string or string
Image any `yaml:"image"`
Services []any `yaml:"services"`
Variables map[string]string `yaml:"variables"`
Rules []Rule `yaml:"rules"`
Variables map[string]any `yaml:"variables"` // string or {value,description,options} map
Rules []Rule `yaml:"rules"`
Only any `yaml:"only"`
Except any `yaml:"except"`
Needs []any `yaml:"needs"`
@@ -68,8 +82,8 @@ type Job struct {
type Rule struct {
If string `yaml:"if"`
When string `yaml:"when"`
Changes []string `yaml:"changes"`
Exists []string `yaml:"exists"`
Changes any `yaml:"changes"` // []string or {paths,compare_to} map
Exists any `yaml:"exists"` // []string or map form
}
// ReservedKeys are top-level GitLab CI keys that are NOT job definitions.
+44 -1
View File
@@ -90,7 +90,14 @@ func resolveIncludes(p *model.Pipeline, includes []any, cfg fetcher.GitLabConfig
continue
}
// remote, template — resolved by GitLab at runtime, skip silently.
if remote, _ := entry["remote"].(string); remote != "" {
w, ew := resolveRemoteInclude(p, remote, cfg, rootDir, visited)
warnings = append(warnings, w...)
extWarnings = append(extWarnings, ew...)
continue
}
// template — resolved by GitLab at runtime, skip silently.
}
return warnings, extWarnings
}
@@ -117,6 +124,7 @@ func resolveLocalInclude(p *model.Pipeline, rawPath string, cfg fetcher.GitLabCo
if err != nil {
return []IncludeWarning{{Label: label, Err: fmt.Errorf("parsing YAML: %w", err)}}, nil
}
included.SetJobOrigin(absPath)
// Recursively resolve the included file's own includes first, merging
// everything into `included` before we merge it into the parent `p`.
@@ -132,6 +140,39 @@ func resolveLocalInclude(p *model.Pipeline, rawPath string, cfg fetcher.GitLabCo
return warnings, extWarnings
}
// resolveRemoteInclude fetches a plain HTTPS URL, parses it as CI YAML, and
// merges it into p. Sub-includes of the fetched file are resolved recursively.
func resolveRemoteInclude(p *model.Pipeline, rawURL string, cfg fetcher.GitLabConfig, rootDir string, visited map[string]bool) ([]IncludeWarning, []ExtendWarning) {
label := "remote " + rawURL
if visited[rawURL] {
return nil, nil
}
visited[rawURL] = true
data, err := fetcher.FetchURL(rawURL)
if err != nil {
return []IncludeWarning{{Label: label, Err: err}}, nil
}
included, err := model.ParseBytes(data)
if err != nil {
return []IncludeWarning{{Label: label, Err: fmt.Errorf("parsing YAML: %w", err)}}, nil
}
included.SetJobOrigin(rawURL)
var warnings []IncludeWarning
var extWarnings []ExtendWarning
if len(included.Include) > 0 {
w, ew := resolveIncludes(included, included.Include, cfg, rootDir, visited)
warnings = append(warnings, w...)
extWarnings = append(extWarnings, ew...)
}
mergeIncluded(p, included)
return warnings, extWarnings
}
// resolveProjectInclude fetches all files listed under a single project: entry
// and merges them into p.
func resolveProjectInclude(p *model.Pipeline, entry map[string]any, project string, cfg fetcher.GitLabConfig, rootDir string, visited map[string]bool) ([]IncludeWarning, []ExtendWarning) {
@@ -161,6 +202,7 @@ func resolveProjectInclude(p *model.Pipeline, entry map[string]any, project stri
warnings = append(warnings, IncludeWarning{Label: label, Err: fmt.Errorf("parsing YAML: %w", err)})
continue
}
included.SetJobOrigin(label)
if len(included.Include) > 0 {
w, ew := resolveIncludes(included, included.Include, cfg, rootDir, visited)
@@ -200,6 +242,7 @@ func resolveComponentInclude(p *model.Pipeline, ref string, cfg fetcher.GitLabCo
if err != nil {
return IncludeWarning{Label: label, Err: fmt.Errorf("parsing component YAML: %w", err)}, nil, true
}
included.SetJobOrigin(label)
var extWarnings []ExtendWarning
if len(included.Include) > 0 {
+14
View File
@@ -0,0 +1,14 @@
---
# Exercises include: remote: URL fetching and sub-include recursion.
# The remote file is a real public GitLab CI template; it may contain its own
# includes which should also be resolved. Exit code is 0 (warnings allowed).
stages:
- test
include:
- remote: https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Bash.gitlab-ci.yml
local-job:
stage: test
script:
- echo "local job alongside remote include"
+9
View File
@@ -38,3 +38,12 @@ missing-needs-job:
- echo "bad"
needs:
- nonexistent-job # ERROR: job doesn't exist
# optional: true — missing dep should be WARNING not ERROR
optional-needs-job:
stage: build
script:
- echo "I depend on something that may not exist"
needs:
- job: nonexistent-optional-job
optional: true
+87
View File
@@ -0,0 +1,87 @@
---
# Exercises multi-line script patterns and extended variable declarations.
# Ref: https://docs.gitlab.com/ci/yaml/script/#split-long-commands
# Ref: https://docs.gitlab.com/ee/ci/yaml/#variablesdescription
# All patterns here must parse cleanly (exit 0).
stages:
- build
- test
- deploy
# Pipeline-level variables: plain strings and extended {value, description} map form.
variables:
PLAIN_VAR: "hello"
DEPLOY_ENV:
value: "staging"
description: "The deployment target. Set to staging or production."
RETRIES:
value: "3"
description: "Number of retry attempts."
options:
- "1"
- "3"
- "5"
default:
# image in map form (name + pull_policy)
image:
name: alpine:latest
pull_policy: if-not-present
# before_script as a block scalar (not a list)
before_script:
- apk add --no-cache curl git
build-literal-block:
stage: build
# script items using literal block scalar (|)
script:
- |
if [[ "$DEPLOY_ENV" == "production" ]]; then
echo "Production build"
else
echo "Non-production build"
fi
- echo "Build step done"
build-folded-block:
stage: build
# script items using folded block scalar (>)
script:
- >
apt-get update -qq &&
apt-get install -y curl wget
- echo "Packages installed"
before_script:
- |
echo "Job-level before_script"
echo "Using literal block scalar"
test-job:
stage: test
script:
- echo "Running tests"
- |
set -e
go test ./...
echo "Tests passed"
# Job-level variable with extended form
variables:
TEST_FLAG:
value: "true"
description: "Enable verbose test output"
# rules.changes in map form (GitLab 15.3+)
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
paths:
- "**/*.go"
compare_to: "main"
when: on_success
- when: on_success
deploy-job:
stage: deploy
script:
- echo "Deploying to $DEPLOY_ENV"
when: manual