From fea393998082b6670ed7c6b13beab8b32c1de932 Mon Sep 17 00:00:00 2001 From: Fu Hanxi Date: Tue, 9 Feb 2021 12:50:24 +0800 Subject: [PATCH] docs(ci): add docs about rules.yml and the auto-generate feature --- .gitlab/ci/README.md | 220 +++++++++++++++++++++++------- .gitlab/ci/dependencies/README.md | 100 ++++++++++++++ 2 files changed, 268 insertions(+), 52 deletions(-) create mode 100644 .gitlab/ci/dependencies/README.md diff --git a/.gitlab/ci/README.md b/.gitlab/ci/README.md index 9fdf98affc..f176397f3b 100644 --- a/.gitlab/ci/README.md +++ b/.gitlab/ci/README.md @@ -1,126 +1,241 @@ -# Rules for `rules.yml` +# IDF CI -- [Rules for `rules.yml`](#rules-for-rulesyml) +- [IDF CI](#idf-ci) + - [General Workflow](#general-workflow) + - [What if Expected Jobs ARE NOT Created?](#what-if-expected-jobs-are-not-created) + - [MR labels for additional jobs](#mr-labels-for-additional-jobs) + - [Supported MR Labels](#supported-mr-labels) + - [Usages](#usages) + - [How to trigger a `detached` pipeline without pushing new commits?](#how-to-trigger-a-detached-pipeline-without-pushing-new-commits) - [How to Develop With `rules.yml`?](#how-to-develop-with-rulesyml) + - [General Concepts](#general-concepts) - [How to Add a New `Job`?](#how-to-add-a-new-job) - [How to Add a New `Rules` Template?](#how-to-add-a-new-rules-template) - [How to Add a New `if` Anchor?](#how-to-add-a-new-if-anchor) - - [Naming Rules](#naming-rules) - - [Common Naming Rules](#common-naming-rules) - - [`if` Anchors Naming Rules](#if-anchors-naming-rules) - - [Common Phrases/Abbreviations](#common-phrasesabbreviations) - - [`rules` Template Naming Rules](#rules-template-naming-rules) + - [Naming Rules](#naming-rules) + - [Common Naming Rules](#common-naming-rules) + - [`if` Anchors Naming Rules](#if-anchors-naming-rules) + - [`rules` Template Naming Rules](#rules-template-naming-rules) - [Reusable Shell Script `tools/ci/utils.sh`](#reusable-shell-script-toolsciutilssh) - [Functions](#functions) - [CI Job Related](#ci-job-related) - [Shell Script Related](#shell-script-related) +## General Workflow + +1. Push to a remote branch +2. Create an MR, choose related labels (not required) +3. A `detached` pipeline will be created. +4. if you push a new commit, a new pipeline will be created automatically. + +**Details:** + +1. If an MR title starts with `WIP:` or `Draft:`, push commit will NOT trigger a merge-request pipeline +2. If a commit message starts with `test ci:`, pushing a commit will trigger a merge-request pipeline even when the MR title starts with `WIP:` or `Draft:`. +3. If a commit message starts with `WIP:` or `Draft:`, push commit will NOT trigger a pipeline + +## What if Expected Jobs ARE NOT Created? + +1. check the file patterns + + If you found a job that is not running as expected with some file changes, a git commit to improve the `pattern` will be appreciated. + +2. please add MR labels to run additional tests + +## MR labels for additional jobs + +### Supported MR Labels + +- `build` +- `build_docs` +- `component_ut[_esp32/esp32s2/...]` +- `custom_test[_esp32/esp32s2/...]` +- `docker` +- `docs` +- `example_test[_esp32/esp32s2/...]` +- `fuzzer_test` +- `host_test` +- `integration_test` +- `iperf_stress_test` +- `macos` +- `macos_test` +- `nvs_coverage` +- `unit_test[_esp32/esp32s2/...]` +- `weekend_test` +- `windows` + +There are two general labels (not recommended since these two labels will trigger a lot of jobs) + +- `target_test`: includes all target for `example_test`, `custom_test`, `component_ut`, `unit_test`, `integration_test` +- `all_test`: includes all test labels + +### Usages + +We have two ways to run additional jobs + +- Add these labels in the MR `labels` +- Add these labels in the commit message (not the first line). For example: + + ``` + ci: detect file changes to assign jobs + + test labels: example_test_esp32, custom_test_esp32 + ``` + + The additional test labels line should start with `test label(s):` and the labels should be separated by space or comma. + +### How to trigger a `detached` pipeline without pushing new commits? + +Go to MR web page -> `Pipelines` tab -> click `Run pipeline` button + ## How to Develop With `rules.yml`? +### General Concepts + +- `pattern`: Defined in an array. A GitLab job will be created if the changed files in this MR matched one of the patterns. For example: + + ```yaml + .patterns-python-files: &patterns-python-files + - "**/*.py" + ``` + +- `label`: (deprecated). Defined in an if clause, similar as the previous bot command. A GitLab job will be created if the pipeline variables contains variables in `BOT_LABEL_xxx` format. For example: + + ```yaml + .if-label-build_docs: &if-label-build_docs + if: '$BOT_LABEL_BUILD_DOCS' + ``` + +- `title`: Defined in an if clause. A GitLab job will be created if this title included in the MR labels or in the commit message title. For example: + + ```yaml + .if-title-docs: &if-title-docs + if: '$CI_MERGE_REQUEST_LABELS =~ /^(?:\w+,)*docs(?:,\w+)*$/i || $CI_COMMIT_TITLE =~ /\((?:\w+\s+)*docs(?:\s+\w+)*\)$/i' + ``` + +- `rule`: A combination of various patterns, labels, and titles. It will be used by GitLab YAML `extends` keyword to tell GitLab in what conditions will this job be created. For example: + + ```yaml + .rules:build:docs: + rules: + - <<: *if-protected + - <<: *if-label-build + - <<: *if-title-build + - <<: *if-label-build_docs + - <<: *if-title-build_docs + - <<: *if-label-docs + - <<: *if-title-docs + - <<: *if-dev-push + changes: *patterns-docs + ``` + + An example for GitLab job on how to use extends: + + ```yaml + check_docs_lang_sync: + extends: + - .pre_check_job_template + - .rules:build:docs + script: + - cd docs + - ./check_lang_folder_sync.sh + ``` + ### How to Add a New `Job`? check if there's a suitable `.rules:` template 1. if there is, put this in the job `extends`. All done, now you can close this window. (`extends` could be array or string) 2. if there isn't - 1. check [How to Add a New `Rules` Template?](#how-to-add-a-new-rules-template), create a suitable one - 2. follow step 1 + 1. check [How to Add a New `Rules` Template?](#how-to-add-a-new-rules-template), create a suitable one + 2. follow step 1 ### How to Add a New `Rules` Template? +check if this rule is related to `labels`, `patterns` + +1. if it is, please refer to [dependencies/README.md](./dependencies/README.md) and add new rules by auto-generating +2. if it isn't, please continue reading + check if there's a suitable `.if-` anchor - 1. if there is, create a rule following [`rules` Template Naming Rules](#rules-template-naming-rules).For detail information, please refer to [GitLab Documentation `rules-if`](https://docs.gitlab.com/ee/ci/yaml/README.html#rulesif). Here's an example. +1. if there is, create a rule following [`rules` Template Naming Rules](#rules-template-naming-rules).For detail information, please refer to [GitLab Documentation `rules-if`](https://docs.gitlab.com/ee/ci/yaml/README.html#rulesif). Here's an example. -```yaml -.rules:dev: -rules: - - <<: *if-trigger - - <<: *if-dev-push -``` + ```yaml + .rules:dev: + rules: + - <<: *if-trigger + - <<: *if-dev-push + ``` 2. if there isn't - 1. check [How to Add a New `if` Anchor?](#how-to-add-a-new-if-anchor), create a suitable one - 2. follow step 1 + 1. check [How to Add a New `if` Anchor?](#how-to-add-a-new-if-anchor), create a suitable one + 2. follow step 1 ### How to Add a New `if` Anchor? -Create an `if` anchor following [`if` Anchors Naming Rules](#if-anchors-naming-rules). For detail information about how to write the condition clause, please refer to [GitLab Documentation `only/except (advanced)](https://docs.gitlab.com/ee/ci/yaml/README.html#onlyexcept-advanced). Here's an example. +Create an `if` anchor following [`if` Anchors Naming Rules](#if-anchors-naming-rules). For detailed information about how to write the condition clause, please refer to [GitLab Documentation `only/except (advanced)](https://docs.gitlab.com/ee/ci/yaml/README.html#onlyexcept-advanced). Here's an example. ```yaml .if-schedule: &if-schedule: if: '$CI_PIPELINE_SOURCE == "schedule"' ``` -## Naming Rules +### Naming Rules -### Common Naming Rules +#### Common Naming Rules -if a phrase has multi words, use `_` to concat them. +if a phrase has multi words, use `_` to concatenate them. > e.g. `regular_test` -if a name have multi phrase, use `-` to concat them. +if a name has multi phrases, use `-` to concatenate them. > e.g. `regular_test-example_test` -### `if` Anchors Naming Rules +#### `if` Anchors Naming Rules - if it's a label: `.if-label-` - if it's a ref: `.if-ref-` - if it's a branch: `.if-branch-` - if it's a tag: `.if-tag-` -- if it's a operating system: `.if-os-mac` - if it's multi-type combination: `.if-ref--branch-` -#### Common Phrases/Abbreviations + **Common Phrases/Abbreviations** -- `no_label` + - `no_label` - `$BOT_TRIGGER_WITH_LABEL == null` + `$BOT_TRIGGER_WITH_LABEL == null` -- `protected` + - `protected` - `($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/)` + `($CI_COMMIT_REF_NAME == "master" || $CI_COMMIT_BRANCH =~ /^release\/v/ || $CI_COMMIT_TAG =~ /^v\d+\.\d+(\.\d+)?($|-)/)` -- `target_test` + - `target_test` - `example_test` or `custom_test` or `unit_test-all_targets` + a combination of `example_test`, `custom_test`, `unit_test`, `component_ut`, `integration_test` and all targets -### `rules` Template Naming Rules +#### `rules` Template Naming Rules -- if it's os related: `.rules:os:` - if it's tag related: `.rules:tag:-` - if it's label related: `.rules:labels:-` - - By default, all `.rules:labels` should include both `if-label-regular_test` and `if-protected-no-label` implicitly, unless they have special postfixes: - - - slim: `regular_test` not included - - only: only have the phrases listed - -- if it's target test related: `.rules:tests:-` - - By default, all `.rules:tests` should include `if-protected-no_label` implicitly, unless they have special postfixes (same as above) - -- if it needs to build at first, then do some target test: `.rules:build_tests:-` - - By default, all `.rules:build_tests` should include `if-protected-no-label`, `if-label-build`, `if-label-regular-test` implictly, unless they have special postfixes (same as above) - -- if a job supports all targets, use `-all_targets` as postfix +- if it's test related: `.rules:test:` +- if it's build related: `.rules:build:` +- if it's pattern related: `.rules:patterns:` ## Reusable Shell Script `tools/ci/utils.sh` -It is used to put all the reusable shell script as small functions. If you want to set `before_script: []` for you job, now you can set `extends: .before_script_slim` instead. it will only run `source tools/ci/utils.sh` +It is used to put all the reusable shell scripts as small functions. If you want to set `before_script: []` for you job, now you can set `extends: .before_script_slim` instead. it will only run `source tools/ci/utils.sh` -If you're developing CI shell scripts, you can use these functions without source. They're already included in all `before_script` +If you're developing CI shell scripts, you can use these functions without `source` them. They're already included in all `before_script` To run these commands in shell script locally, place `source tools/ci/utils.sh` at the very beginning. ### Functions #### CI Job Related -- `apply_bot_filter` + - `add_gitlab_ssh_keys` - `add_github_ssh_keys` - `add_doc_server_ssh_keys` @@ -128,6 +243,7 @@ To run these commands in shell script locally, place `source tools/ci/utils.sh` - `get_all_submodules` #### Shell Script Related + - `error`: log in red color - `warning`: log in orange color - `info`: log in green color diff --git a/.gitlab/ci/dependencies/README.md b/.gitlab/ci/dependencies/README.md new file mode 100644 index 0000000000..ae1c3a50e7 --- /dev/null +++ b/.gitlab/ci/dependencies/README.md @@ -0,0 +1,100 @@ +# How the `generate_rules.py` works + +## Functionalities + +This script can do only two things: + +1. Auto-generate some labels/titles/rules we need and update them in `rules.yml` +2. Generate a dependency tree graph + +## Schema + +This file only used basic YAML grammar and has nothing to do with the GitLab version YAML file. + +It has five custom keywords: + +- `matrix`: An array of sub-arrays, used to replicate rules by formatting strings. You can use the format string everywhere, it will be formatted recursively +- `labels`: An array of `labels`. Also indicates `titles` with the same names +- `patterns`: An array of `patterns`. Patterns that not included +- `included_in`: An array of other `rule` names. It indicates the `labels` and `patterns` will be included in all specified `rules` as well +- `deploy`: An array of strings, used to replicate rules by adding postfix `-`. It indicates the extra `label` used in `rules`, which will explain later. + +## How to use this file to generate `rules.yml` + +Let's take a complicated example to help understand the process + +```yaml +"test-{0}-{1}": + matrix: + - [a, b] + - [c, d] + labels: + - "{0}-{1}" + patterns: + - "{0}" + - pattern-not-exist + included_in: + - build-{0} +``` + +1. expand the mapping dicts defined by `matrix` + + After this step, it will turn into 4 dicts: + + | key | labels | patterns | included_in | + | -------- | ------ | -------- | ----------- | + | test-a-c | a-c | a | build-a | + | test-a-d | a-d | a | build-a | + | test-b-c | b-c | b | build-b | + | test-b-d | b-d | b | build-b | + + **Advanced Usage: You can overwrite a mapping by declaring it again later**, For example: + + If we concatenate this part to the previous example, + + ```yaml + # ... The same as the previous example + + test-a-c: + labels: + - overwrite + ``` + + `rule` `test-a-c` will be turned into: + + | key | labels | + | -------- | --------- | + | test-a-c | overwrite | + + **Mappings with the keyword `deploy` will also replicate by adding a postfix `-` to the mapping key** + +2. create rules by `included_in` + + After this step, it will turn into 6 mapping dicts: + + | key | labels | patterns | + | -------- | -------- | -------- | + | test-a-c | a-c | a | + | test-a-d | a-d | a | + | test-b-c | b-c | b | + | test-b-d | b-d | b | + | build-a | a-c, a-d | a | + | build-b | b-c, b-d | b | + +3. replace the auto-generated region in `rules.yml` with `labels`, `titles`, and `rules`. Each mapping will generate a `rule` and all the required labels/titles. `patterns` are pre-defined in `rules.yml` and could not be generated automatically. If a mapping is using a `pattern` undefined, the `pattern` will be ignored. + + - If a mapping key has postfix '-preview', no `if-protected-xxx` clause will be added + - else if a mapping key has postfix '-production', `if-protected-no_label` clause will be added + - else: a mapping key `if-protected` clause will be added + +## Graph + +All `label` nodes are in green, `pattern` nodes are in cyan, `rule` nodes are in blue + +### Requirements + +There are a few extra dependencies while generating the dependency tree graph, please refer to [pygraphviz](https://github.com/pygraphviz/pygraphviz/blob/master/INSTALL.txt) documentation to install both `graphviz` and `pygraphviz` + +### CLI usage + +`python generate_rules.py --graph OUTPUT_PATH`