From 04b707fb1e4dd4ef66eb0bbfb08bc24f537bfd2c Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Fri, 4 Sep 2026 11:41:34 +0200 Subject: [PATCH 1/8] expand keypoints in collaborative CI exercise Added keypoints about GitHub Actions and CI/CD setup. --- content/full-cycle-ci.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/content/full-cycle-ci.md b/content/full-cycle-ci.md index 6edf576a..d240dee7 100644 --- a/content/full-cycle-ci.md +++ b/content/full-cycle-ci.md @@ -342,6 +342,8 @@ Your goal: ```{keypoints} +- GitHub Actions (and other services, like GitLab CI/CD pipelines) can be used to run automatically the test suite when pushing +- Setting up is usually done with YAML files. Every platform has their own syntax, but there are templates one can use - When fixing bugs or other problems reported in issues, use the issue autoclosing mechanism when you send the pull/merge request. ``` From 8a144595efd8665231c1071753058726b4f596ce Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 10:59:19 +0200 Subject: [PATCH 2/8] move motivations from index, lesson descr in index Related to first comment in #245. Add lesson description in index as in the git intro https://coderefinery.github.io/git-intro/ --- content/index.rst | 43 ++++++++++++++++++++++++++++--------------- content/motivation.md | 41 +++++++++++++++++++++++++++++++++-------- 2 files changed, 61 insertions(+), 23 deletions(-) diff --git a/content/index.rst b/content/index.rst index c3f6a2fd..47370dec 100644 --- a/content/index.rst +++ b/content/index.rst @@ -1,21 +1,34 @@ Automated testing - Preventing yourself and others from breaking your functioning code ====================================================================================== -Have you ever had some of these problems?: - -- You change B and C, and suddenly A doesn't work anymore. Time - wasted trying to figure out what changed. -- There was some simple problem, systematically testing could have - found it. -- You get someone else's code and are afraid to touch it because who - knows what might break. Plot twist: it's your own code! - -People have learned that some automatic way to check problems makes -software development much easier. This lesson will talk about the -places it's useful for research code, and how easy it can be. -We will discuss why testing often needs to be part of the -software development cycle and how such a cycle can be implemented. We will -see how automated testing works and practice designing and writing tests. +In this lesson we discuss the basics of automated testing. + +We start discussing why automated testing is important. +We then show how to set up automated testing in your projects +in a few programming languages, +so that you can run a test suite conveniently on your own computer. +We will then show how to make GitHub (or GitLab) +run the test suite automatically +(typically whenever someone pushes to the repository), +and tell us when there was a problem. + +If time allows, +we might do that in a collaborative fashion, +running the test suite on a pull request +to inform the code review process. + +Writing tests can be challenging sometimes, +so we will discuss typical problems in test design, +mentioning also *Test Driven Develpment*, +and practice designing and writing tests. + +The goals of the module +is to make the learners feel comfortable +with setting up a test suite of automated tests, +feel familiar with the automation options +on Software forges (e.g. GitHub and GitLab), +and make them aware of the typical challenges +in writing automated tests. .. prereq:: diff --git a/content/motivation.md b/content/motivation.md index 2224b139..71a8bcb2 100644 --- a/content/motivation.md +++ b/content/motivation.md @@ -5,8 +5,37 @@ - Understand various benefits of testing ``` +Most scientists nowadays depend on software for research. -## Untested software can be compared to uncalibrated detectors +What can go wrong when research software has bugs? Look no further: + +- [A Scientist's Nightmare: Software Problem Leads to Five Retractions](https://science.sciencemag.org/content/314/5807/1856.summary) +- [Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/) + +How can we avoid problems like these? + +## What are typical problems that *automated* tests can address? + +Have you ever had some of these problems? + +- You change B and C, and suddenly A doesn't work anymore. Time + wasted trying to figure out what changed. +- There was some simple problem, systematically testing could have + found it. + But testing manually takes too much time, + so nobody ever did it + with the appropriate care. +- You get someone else's code and are afraid to touch it because who + knows what might break. Plot twist: it's your own code! + +People have learned that some automatic way to check problems makes +software development much easier. This lesson will talk about the +places it's useful for research code, and how easy it can be. + +Most people test their code in some way, typically manually. + + +## Untested software can be compared to uncalibrated measurement devices *"Before relying on a new experimental device, an experimental scientist always establishes its accuracy. A new detector is calibrated when the scientist @@ -17,18 +46,14 @@ calibration are compared against the expected response."* With testing, simulations and analysis using software *can* be held to the same standards as experimental measurement devices! -What can go wrong when research software has bugs? Look no further: - -- [A Scientist's Nightmare: Software Problem Leads to Five Retractions](https://science.sciencemag.org/content/314/5807/1856.summary) -- [Researchers find bug in Python script may have affected hundreds of studies](https://arstechnica.com/information-technology/2019/10/chemists-discover-cross-platform-python-scripts-not-so-cross-platform/) - --- ## Testing in a nutshell -In software tests, expected results are compared with observed results +In the most basic form of software tests, +expected results are compared with observed results in order to establish accuracy. Why are we not comparing directly all -digits with the expected result?: +digits with the expected result? ````{tabs} ```{group-tab} Python From 41d18704e9b206daed076a5959545212a6d69eb0 Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 12:09:38 +0200 Subject: [PATCH 3/8] problems/solutions/for whom in motivations also modified glossary --- content/motivation.md | 40 +++++++++++++++++++++++++++++++++------- 1 file changed, 33 insertions(+), 7 deletions(-) diff --git a/content/motivation.md b/content/motivation.md index 71a8bcb2..6b693523 100644 --- a/content/motivation.md +++ b/content/motivation.md @@ -104,12 +104,34 @@ CORRECT ## What can tests help you do? -**Preserving expected functionality** -- Check old things when you add new ones +```{list-table} Problems, Solutions and who is affected? +:widths: 40 30 30 +* - Problem + - Solution + - Who is affected? +* - Breaking old functionality + when adding new features + - End-to-End tests + - Developers +* - Verify installation + - Smoke tests + - Users +* - Showing up-to-date example + - End-to-End tests + - Users +* - Improve readability and names + - Unit tests + - Developers +* - Refactor and restructure + - All tests + - Developers +* - Documentation out of date + - Executable notebooks + and [nbval](https://github.com/computationalmodelling/nbval), + End-to-End tests + - Users -**Help users of your code** -- Verify it's installed correctly and works. -- See examples of what it should do. +``` **Help other developers modify it** - Change things with confidence that nothing is breaking. @@ -152,14 +174,18 @@ Use the collaborative notes to answer these questions: ## Testing vocabulary -* Test functions one at a time - **Unit tests** +* Test functions and methods one at a time - **Unit tests** * Test how parts work together - **Integration tests** -* Test the whole thing running - **End-to-end tests** +* Test the whole thing running, checking the output - **End-to-end tests** * For example, running on sample data. +* Test that the whole thing runs in the simplest scenario possible - **Smoke Test** + * if this fails, no point in testing other things, usually. + * Check results are the same as before - **Regression tests** + * Other names for the same thing: **Acceptance Tests**, **Golden-Master Tests**, **Characterization Tests** * Write test first (the output), then write code to make test pass - **Test-driven development** From bfeefb072afd091f74e00ce818317bf992d734af Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 12:56:55 +0200 Subject: [PATCH 4/8] review episode titles, what is CI stress automated vs non automated testing also, add definition of CI and common meaning of CI --- content/continuous-integration.md | 48 ++++++++++++++++++++----------- content/index.rst | 2 +- content/locally.md | 5 ++-- 3 files changed, 36 insertions(+), 19 deletions(-) diff --git a/content/continuous-integration.md b/content/continuous-integration.md index c59b52c1..d5217208 100644 --- a/content/continuous-integration.md +++ b/content/continuous-integration.md @@ -1,15 +1,31 @@ -# Automated testing +# Automated testing and "Continuous Integration" ```{questions} - How can we implement automatic testing each time we push changes to the repository? - Why is it good to autoclose issues with commit messages? ``` -## Continuous integration We will now learn to set up automatic tests using either GitHub Actions or GitLab CI - you can choose which one to use and instructions are provided for both. + +```{note} + +## What is Continuous integration? + +From [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration): + +> *Continuous integration (CI) is the practice +> of integrating source code changes frequently +> and ensuring that the integrated codebase is in a workable state.* + +By extension +the expression **Continous Integration** is also commonly used +to mean the automation *tools* that facilitate the practice. + +```` + This exercise can be run in "collaborative mode" by following instead the instructions in [Full-cycle collaborative workflow](./full-cycle-ci). In the collaborative version steps C-D below are performed by a collaborator. @@ -27,7 +43,7 @@ In this exercise, we will: - **F.** Create a test to increase the code coverage of our tests. ``` -### Prerequisites +## Prerequisites If you are new to Git, you can find a step-by-step guide to setting up repositories and making commits in @@ -36,9 +52,9 @@ If you are new to pull requests / merge requests, you can learn all about them in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborative/). -### Step 1: Create a new repository on GitHub/GitLab OR fork from the example repo +## Step 1: Create a new repository on GitHub/GitLab OR fork from the example repo -#### Create a new repository +### Create a new repository - Begin by creating a repository called (for example) *example-ci*. - **Before** you create the repository, select **"Initialize this repository @@ -191,14 +207,14 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati -#### Fork and clone an existing example repository +### Fork and clone an existing example repository - Fork the example repo. There are two options one for [Python](https://github.com/AaltoRSE/PyTestingExample) and one for [R](https://github.com/AaltoRSE/RTestingExample). - Clone your fork (`git clone git@github.com:/TestingExample.git`). -### Step 2: Run tests locally +## Step 2: Run tests locally `````{tabs} ````{group-tab} Python You can now run your tests locally with @@ -216,7 +232,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati -### Step 3: Enable automated testing +## Step 3: Enable automated testing `````{tabs} ````{group-tab} GitHub-Python @@ -439,7 +455,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati ````` -### Step 4: Verify that tests have been automatically run +## Step 4: Verify that tests have been automatically run `````{tabs} ````{group-tab} GitHub-Python @@ -488,7 +504,7 @@ in the [Collaborative Git lesson](https://coderefinery.github.io/git-collaborati ````` -### Step 5: Add a test which reveals a problem +## Step 5: Add a test which reveals a problem After you committed the workflow file, your GitHub/GitLab repository will be ahead of your local cloned repository. Update your local cloned repository: @@ -505,14 +521,14 @@ Verify that the test suite now fails on the "Actions" tab (GitHub) or the "CI/CD->Pipelines" tab (GitLab). -### Step 6: Open an issue on GitHub/GitLab +## Step 6: Open an issue on GitHub/GitLab Open a new issue in your repository about the broken test (click the "Issues" button on GitHub or GitLab and write a title for the issue). The plan is that we will fix the issue through a pull/merge request. -### Step 7: Fix the broken test +## Step 7: Fix the broken test Now fix the code **on a new branch**, you can call it `yourname/bugfix`. After you have fixed the code on the new branch, commit the following @@ -531,7 +547,7 @@ you try to fix issue number 1). Then push to your repository. -### Step 8: Open a pull request (GitHub)/ merge request (GitLab) +## Step 8: Open a pull request (GitHub)/ merge request (GitLab) Go back to the repository on GitHub or GitLab and open a pull/merge request. **In a collaborative setting, you could request a code @@ -544,7 +560,7 @@ can still add it to the pull/merge request: `my pull/merge request title, closes #1`. -### Step 9: Accept the pull/merge request +## Step 9: Accept the pull/merge request Observe how accepting the pull/merge request automatically closes the issue (provided the commit message or the pull/merge request contained the correct issue number). @@ -555,14 +571,14 @@ See also: Discuss whether this is a useful feature. And if it is, why do you think is it useful? -### Step 10: Increase your code coverage +## Step 10: Increase your code coverage We are currently missing several functions in our tests. Write a test for the `multiply` function in a new branch and create a pull request. On Python you can directly observe the increase in code coverage. On R you can have a look at the action (`Actions -> last run of your action -> Select a job -> Test coverage`). If you compare this with the previous run, you should see an increase once the update is in. -### Step 11 (optional): Repeat steps 5-9 for the `convert_fahrenheit_to_celsius` function: +## Step 11 (optional): Repeat steps 5-9 for the `convert_fahrenheit_to_celsius` function: Repetition helps learning, so let's do the testing again for our `convert_fahrenheit_to_celsius` function. Uncomment the test for the `convert_fahrenheit_to_celsius` function and repeat steps 5 to 9 fixing the bug this test exposes. diff --git a/content/index.rst b/content/index.rst index 47370dec..fa6fd463 100644 --- a/content/index.rst +++ b/content/index.rst @@ -1,7 +1,7 @@ Automated testing - Preventing yourself and others from breaking your functioning code ====================================================================================== -In this lesson we discuss the basics of automated testing. +In this lesson we discuss the basics of *automated* testing. We start discussing why automated testing is important. We then show how to set up automated testing in your projects diff --git a/content/locally.md b/content/locally.md index 4aae85bc..400bb9a4 100644 --- a/content/locally.md +++ b/content/locally.md @@ -1,4 +1,4 @@ -# Testing locally +# Automated Testing on your computer ```{questions} - How hard is it to set up a test suite for a first unit test? @@ -9,7 +9,8 @@ In this exercise we will make a simple function and use -one of the language specific test frameworks to test it. +one of the language specific test frameworks +to test it automatically. * This is easy to use by almost any project and doesn't rely on any other servers or services. From d0ab4feb35f8e79e414927e1e2168d0ece72d623 Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Wed, 9 Sep 2026 13:37:04 +0200 Subject: [PATCH 5/8] rephrasing problems to solve with autom. testing --- content/motivation.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/content/motivation.md b/content/motivation.md index 6b693523..7468fe5b 100644 --- a/content/motivation.md +++ b/content/motivation.md @@ -23,18 +23,21 @@ Have you ever had some of these problems? - There was some simple problem, systematically testing could have found it. But testing manually takes too much time, - so nobody ever did it + so nobody has ever done it with the appropriate care. -- You get someone else's code and are afraid to touch it because who - knows what might break. Plot twist: it's your own code! +- You get someone else's code. + You really need to change it + but are afraid to touch it because who + knows what might break. + Plot twist: it's your own code! +- You implement features to someone else's code and want to merge it, + but they are not sure your changes haven't broken anything + and it's time consuming to test that. People have learned that some automatic way to check problems makes software development much easier. This lesson will talk about the places it's useful for research code, and how easy it can be. -Most people test their code in some way, typically manually. - - ## Untested software can be compared to uncalibrated measurement devices *"Before relying on a new experimental device, an experimental scientist always From e32339377bddcdde6a00e94734dccff1c3f3dec2 Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Thu, 10 Sep 2026 13:28:44 +0200 Subject: [PATCH 6/8] "continuous integration" -> "cloud" continuous integration is a "difficult" expression, meaning originally the practice of merging into main/master/trunk often, then slightly abused because of GitLab CI. Co-Authored-by: Anja Virkkunen --- .../{continuous-integration.md => cloud.md} | 18 ++---------------- content/full-cycle-ci.md | 4 ++-- content/index.rst | 4 ++-- 3 files changed, 6 insertions(+), 20 deletions(-) rename content/{continuous-integration.md => cloud.md} (97%) diff --git a/content/continuous-integration.md b/content/cloud.md similarity index 97% rename from content/continuous-integration.md rename to content/cloud.md index d5217208..9af27db9 100644 --- a/content/continuous-integration.md +++ b/content/cloud.md @@ -1,4 +1,4 @@ -# Automated testing and "Continuous Integration" +# Automated testing in the Cloud ```{questions} - How can we implement automatic testing each time we push changes to the repository? @@ -12,26 +12,12 @@ GitLab CI - you can choose which one to use and instructions are provided for bo ```{note} -## What is Continuous integration? - -From [Wikipedia](https://en.wikipedia.org/wiki/Continuous_integration): - -> *Continuous integration (CI) is the practice -> of integrating source code changes frequently -> and ensuring that the integrated codebase is in a workable state.* - -By extension -the expression **Continous Integration** is also commonly used -to mean the automation *tools* that facilitate the practice. - -```` - This exercise can be run in "collaborative mode" by following instead the instructions in [Full-cycle collaborative workflow](./full-cycle-ci). In the collaborative version steps C-D below are performed by a collaborator. -```{challenge} Exercise CI-1: Create and use a continuous integration workflow on GitHub or GitLab +```{challenge} Exercise CI-1: Create and use a workflow on GitHub or a pipeline on GitLab In this exercise, we will: diff --git a/content/full-cycle-ci.md b/content/full-cycle-ci.md index d240dee7..3c5a3c7d 100644 --- a/content/full-cycle-ci.md +++ b/content/full-cycle-ci.md @@ -8,10 +8,10 @@ ## Exercise a full-cycle collaborative workflow -This exercise is a collaborative version of the [Automated testing exercise](./continuous-integration). +This exercise is a collaborative version of the [Automated testing exercise](./cloud). ```{exercise} FullCI-1: Create and use a continuous integration workflow on GitHub or GitLab with pull requests and issues -This is an expanded version of the {doc}`automated testing demonstration `. +This is an expanded version of the {doc}`automated testing demonstration on the cloud `. The exercise is performed in a collaborative circle within the exercise group (breakout room). diff --git a/content/index.rst b/content/index.rst index fa6fd463..8882c8a9 100644 --- a/content/index.rst +++ b/content/index.rst @@ -56,7 +56,7 @@ in writing automated tests. 15 min ; :doc:`motivation` 25 min ; :doc:`locally` - 30 min ; :doc:`continuous-integration` + 30 min ; :doc:`cloud` 30 min ; :doc:`test-design` 5 min ; :doc:`conclusions` @@ -67,7 +67,7 @@ in writing automated tests. motivation locally - continuous-integration + cloud test-design conclusions full-cycle-ci From f7c23e7a647717966bb915d88e11682f75280bbf Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Fri, 11 Sep 2026 11:25:40 +0200 Subject: [PATCH 7/8] Fixes suggested by Anja Co-Authored-by: Anja Virkkunen Co-authored-by: Michele Mesiti --- content/cloud.md | 3 ++- content/full-cycle-ci.md | 4 ++-- content/index.rst | 4 ++-- content/locally.md | 2 +- 4 files changed, 7 insertions(+), 6 deletions(-) diff --git a/content/cloud.md b/content/cloud.md index 9af27db9..80d2b802 100644 --- a/content/cloud.md +++ b/content/cloud.md @@ -1,4 +1,4 @@ -# Automated testing in the Cloud +# Automated testing in the cloud ```{questions} - How can we implement automatic testing each time we push changes to the repository? @@ -15,6 +15,7 @@ GitLab CI - you can choose which one to use and instructions are provided for bo This exercise can be run in "collaborative mode" by following instead the instructions in [Full-cycle collaborative workflow](./full-cycle-ci). In the collaborative version steps C-D below are performed by a collaborator. +``` ```{challenge} Exercise CI-1: Create and use a workflow on GitHub or a pipeline on GitLab diff --git a/content/full-cycle-ci.md b/content/full-cycle-ci.md index 3c5a3c7d..df87007d 100644 --- a/content/full-cycle-ci.md +++ b/content/full-cycle-ci.md @@ -342,8 +342,8 @@ Your goal: ```{keypoints} -- GitHub Actions (and other services, like GitLab CI/CD pipelines) can be used to run automatically the test suite when pushing -- Setting up is usually done with YAML files. Every platform has their own syntax, but there are templates one can use +- GitHub Actions (and other services, like GitLab CI/CD pipelines) can be used to run automatically the test suite when pushing. +- Setting up is usually done with YAML files. Every platform has their own syntax, but there are templates one can use. - When fixing bugs or other problems reported in issues, use the issue autoclosing mechanism when you send the pull/merge request. ``` diff --git a/content/index.rst b/content/index.rst index 8882c8a9..bc2225a0 100644 --- a/content/index.rst +++ b/content/index.rst @@ -7,7 +7,7 @@ We start discussing why automated testing is important. We then show how to set up automated testing in your projects in a few programming languages, so that you can run a test suite conveniently on your own computer. -We will then show how to make GitHub (or GitLab) +We will continue by showing how to have GitHub (or GitLab) run the test suite automatically (typically whenever someone pushes to the repository), and tell us when there was a problem. @@ -19,7 +19,7 @@ to inform the code review process. Writing tests can be challenging sometimes, so we will discuss typical problems in test design, -mentioning also *Test Driven Develpment*, +mentioning also *Test Driven Development*, and practice designing and writing tests. The goals of the module diff --git a/content/locally.md b/content/locally.md index 400bb9a4..0b598d50 100644 --- a/content/locally.md +++ b/content/locally.md @@ -1,4 +1,4 @@ -# Automated Testing on your computer +# Automated testing on your computer ```{questions} - How hard is it to set up a test suite for a first unit test? From b4b16422454e5cd91d5c8d225913996e6c0ac0b5 Mon Sep 17 00:00:00 2001 From: Michele Mesiti Date: Fri, 11 Sep 2026 12:05:37 +0200 Subject: [PATCH 8/8] cloud -> remotely for additional clarity first we removed "Continous integration" because of the ambiguity but then we replaced it with "cloud", which is also not very clear. "remote/remotely" is probably the cleanest approach here. --- content/full-cycle-ci.md | 4 ++-- content/index.rst | 4 ++-- content/{cloud.md => remotely.md} | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) rename content/{cloud.md => remotely.md} (99%) diff --git a/content/full-cycle-ci.md b/content/full-cycle-ci.md index df87007d..874e33cb 100644 --- a/content/full-cycle-ci.md +++ b/content/full-cycle-ci.md @@ -8,10 +8,10 @@ ## Exercise a full-cycle collaborative workflow -This exercise is a collaborative version of the [Automated testing exercise](./cloud). +This exercise is a collaborative version of the [Automated testing exercise](./remotely). ```{exercise} FullCI-1: Create and use a continuous integration workflow on GitHub or GitLab with pull requests and issues -This is an expanded version of the {doc}`automated testing demonstration on the cloud `. +This is an expanded version of the {doc}`automated testing demonstration on the cloud `. The exercise is performed in a collaborative circle within the exercise group (breakout room). diff --git a/content/index.rst b/content/index.rst index bc2225a0..3fabe586 100644 --- a/content/index.rst +++ b/content/index.rst @@ -56,7 +56,7 @@ in writing automated tests. 15 min ; :doc:`motivation` 25 min ; :doc:`locally` - 30 min ; :doc:`cloud` + 30 min ; :doc:`remotely` 30 min ; :doc:`test-design` 5 min ; :doc:`conclusions` @@ -67,7 +67,7 @@ in writing automated tests. motivation locally - cloud + remotely test-design conclusions full-cycle-ci diff --git a/content/cloud.md b/content/remotely.md similarity index 99% rename from content/cloud.md rename to content/remotely.md index 80d2b802..f75abf4c 100644 --- a/content/cloud.md +++ b/content/remotely.md @@ -1,4 +1,4 @@ -# Automated testing in the cloud +# Automated testing remotely ```{questions} - How can we implement automatic testing each time we push changes to the repository?