Aut2026review batch2 by mmesiti · Pull Request #252 · coderefinery/testing

mmesiti · 2026-09-17T14:03:35Z

This is in addition to #246, with another batch of changes to address #245

Pining @Vinye for awareness. Later I will ping maintainers for approval.

Some connection to glossary from the "motivations" episode

removed paragraph whose topic was already covered in the table

Addressing comment in coderefinery#246 (review)

Vinye · 2026-09-18T11:44:37Z

+  if you need more control on the running environment.

 ```{keypoints}
 - When fixing bugs or other problems reported in issues, use the issue


This keypoint is a little lonely 💔

Thinking about it, I have trouble seeing the significance of issue auto-closing when compared to the rest of the exercise. Is auto-closing really the most important thing learner should remember from this exercise?

I see two options:

We could add something along the lines of "setting up testing workflows on Github and Gitlab is straightforward with the templates the services provide". Couldn't think right now a better way to summarize the exercise.

Drop the keypoints completely on this page.

I do agree:

in general the "metadata department" (keypoints and objectives) of the whole lesson is a little underdeveloped, and I think we should try to get to parity with other lessons.

in paticular, autoclosing issues is cool because it links nicely with PRs as discussed in week 1, but I do agree this cannot be the only thing here

Opened #254 to this aim

Vinye · 2026-09-18T11:47:27Z

+
 - GitHub Actions has a [Marketplace](https://github.com/marketplace?type=actions) which offer wide range of automatic workflows
 - On GitLab use [GitLab CI](https://about.gitlab.com/product/continuous-integration/)
 - For Windows builds you can also use [Appveyor](https://www.appveyor.com)


Github has runners for all the three major OSes, so I think Appveyor is not the only option. But I don't know about GitLab, I would assume they also have runners. Could be worth mentioning here.

I do not have experience with appveyor and to my modes experience the appveyor bit of information is too specific and partial, so I think it does not fit here.
Also both github and gitlab have windows runners.

So removed appveyor and expanded on github.com and gitlab.com hosted runners in 4fd8d0d

is my approach here too drastic?

Vinye · 2026-09-18T11:50:09Z

+- Note that this works also for self-hosted GitLab servers or GitHub Enterprise 
+  (i.e., not gitlab.com or github.com).
+  Your data does not have to go to a cloud!
+- It is also possible to run these workflows on a host


You got me curious here, how? Do you have a link to point learners to?

If you refer to

It is also possible to run these workflows on a host

I'm referring to the GitHub self hosted runners and GitLab runner

I will add these links to the text.

see 4fd8d0d

Vinye · 2026-09-18T12:05:32Z

+  This should match (or serve as) an **example in the code documentation** anyway.
+  - Describe in words how *you* check whether the code still works.
+  - Translate the words into a script.
+  - Run the script as often as reasonable 


Inconsistent punctuation throughout all the itemized lists on this page, maybe end all in period?

Fair point. Will work on that

should be done now, consider resolving

Vinye · 2026-09-18T12:07:37Z

 ```

+Most scientists nowadays depend on software for research.



Maybe this newline can be removed, so the sentences render as one paragraph.

Removed newline, but added double space at end so the sentence stays on a single line.
Done in 17a80d2

Vinye · 2026-09-18T12:13:52Z

I think this renders poorly (https://mmesiti.github.io/cr-testing/branch/aut2026review-batch2/motivation/#untested-software-can-be-compared-to-uncalibrated-measurement-devices)

I found epigraph/pull-quote in the sphinx documentation, could we use either of these instead?

I need to try this, I'll see what I can do (I don't like this either, but I haven't found anything better, perhaps epigraph is what I was looking for all the time?)

Done in 5cfed98

Vinye · 2026-09-18T12:21:49Z

 in order to establish accuracy.  Why are we not comparing directly all
-digits with the expected result?:
+digits with the expected result?


This talk about numeric accuracy should be moved to locally.md as we discussed before.

Please see general restructuring of that part in 598f314

Vinye · 2026-09-18T12:23:18Z

This sentence connects poorly to the previous sentences, at least I have hard time parsing what is meant here.

Please see general restructuring of that part in 598f314

Vinye · 2026-09-18T12:24:07Z

+
+```
+Very few people are proud of the code they write 
+the first time they write it.


Heh, this reminds me of writing in general, "it's a process" 😁

We might point this out, that this might sound familiar to anyone making anything original (this applies to art too)

Vinye · 2026-09-18T12:26:51Z

@@ -177,13 +209,6 @@ Use the collaborative notes to answer these questions:

 ## Where to start


This sections is now an awkward stub when most of the content was removed, maybe remove the whole section?

Ehm, I think something went wrong here. I'll have a look.

I think that the content of that section are now all covered in the conclusion, so it can be removed.
Fixed in e68959f

Vinye

Made a second pass and added some comments 👀

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi> Co-authored-by: Michele Mesiti <mmesiti@users.noreply.github.com>

it belongs to conclusions, where the topic is already covered.

Folliwing up from coderefinery#252 (comment)

Follow up from coderefinery#252 (comment)

Keeping the italics, though. As suggested here: coderefinery#252 (comment) Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

Vinye · 2026-09-21T07:24:41Z

+There are many forms of testing.
+
+One can write test programs and run them:
+```console
+$ python3 run-test.py
+
+running: sample_data/set1.csv --output=tests/set1.txt
+CORRECT
+```
+
+In the most basic form of a software test, 
+the observed result is compared with expected result (an "*oracle*")
+in order to establish correctness.  
+Here are some examples of this testing pattern 
+in different programming languages:


Should we actually point out that the first one is an end-to-end test and the latter is an example of an unit test? (and links to glossary, just like in the above table)

I had to think a little but you're right:

edited: suggestion as rendered here is misleading, github review suggestions don't work well with backtick blocks

The problem is that I can't fix it here on github. Will fix separately

see 5a068cf

Vinye · 2026-09-21T08:24:34Z

+## The basics: what knowledge do you need?
+
+Learn one test framework well enough for basics:
+- Explore and use the good tools that exist out there


Link learner to quick-reference/#available-tools for a starting point?

Oops, it's in the next item, but the link could be changed such that it points to #available-tools instead of the top of the page

Done in 77f7b50

Vinye

Took a look again, made some new observations 🔍

Also, just to make sure, sphinx has flagged some lines, I assume you have seen those warnings?

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

I think it fits better there, after we have shown why automated tests are important. Also before showing automated tests many of the "typical problems" might sound abstruse or unfamiliar.

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

mmesiti · 2026-09-22T17:35:34Z

Sorry for all the force-pushing. I think I almost addressed all the comments so far... Still some more work to come.

Thanks @Vinye for the awesome feedback (GitHub should have an award for the most patient reviewer on earth)!

Vinye

I went through the texts with a fine-tooth comb once more (I'd like to use a Finnish idiom here: read it like the devil reads the Bible 👿 ), so expect a lot of typo and stylistic fixes.

Moreover, I suggested fixes for the Sphinx warnings in quick-reference.md (some extraneous spaces on new lines). The Sphinx warning in conclusions.md on the other hand could be fixed by adding myst_heading_anchors = 3 to the conf.py file, see here. The link works even without this, but this would silence the warning.

Vinye · 2026-09-23T06:34:55Z

+
+Fortunately for us, 
+someone has already found a solution for most of these
+and created {term}`testing framework`s that we can use


Suggested change

Fortunately for us,

someone has already found a solution for most of these

and created {term}`testing framework`s that we can use

Fortunately for us,

someone has already found a solution for most of these

and created {term}`testing framework`s that we can use.

This looks to me like a repeat of the previous fix, so I won't apply it

Vinye · 2026-09-23T06:39:24Z

+
+Note: not all frameworks solve all problems 
+(also because sometimes the underlying language does not have the necessary features).


I'd move this note below the table (now we are giving a note about problems before we present the problems in the table)

I'd argue against this just because this table is not meant to be discussed in detail, it's more there as a collection of reasons why one should use a testing framework. The message "... but not all frameworks tackle all the problems" is perhaps more important than the details contained in the table.

Perhaps, we should move the whole table to an appendix (or just the "quick-reference" episode, which isn't that quick anymore), put a link here, and go through it if permits? Not sure about that though.

Punctuation, Jargon, Capitalization, repeated words and most importantly sphinx warning errors Co-authored-by: Anja <Vinye@users.noreply.github.com> Co-authored-by: Michele Mesiti <mmesiti@users.noreply.github.com>

also, made title of forge feature comparison table more sober.

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

slimmer columns for readability

mmesiti · 2026-09-23T19:04:20Z

Moreover, I suggested fixes for the Sphinx warnings in quick-reference.md (some extraneous spaces on new lines). The Sphinx warning in conclusions.md on the other hand could be fixed by adding myst_heading_anchors = 3 to the conf.py file, see here. The link works even without this, but this would silence the warning.

Thank you for pointing out where the problems were. Eventually I added a manual anchor to fix the warning, probably less invasive than changing the conf.py (I think it makes sense to keep the conf.py as uniform as possible across the lessons).

Adding a picture here might be unnecessary since there's so many that one can find online.

false positive is a bug is flagged when not present false negative is when a bug is there but it's not flagged. Notice that tests can only be used to prove incorrectness, not to prove correctness, so the condition we are testing for is the presence of bugs, not its absence.

mmesiti · 2026-09-24T17:56:48Z

Also with dfb8fb7 corrected this issue that I would like to stress:

local testing: false negative -> false positive

false positive is a bug is flagged when not present
false negative is when a bug is there but it's not flagged.

Notice that tests can only be used to prove incorrectness,
not to prove correctness,
so the condition we are testing for is the presence of bugs,
not its absence.

mmesiti · 2026-09-24T18:00:32Z

I think that, apart from the test design episode, these are all the changes that I would make. Thanks @Vinye for the review and I'd like @johanhellsvik to have a final look, I don't expecting making other changes to the general structure of the lesson.

(I think that the test design episode might deserve separate PRs, which are partially already there)

mmesiti added 3 commits September 11, 2026 13:33

Vocabulary from motivation to quick reference

319d791

Some connection to glossary from the "motivations" episode

Fix "what can test help you do?" section

03bf5d9

removed paragraph whose topic was already covered in the table

moved recommendations to the conclusions

bdbe948

mmesiti mentioned this pull request Sep 17, 2026

Autumn 2026 review #246

Merged

Add comment on self-hosting

b13270d

Addressing comment in coderefinery#246 (review)

Vinye reviewed Sep 18, 2026

View reviewed changes

Comment thread content/remotely.md

Vinye reviewed Sep 18, 2026

View reviewed changes

Comment thread content/conclusions.md Outdated

Vinye reviewed Sep 18, 2026

View reviewed changes

Comment thread content/conclusions.md Outdated

Vinye reviewed Sep 18, 2026

View reviewed changes

mmesiti mentioned this pull request Sep 19, 2026

Review objectives and keypoints #254

Open

mmesiti and others added 7 commits September 19, 2026 15:21

Small rewording fixes

afad9c8

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi> Co-authored-by: Michele Mesiti <mmesiti@users.noreply.github.com>

Completely remove "where to start" from motivation

e68959f

it belongs to conclusions, where the topic is already covered.

Elaborate on runner self-hosting

4fd8d0d

Folliwing up from coderefinery#252 (comment)

Motivations: small reformatting for readability

17a80d2

Follow up from coderefinery#252 (comment)

fix typo

52aae85

Use epigraph directive for quote in motivations

5cfed98

Keeping the italics, though. As suggested here: coderefinery#252 (comment) Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

Motivations: refactor + frameworks

598f314

Vinye reviewed Sep 21, 2026

View reviewed changes

Comment thread content/motivation.md Outdated

Vinye reviewed Sep 21, 2026

View reviewed changes

Comment thread content/motivation.md Outdated

Vinye reviewed Sep 21, 2026

View reviewed changes

Comment thread content/conclusions.md Outdated

Vinye reviewed Sep 21, 2026

View reviewed changes

Comment thread content/motivation.md Outdated

mmesiti commented Sep 22, 2026

View reviewed changes

Comment thread content/remotely.md Outdated

mmesiti force-pushed the aut2026review-batch2 branch 3 times, most recently from 4b52849 to 801d877 Compare September 22, 2026 17:10

mmesiti and others added 2 commits September 22, 2026 19:12

Fixes from Anja's review

f5d4062

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

move "why use a testing framework" to conclusions

6a2db30

I think it fits better there, after we have shown why automated tests are important. Also before showing automated tests many of the "typical problems" might sound abstruse or unfamiliar.

mmesiti force-pushed the aut2026review-batch2 branch from 801d877 to 6a2db30 Compare September 22, 2026 17:12

mmesiti and others added 3 commits September 22, 2026 19:25

mention e2e and unit test in examples in motivations

5a068cf

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

Some addition to conclusions and quick reference

3189516

modularity <-> testability in conclusions

04abe9a

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

mmesiti force-pushed the aut2026review-batch2 branch from 0817bc7 to 04abe9a Compare September 22, 2026 17:31

Vinye reviewed Sep 23, 2026

View reviewed changes

mmesiti and others added 5 commits September 23, 2026 14:57

Lots of fixes by Anja

d481962

Punctuation, Jargon, Capitalization, repeated words and most importantly sphinx warning errors Co-authored-by: Anja <Vinye@users.noreply.github.com> Co-authored-by: Michele Mesiti <mmesiti@users.noreply.github.com>

further sphinx warning fixes

c70cdf9

(minor) terser / better wording

906dea2

also, made title of forge feature comparison table more sober.

conclusions: Improve wording on knowledge needed

77f7b50

Co-Authored-by: Anja Virkkunen <anja.virkkunen@aalto.fi>

conclusions: test framework feat table col fixes

084c7a1

slimmer columns for readability

mmesiti marked this pull request as ready for review September 23, 2026 19:04

mmesiti requested a review from johanhellsvik September 23, 2026 19:05

mmesiti added 3 commits September 24, 2026 19:42

Mention the test pyramid

a187734

Adding a picture here might be unnecessary since there's so many that one can find online.

Split 'Exercise' in local testing in 2 sections

ce9039a

		```

		Most scientists nowadays depend on software for research.

		@@ -177,13 +209,6 @@ Use the collaborative notes to answer these questions:

		## Where to start


		Note: not all frameworks solve all problems
		(also because sometimes the underlying language does not have the necessary features).

Conversation

mmesiti commented Sep 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmesiti Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vinye Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vinye left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmesiti Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mmesiti commented Sep 17, 2026 •

edited

Loading

mmesiti Sep 19, 2026 •

edited

Loading

Vinye Sep 18, 2026 •

edited

Loading

mmesiti Sep 22, 2026 •

edited

Loading

Vinye left a comment •

edited

Loading

mmesiti commented Sep 22, 2026 •

edited

Loading

mmesiti Sep 23, 2026 •

edited

Loading