[Scons-users] best practices: unit tests

[Mats Wichmann]

I have a few "is there a Best Known Method" type questions that I'll
dribble out over a few messages (that is, if I can keep my focus, I keep
getting dragged off onto other things).

This one has several interesting hits on the internet, to my
recollection - wiki pages, blogs, stackexchange, etc. The ones I've
looked at over several months didn't seem terribly comprehensive, though.

Unit tests are a key part of many software projects these days, and
often a part of acceptance criteria in code review systems. scons itself
follows this model - I see tests running every time you submit a pull
request on github.

What do people do here? Integrate the running of tests into the scons
builds? Keep them external? If integrated, how? On a smaller project I
suspect it could make sense to integrate the test tightly into the
build, such that a builder doesn't even report success until the
matching unit test has also passed - does anyone actually do that? In a
more complex project, the ordering can get confusing: e.g. if several
different directories each build a shared library and a unit test
program for one shared library may require use of others of the shared
libraries, you can't bind it that simply. What then - e.g. if you ought
to wait for the whole project to finish building before running the
various tests? In the case of the project I'm working on, tests are
built with googletest, which is told on invocation to generate a result
file and the CI system swings through and collects the various result
files - thus the pass/fail decision is external to scons - but the
launching of the tests is not. Indeed, it's big and complex enough that
running the unit tests is optional, which several of the CI builds do,
but a developer might not when testing a patch initially, as it could
slow them down too much.

Again... I realize that if it works, it's not wrong, I'm just curious
for thoughts on best practices, I assume others have had to think
through this question?

[R0b0t1]

On Thu, Feb 1, 2018 at 4:05 PM, Mats Wichmann <***@wichmann.us> wrote:
>
> I have a few "is there a Best Known Method" type questions that I'll
> dribble out over a few messages (that is, if I can keep my focus, I keep
> getting dragged off onto other things).
>
> This one has several interesting hits on the internet, to my
> recollection - wiki pages, blogs, stackexchange, etc. The ones I've
> looked at over several months didn't seem terribly comprehensive, though.
>
> Unit tests are a key part of many software projects these days, and
> often a part of acceptance criteria in code review systems. scons itself
> follows this model - I see tests running every time you submit a pull
> request on github.
>
> What do people do here? Integrate the running of tests into the scons
> builds? Keep them external? If integrated, how? On a smaller project I
> suspect it could make sense to integrate the test tightly into the
> build, such that a builder doesn't even report success until the
> matching unit test has also passed - does anyone actually do that? In a
> more complex project, the ordering can get confusing: e.g. if several
> different directories each build a shared library and a unit test
> program for one shared library may require use of others of the shared
> libraries, you can't bind it that simply. What then - e.g. if you ought
> to wait for the whole project to finish building before running the
> various tests? In the case of the project I'm working on, tests are
> built with googletest, which is told on invocation to generate a result
> file and the CI system swings through and collects the various result
> files - thus the pass/fail decision is external to scons - but the
> launching of the tests is not. Indeed, it's big and complex enough that
> running the unit tests is optional, which several of the CI builds do,
> but a developer might not when testing a patch initially, as it could
> slow them down too much.
>
> Again... I realize that if it works, it's not wrong, I'm just curious
> for thoughts on best practices, I assume others have had to think
> through this question?
>

Unit tests are a mixed bag. If your program's behavior can be defined
precisely and is limited in scope, unit tests may save you time; on
the other hand, if your program's behavior is not limited in scope
(i.e. it has lots of user interaction or takes continuous inputs) unit
tests may not save you time.

The reason for this is that, in general, it is not possible to prove
what something is by saying what it is not, so you can not prove your
program is "correct" by showing various ways it is known to not be
incorrect.

Using a precise example: If I pick an integer and let you guess what
it is and report whether your guess was correct, in the worst case,
you will spend forever guessing to find the number I picked, because
there are an infinite number of integers.

For this reason, view software projects that rely heavily on tests to
prove they are consistent[1] with a great deal of suspicion.


It sounds like referring you to an existing system would help - so,
you might look at Ant and Maven and the structure of large Java
projects.[2] There are typically src/ and test/ directories in the
project. The former is built as it is a dependency for everything in
the latter, and the latter should not have any dependencies between
themselves (or at least I've never seen any tests that depend on each
other, but it could happen).

Enumerating the tests and tracking their completion is quite a bit of
work, so there are frameworks or harnesses to do this, if it is not
part of the build tool. This is where my knowledge stops. Perl has
good unit test frameworks, and I have seen some that use Makefiles.

Cheers,
R0b0t1


[1]: https://github.com/ruby/ruby
[2]: https://github.com/scala/scala

[Mats Wichmann]

On 02/01/2018 04:11 PM, R0b0t1 wrote:

> Unit tests are a mixed bag.

Yeah, I know nearly all the arguments. Made a lot of them myself :) But
that's how modern projects work: the founders lay down TDD-style "thou
shalt not write an API without unit tests" rules. It's not my call to
change those requirements, and I think they're actually fine: API tests
work on verifying documented behavior and don't have the same
variability problems you cite.

> It sounds like referring you to an existing system would help - so,
> you might look at Ant and Maven and the structure of large Java
> projects.

Sure, you can depend on external stuff. The project I'm on does (in
particular, the java additions do make use of Maven, but I don't touch
those bits, so...). The question was about doing this in scons, since
this is an scons list. If the best practice is: don't try to do it in
scons, that's fine, I was just trying to flush out opinions.

[R0b0t1]

On Thu, Feb 1, 2018 at 8:19 PM, Mats Wichmann <***@wichmann.us> wrote:
> On 02/01/2018 04:11 PM, R0b0t1 wrote:
>
>> Unit tests are a mixed bag.
>
> Yeah, I know nearly all the arguments. Made a lot of them myself :) But
> that's how modern projects work: the founders lay down TDD-style "thou
> shalt not write an API without unit tests" rules. It's not my call to
> change those requirements, and I think they're actually fine: API tests
> work on verifying documented behavior and don't have the same
> variability problems you cite.
>

But they don't. API tests verify that the input selected gives the
output in the test. It is not a description that holds for all valid
input.

There is an even larger problem, which is that even if you can prove
that a program is "type safe" all you have done is prove that it is
logically consistent within some theoretical framework. Even if you
can prove it adheres to a specification, you still haven't proved that
the specification reflects what you actually want to happen.

>> It sounds like referring you to an existing system would help - so,
>> you might look at Ant and Maven and the structure of large Java
>> projects.
>
> Sure, you can depend on external stuff. The project I'm on does (in
> particular, the java additions do make use of Maven, but I don't touch
> those bits, so...). The question was about doing this in scons, since
> this is an scons list. If the best practice is: don't try to do it in
> scons, that's fine, I was just trying to flush out opinions.
>

Sorry, I wasn't suggesting you rely on Ant/Maven. But for e.g. Python
you just use the unittest module, or Nose and py.test and then run it.
With Java projects, the tests/ directory is usually just complete
programs that run. It may be one program. It doesn't matter which
build system you are using.

The answer is very language specific.

Cheers,
R0b0t1

[Rob Boehne]

R030t1,

You're argument that unit tests are not perfect is quite valid. I personally find them indispensable, I'd sooner find a new employer than be responsible for code that I'm not allowed to develop with TDD. However that's really off-topic for this list, and doesn't address Mats' question of "how" to do it well.

Mats:

Having the unit tests part of the Scons tree is the best way to do this. Because giving a unit test target should rebuild the product being tested when necessary, otherwise you could test out of date products. Using Scons to build tests, run tests and generate reports is easier because you can reuse environments, and I keeps track of what is current.
I say this mainly because we have a product that builds the tests with Scons, and then has a separate python script to run the tests. When we decided to run the tests in parallel, we had to implement this ourselves, whereas Scons would have already been doing that. My advice would be to think of a tree where the ultimate output is a report of the test run, output from each test are dependencies, and so on.

HTH,

Rob Boehne

On 2/1/18, 9:33 PM, "Scons-users on behalf of R0b0t1" <scons-users-***@scons.org on behalf of ***@gmail.com> wrote:

On Thu, Feb 1, 2018 at 8:19 PM, Mats Wichmann <***@wichmann.us> wrote:
> On 02/01/2018 04:11 PM, R0b0t1 wrote:
>
>> Unit tests are a mixed bag.
>
> Yeah, I know nearly all the arguments. Made a lot of them myself :) But
> that's how modern projects work: the founders lay down TDD-style "thou
> shalt not write an API without unit tests" rules. It's not my call to
> change those requirements, and I think they're actually fine: API tests
> work on verifying documented behavior and don't have the same
> variability problems you cite.
>

But they don't. API tests verify that the input selected gives the
output in the test. It is not a description that holds for all valid
input.

There is an even larger problem, which is that even if you can prove
that a program is "type safe" all you have done is prove that it is
logically consistent within some theoretical framework. Even if you
can prove it adheres to a specification, you still haven't proved that
the specification reflects what you actually want to happen.

>> It sounds like referring you to an existing system would help - so,
>> you might look at Ant and Maven and the structure of large Java
>> projects.
>
> Sure, you can depend on external stuff. The project I'm on does (in
> particular, the java additions do make use of Maven, but I don't touch
> those bits, so...). The question was about doing this in scons, since
> this is an scons list. If the best practice is: don't try to do it in
> scons, that's fine, I was just trying to flush out opinions.
>

Sorry, I wasn't suggesting you rely on Ant/Maven. But for e.g. Python
you just use the unittest module, or Nose and py.test and then run it.
With Java projects, the tests/ directory is usually just complete
programs that run. It may be one program. It doesn't matter which
build system you are using.

The answer is very language specific.

Cheers,
R0b0t1
_______________________________________________
Scons-users mailing list
Scons-***@scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users

[Gary Oberbrunner]

I've been busy and just saw this.

> What do people do here? Integrate the running of tests into the scons
> builds? Keep them external? If integrated, how?

I create a UnitTests alias that builds my unit tests, and that's part of my
default build. I usually run them manually after building them with SCons;
trying to have SCons run them works but then sometimes you want to re-run
them without doing anything else and that gets trickier.

-- Gary


On Thu, Feb 1, 2018 at 5:05 PM, Mats Wichmann <***@wichmann.us> wrote:

>
> I have a few "is there a Best Known Method" type questions that I'll
> dribble out over a few messages (that is, if I can keep my focus, I keep
> getting dragged off onto other things).
>
> This one has several interesting hits on the internet, to my
> recollection - wiki pages, blogs, stackexchange, etc. The ones I've
> looked at over several months didn't seem terribly comprehensive, though.
>
> Unit tests are a key part of many software projects these days, and
> often a part of acceptance criteria in code review systems. scons itself
> follows this model - I see tests running every time you submit a pull
> request on github.
>
> What do people do here? Integrate the running of tests into the scons
> builds? Keep them external? If integrated, how? On a smaller project I
> suspect it could make sense to integrate the test tightly into the
> build, such that a builder doesn't even report success until the
> matching unit test has also passed - does anyone actually do that? In a
> more complex project, the ordering can get confusing: e.g. if several
> different directories each build a shared library and a unit test
> program for one shared library may require use of others of the shared
> libraries, you can't bind it that simply. What then - e.g. if you ought
> to wait for the whole project to finish building before running the
> various tests? In the case of the project I'm working on, tests are
> built with googletest, which is told on invocation to generate a result
> file and the CI system swings through and collects the various result
> files - thus the pass/fail decision is external to scons - but the
> launching of the tests is not. Indeed, it's big and complex enough that
> running the unit tests is optional, which several of the CI builds do,
> but a developer might not when testing a patch initially, as it could
> slow them down too much.
>
> Again... I realize that if it works, it's not wrong, I'm just curious
> for thoughts on best practices, I assume others have had to think
> through this question?
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-***@scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>



--
Gary

[Hill, Steve (FP COM)]

I’ve fully integrated our unit-tests (and module tests) into SCons, though as a separate build. The test suites get built and installed into a well-known directory and they are run (by SCons) from there. This means that a) only those unit-tests affected by a given change-set get rerun – but all those affected do get rebuilt and rerun, and b) the test-suites are easy to find if someone wants to run one manually for debugging, for instance.



S.



From: Scons-users [mailto:scons-users-***@scons.org] On Behalf Of Gary Oberbrunner
Sent: 18 February 2018 00:19
To: SCons users mailing list
Subject: Re: [Scons-users] best practices: unit tests



I've been busy and just saw this.



> What do people do here? Integrate the running of tests into the scons

> builds? Keep them external? If integrated, how?



I create a UnitTests alias that builds my unit tests, and that's part of my default build. I usually run them manually after building them with SCons; trying to have SCons run them works but then sometimes you want to re-run them without doing anything else and that gets trickier.



-- Gary





On Thu, Feb 1, 2018 at 5:05 PM, Mats Wichmann <***@wichmann.us> wrote:


I have a few "is there a Best Known Method" type questions that I'll
dribble out over a few messages (that is, if I can keep my focus, I keep
getting dragged off onto other things).

This one has several interesting hits on the internet, to my
recollection - wiki pages, blogs, stackexchange, etc. The ones I've
looked at over several months didn't seem terribly comprehensive, though.

Unit tests are a key part of many software projects these days, and
often a part of acceptance criteria in code review systems. scons itself
follows this model - I see tests running every time you submit a pull
request on github.

What do people do here? Integrate the running of tests into the scons
builds? Keep them external? If integrated, how? On a smaller project I
suspect it could make sense to integrate the test tightly into the
build, such that a builder doesn't even report success until the
matching unit test has also passed - does anyone actually do that? In a
more complex project, the ordering can get confusing: e.g. if several
different directories each build a shared library and a unit test
program for one shared library may require use of others of the shared
libraries, you can't bind it that simply. What then - e.g. if you ought
to wait for the whole project to finish building before running the
various tests? In the case of the project I'm working on, tests are
built with googletest, which is told on invocation to generate a result
file and the CI system swings through and collects the various result
files - thus the pass/fail decision is external to scons - but the
launching of the tests is not. Indeed, it's big and complex enough that
running the unit tests is optional, which several of the CI builds do,
but a developer might not when testing a patch initially, as it could
slow them down too much.

Again... I realize that if it works, it's not wrong, I'm just curious
for thoughts on best practices, I assume others have had to think
through this question?



_______________________________________________
Scons-users mailing list
Scons-***@scons.org
https://pairlist4.pair.net/mailman/listinfo/scons-users







--

Gary

[Bill Deegan]

Would someone like to summarize this email thread an publish in the wiki?

On Mon, Feb 19, 2018 at 3:29 AM, Hill, Steve (FP COM) <***@cobham.com
> wrote:

> I’ve fully integrated our unit-tests (and module tests) into SCons, though
> as a separate build. The test suites get built and installed into a
> well-known directory and they are run (by SCons) from there. This means
> that a) only those unit-tests affected by a given change-set get rerun –
> but all those affected do get rebuilt and rerun, and b) the test-suites are
> easy to find if someone wants to run one manually for debugging, for
> instance.
>
>
>
> S.
>
>
>
> *From:* Scons-users [mailto:scons-users-***@scons.org] *On Behalf Of *Gary
> Oberbrunner
> *Sent:* 18 February 2018 00:19
> *To:* SCons users mailing list
> *Subject:* Re: [Scons-users] best practices: unit tests
>
>
>
> I've been busy and just saw this.
>
>
>
> > What do people do here? Integrate the running of tests into the scons
>
> > builds? Keep them external? If integrated, how?
>
>
>
> I create a UnitTests alias that builds my unit tests, and that's part of
> my default build. I usually run them manually after building them with
> SCons; trying to have SCons run them works but then sometimes you want to
> re-run them without doing anything else and that gets trickier.
>
>
>
> -- Gary
>
>
>
>
>
> On Thu, Feb 1, 2018 at 5:05 PM, Mats Wichmann <***@wichmann.us> wrote:
>
>
> I have a few "is there a Best Known Method" type questions that I'll
> dribble out over a few messages (that is, if I can keep my focus, I keep
> getting dragged off onto other things).
>
> This one has several interesting hits on the internet, to my
> recollection - wiki pages, blogs, stackexchange, etc. The ones I've
> looked at over several months didn't seem terribly comprehensive, though.
>
> Unit tests are a key part of many software projects these days, and
> often a part of acceptance criteria in code review systems. scons itself
> follows this model - I see tests running every time you submit a pull
> request on github.
>
> What do people do here? Integrate the running of tests into the scons
> builds? Keep them external? If integrated, how? On a smaller project I
> suspect it could make sense to integrate the test tightly into the
> build, such that a builder doesn't even report success until the
> matching unit test has also passed - does anyone actually do that? In a
> more complex project, the ordering can get confusing: e.g. if several
> different directories each build a shared library and a unit test
> program for one shared library may require use of others of the shared
> libraries, you can't bind it that simply. What then - e.g. if you ought
> to wait for the whole project to finish building before running the
> various tests? In the case of the project I'm working on, tests are
> built with googletest, which is told on invocation to generate a result
> file and the CI system swings through and collects the various result
> files - thus the pass/fail decision is external to scons - but the
> launching of the tests is not. Indeed, it's big and complex enough that
> running the unit tests is optional, which several of the CI builds do,
> but a developer might not when testing a patch initially, as it could
> slow them down too much.
>
> Again... I realize that if it works, it's not wrong, I'm just curious
> for thoughts on best practices, I assume others have had to think
> through this question?
>
>
>
> _______________________________________________
> Scons-users mailing list
> Scons-***@scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>
>
>
>
> --
>
> Gary
>
> _______________________________________________
> Scons-users mailing list
> Scons-***@scons.org
> https://pairlist4.pair.net/mailman/listinfo/scons-users
>
>

[Jason Kenny]

Sorry for coming in late in this.
I had made a UnitTest() functions for Scons via an extension I made. It defined a run_utest and utest targets. it would build some code as a test map it to an alias. it would also create a run sandbox to run the test in so it was easy to map custom data as input for the test run. What was nice about this was that with Scons it was a side effect that running the unit tests only the test that had source changes would rebuild and run. this was very nice for larger projects.
It works well and I have someone using it in very clever ways. I think the core part of any "test" API added to SCons needs to be that it is generic. there are many different test frameworks out there. I think that making sure people can use gtest, tut, catch, my custom setup... is a critical component to having a good unit test API for a build system. I think it is going too far and making built-in dashboards, etc. They are nice.. but that should be independent systems from a build system.
Jason
________________________________
From: Scons-users <scons-users-***@scons.org> on behalf of Bill Deegan <***@baddogconsulting.com>
Sent: Monday, February 19, 2018 9:42 AM
To: SCons users mailing list
Subject: Re: [Scons-users] best practices: unit tests

Would someone like to summarize this email thread an publish in the wiki?

On Mon, Feb 19, 2018 at 3:29 AM, Hill, Steve (FP COM) <***@cobham.com<mailto:***@cobham.com>> wrote:

I’ve fully integrated our unit-tests (and module tests) into SCons, though as a separate build. The test suites get built and installed into a well-known directory and they are run (by SCons) from there. This means that a) only those unit-tests affected by a given change-set get rerun – but all those affected do get rebuilt and rerun, and b) the test-suites are easy to find if someone wants to run one manually for debugging, for instance.



S.



From: Scons-users [mailto:scons-users-***@scons.org<mailto:scons-users-***@scons.org>] On Behalf Of Gary Oberbrunner
Sent: 18 February 2018 00:19
To: SCons users mailing list
Subject: Re: [Scons-users] best practices: unit tests



I've been busy and just saw this.



> What do people do here? Integrate the running of tests into the scons

> builds? Keep them external? If integrated, how?



I create a UnitTests alias that builds my unit tests, and that's part of my default build. I usually run them manually after building them with SCons; trying to have SCons run them works but then sometimes you want to re-run them without doing anything else and that gets trickier.



-- Gary





On Thu, Feb 1, 2018 at 5:05 PM, Mats Wichmann <***@wichmann.us<mailto:***@wichmann.us>> wrote:

I have a few "is there a Best Known Method" type questions that I'll
dribble out over a few messages (that is, if I can keep my focus, I keep
getting dragged off onto other things).

This one has several interesting hits on the internet, to my
recollection - wiki pages, blogs, stackexchange, etc. The ones I've
looked at over several months didn't seem terribly comprehensive, though.

Unit tests are a key part of many software projects these days, and
often a part of acceptance criteria in code review systems. scons itself
follows this model - I see tests running every time you submit a pull
request on github.

What do people do here? Integrate the running of tests into the scons
builds? Keep them external? If integrated, how? On a smaller project I
suspect it could make sense to integrate the test tightly into the
build, such that a builder doesn't even report success until the
matching unit test has also passed - does anyone actually do that? In a
more complex project, the ordering can get confusing: e.g. if several
different directories each build a shared library and a unit test
program for one shared library may require use of others of the shared
libraries, you can't bind it that simply. What then - e.g. if you ought
to wait for the whole project to finish building before running the
various tests? In the case of the project I'm working on, tests are
built with googletest, which is told on invocation to generate a result
file and the CI system swings through and collects the various result
files - thus the pass/fail decision is external to scons - but the
launching of the tests is not. Indeed, it's big and complex enough that
running the unit tests is optional, which several of the CI builds do,
but a developer might not when testing a patch initially, as it could
slow them down too much.

Again... I realize that if it works, it's not wrong, I'm just curious
for thoughts on best practices, I assume others have had to think
through this question?



_______________________________________________
Scons-users mailing list
Scons-***@scons.org<mailto:Scons-***@scons.org>
https://pairlist4.pair.net/mailman/listinfo/scons-users<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpairlist4.pair.net%2Fmailman%2Flistinfo%2Fscons-users&data=02%7C01%7C%7C0ea0f6deeb074c73b22808d577af7b00%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636546517915221310&sdata=kMLTfyND4BsLigJlR%2F%2Bh8XEkTx7JHfW7R6t72obwdcM%3D&reserved=0>





--

Gary