Search code, repositories, users, issues, pull requests...

opened

on May 5, 2018

· edited by s-celles

Hello,

In #6646 I mentioned that library manager should display license for each library.

I think a more general issue is that Arduino should be able to automate quality assurance of libraries.
http://downloads.arduino.cc/libraries/library_index.json could help

The presence of a license file is one point, but assurance that the code is still compilable is an other important point. Then, assurance, thanks to unit tests, that library have some chance to be still usable is a last point.

In JuliaLang we have https://pkg.julialang.org/ which list all libraries (outside of any IDE).
So, thanks to continuous integration and this quality assurance testing, we know if a library is compilable or not and if tests of a library are passing or not...

https://github.com/ianfixes/arduino_ci provides scripts to ensure that when a commit is made, it doesn't break library (prevents him from compiling) and that unit tests are still passing.

All this may be insufficient, we could still have broken eggs, but that's a begining.

Require developers to tag their release (with semantic versionning) and set up requirements (with a minimal version number for each dependencies) in a configuration file could be an other important point (discussed in #5795 )

Kind regards

Author

Maybe some developpers with knowledge of C/C++ dependency manager could comment on the last point about requirements.

Code linting could also be checked.

A last, but important part of code quality assurance is code coverage but we are probably still far away from this point.

Collaborator

Here's the only way I can see this working:

When the Library Manager indexer job finds a new tag it looks for the CI configuration file for any standard CI service (e.g. Travis CI, Circle CI, GitLab CI) and runs (or checks existing?) CI build for that tag, then records the result of that build in http://downloads.arduino.cc/libraries/library_index.json.
Filter checkboxes are added to the Arduino IDE's Library Manager GUI:
- Tested: CI build passed.
- Untested: No CI build was done.
- Test Failed: CI build failed.

The creation of the CI configuration file and which tests will be done on the library will be up to the library maintainer. That could include compile testing, unit testing, and linting. It is not feasible for Arduino to generate tests for 3rd party libraries.

Component: Board/Lib Manager

added

on May 6, 2018

mentioned this in 2 issues

on May 13, 2018

Author

As written in #6646 (comment) detect license inconsistencies (ie a different license in library.properties and LICENSE file) could be part of the Arduino libraries quality assurance process

mentioned this

on Jan 1, 2019

Allow specific tests or groups of tests to be run selectively Arduino-CI/arduino_ci#48

Author

Pinging @ianfixes about unit tests / continuous integration for Arduino libraries

mentioned this

on Jan 1, 2019

For your consideration: Enabling unit tests on Travis CI using a different CI script adafruit/ci-arduino#20

Thanks for bringing me into this discussion, and I'm somewhat flattered by this attention towards my testing library. I still consider arduino_ci to have an alpha-level feature set, but so far the concept has proven to be fairly solid.

Unit testing works, period. There's room for the mocks to become more elegant, but there's a lot already supported (e.g. faking a hardware modem via serial). You do not need hardware present to run the unit tests, regardless of architecture.
Memory management issues can be exposed and annotated (via compiler features): Prevent segfaults by implementing the rule of three adafruit/Adafruit-WS2801-Library#20.
Library examples can be targeted to specific architectures via the config file, as an alternative to using preprocessor directives to work around compilation issues.
Extra architectures (any board that can be installed via the arduino CLI) can be declared and configured via the config file
Dependent libraries are installed automatically (although for now they must be declared manually in the config file)

Metadata tests (e.g. your comment about the LICENSE file) don't exist now, but I could certainly integrate them into the ruby side of the test runner. That would either be its own standalone script (similar to arduino_ci_remote.rb, the entry point for CI), or baked into that script and controlled via config.

Should we pick an Arduino library to use as a demo?

Collaborator

@ianfixes do you agree with my previous statement?:

The creation of the CI configuration file and which tests will be done on the library will be up to the library maintainer. That could include compile testing, unit testing, and linting. It is not feasible for Arduino to generate tests for 3rd party libraries.

Or do you think there is a way to do completely automated unit testing via arduino_ci without any configuration on the part of the library author?

As for testing of metadata and library structure, this is possible to do automatically (and in fact I'm already doing that independently). It's actually even more simple with the libraries in the Library Manager index because we know they're in the repository root and thus don't need to deal with the headache of searching for the library in subfolders. Here is the bash script I use to check metadata and library structure:
https://github.com/per1234/arduino-ci-script/blob/1.1.0/arduino-ci-script.sh#L1212-L2326

The Library Manager indexer actually already does a lot of verification of the library.properties and some of library structure (must be in repo root, must not contain executables). Any release that doesn't pass its verification is not added to the Library Manager index

SponsorContributor

It is not feasible for Arduino to generate tests for 3rd party libraries.

Why not? Anyone can select a widely used board, manually open each of library's examples and click Verify. Why can't a CI script do the same?

And really, why isn't doing substantially more considered to be feasible? Even a rather elaborate test which runs the libraries on real hardware and compares against a known-good result is the sort of thing you might expect take 1 person-month to initially create, and then maybe 2-3 hours per library to set up. While there are thousands of libraries, only a few hundred are very widely used. I just don't understand why something like this is assumed to be so far out of reach, when it could be done pretty effectively by a few summer interns working with direction and occasional assistance from a seasoned engineer / developer.

It is not feasible for Arduino to generate tests for 3rd party libraries.

For unit tests, I agree completely: the library maintainers would need to take full responsibility for writing tests as well as maintaining any given CI solution as a whole. Period.

For compiling the examples provided by the library, I disagree slightly: anyone can load & compile those (via the GUI or CLI), including a server-side process. That said, such an effort would fail since not every library works for every architecture. You would need some configuration file format to sort that out (I can humbly offer my own YAML format for consideration), and I can't say whether it would be worth it to roll that out and then support it.

The same reasoning would apply to adopting a system like NPM modules or RubyGems -- formats that include a metadata field for the git repo URL. Although this could enable querying the a given commit to see test status (e.g. on GitHub), this would be impractical. You'd need to correlate the library with a repository, and the library version with a specific commit, and it would rely on a developer base that is not primarily software engineers to spend time maintaining it.

But if you'll allow me take a step back... I took the spirit of this issue to be "enable automated testing" -- that's a prerequisite, no matter who performs the QA¹. A few things currently block such testing, which I had to work around or simply brute-force my way though as I built my library. I would love to work with you to make a more official solution to these.

I guess my question for you would be: What impact would it have on the Arduino ecosystem if we could give library maintainers more confidence in accepting code contributions (i.e. if they could easily set up pull-request testing on GitHub, to automatically verify those contributions)?

¹ minor nitpick @scls19fr : it's "Quality Assurance", not "Insurance"; not sure if that will affect the searchability of this issue

changed the title ~~[-][Library] Automate quality insurance testing of libraries[/-]~~ [Library] Automate quality assurance testing of libraries

on Jan 3, 2019

@PaulStoffregen

Even a rather elaborate test which runs the libraries on real hardware and compares against a known-good result is the sort of thing you might expect take 1 person-month to initially create, and then maybe 2-3 hours per library to set up

See my Arduino CI library: https://github.com/ianfixes/arduino_ci

It mocks all aspects of the hardware for unit testing (including clock), does not require hardware to run (i.e. can be run on Travis or Appveyor CI), and takes roughly 0.5-2 hours to set up on any given library.

That setup and maintenance should fall to the individual library maintainers, not the Arduino team.

12 remaining items

SponsorContributor

Can someone explain to me why using real hardware is considered so infeasible, but all this stuff is?

I mean, this is Arduino we're hypothetically talking about. A capable CI server machine, 2 or 3 of every Arduino product, a selection of popular shields, a pile of USB hubs with power-switch-per-port, and maybe even test gear like USB logic analyzers just isn't that expensive, maybe a couple thousand dollars?

@PaulStoffregen it sounds like you're describing a system where the CI machine would upload a sketch onto a board, and then provide you some scripting capability to send inputs to the I/O pins of the board and verify some expected outputs.

The problem with that type of system is that it can't do unit tests; it can only provide an "end-to-end" or "integration" test capability. Libraries like Queue (mentioned above) don't produce any effects on the I/O pins, they just provide some helpful functions to the developer. (You could go down the path of mapping internal state of a library to some expression on pins, but not all boards have the same pins and you'd be unable to test libraries that use all the pins. So that's a dead end.)

End-to-end testing does interest me (especially via some kind of simulator, which would be more scalable), but this seems like the wrong thread to discuss it (i.e. I don't get the sense that it could be provided by adding code to this repo in particular).

Collaborator

I think PaulStoffregen has a very interesting idea. The way I see it working is for Arduino to provide this as a "hardware-in-the-loop" testing service. They maintain a room full of every official board, with some way of monitoring output and providing input on every pin. An online API is provided to run the tests and check for the expected results. PIO Remote is somewhat related, but they're not providing hardware, just the infrastructure to interface with hardware remotely. This would require a significant investment on Arduino's part for initial setup plus ongoing maintenance (and I think PaulStoffregen is significantly underestimating). However, there is a case to be made that, even providing the service for free, it would end up paying off in the end by increasing the quality of the community written code that is such an important asset to Arduino. Of course they can use the system for their own testing as well.

As with ianfixes' arduino_ci, Adafruit's travis-ci-arduino, or my arduino-ci-script, I see an HIL testing service as being only indirectly related to the feature request made in this issue. It's something library authors would incorporate into their own CI builds. If they wanted to invest even more into the project, Arduino could sponsor someone to submit PRs to the most popular libraries, adding these tests to their CI builds. That could help influence more widespread use of the service.

Can someone explain to me why using real hardware is considered so infeasible

The only thing I consider infeasible is for HIL, unit, or compilation tests to be configured completely automatically. I've already explained why.

SponsorContributor

Ok then, duly noted on the unit tests. Not sure how I would apply that to the couple dozen very old Arduino library I maintain (after they were abandoned by their original authors), plus the ones I've authored... but I'll be watching and looking for examples as this develops.

Also planning to start using @ladyada's CI script soon....

adafruit-adabot

yay! @PaulStoffregen if you can add teensy as an option, and do a PR, that would allow us to test our libs against teensyduino build, could catch lil compilation errors

@PaulStoffregen I'd be interested in taking a crack at one of the old libraries you mention. It would be a good way to double-check the features of my CI library. Which one do you think would give you the most trouble?

SponsorContributor

OneWire would be my first choice.
https://github.com/PaulStoffregen/OneWire

Author

I noticed that arduino-cli is written in Go. I wonder if acceptance of an integrated Arduino CI tool with unit tests won't be higher if it won't depend on a script language such as Ruby.

mentioned this

on Jan 6, 2019

Basic unit testing of OneWire using TravisCI + arduino_ci PaulStoffregen/OneWire#67

There's no magic to my use of ruby, aside from the fact that it enjoys good cross-platform support and does not require executing arbitrary shell commands that are read from a remote source (e.g. source <(curl -SLs https://raw.githubusercontent.com/mostly_harmless.sh))

That type of installation process makes me very uncomfortable; at worst, it's a security risk; at best, there is no guarantee that two successive runs will use the same version of the script (which is an impediment to reproducible results).

Aside from providing semver-based packaging and dependency management (which comes with using a ruby gem), all that the ruby code does is handle the interactions with shell commands to run the arduino executable, the compilers, and the unit test binaries (and provide nice-looking output). Any object-oriented language will do.

OneWire would be my first choice.
https://github.com/PaulStoffregen/OneWire

@PaulStoffregen this work is done (well, 2 months ago now), I'm curious to hear your thoughts PaulStoffregen/OneWire#67

added

Collaborator

There is, in my opinion, another small step forward in this effort today.

As I mentioned above, the Arduino Library Manager system already does some basic checks on the library metadata, structure, and contents and rejects any releases that don't meet the requirements. These requirements are focused on the absolute essentials for a library to be compatible with the Library Manager system and the Arduino development software.

There are opportunities for additional universally applicable checks for things that are not absolutely essential at the moment, but that can provide a better experience for users (best practices) and less chance of breakage in the future (specification compliance). Arduino Lint provides rules spanning this entire spectrum. As of today, Arduino Lint is used as the validation system for Arduino Library Manager submissions and indexing releases. This tool provides the same hard requirements as always, which will cause a library to fail validation and be rejected. But it also runs the suite of additional rules on the libraries, treating violations as a warning. These warnings are presented to the library maintainer:

During the submission process as comments from the bot
During the indexing process as logs published on a dedicated webpage for each library

My hope is that these warnings will serve to encourage conscientious library maintainers to work beyond the bare minimum requirements and toward best practices.

Even though we are limited in what checks can be done in a completely automated fashion universally, even small improvements across thousands of libraries can be very significant.

mentioned this

on Nov 21, 2021

Removal request for "FireBase32" library-registry#708