Question: will this project be deprecated in favor of PlatformIO?
ianfixes opened this issue · comments
I'm trying to set up a template for Continuous Integration of Arduino library code (both compilation and unit tests), using Travis CI.
A Google search led me to travis-ci-arduino as one of the best examples of how to do the "does it compile" test on Travis. However, it looks like PlatformIO is what Travis recommends for this:
https://docs.travis-ci.com/user/integration/platformio/#Examples
Are there advantages to using this method instead of PlatformIO?
Yes, no additional dependency (PlatformIO) and closer to "original" Arduino builds.
Are you saying that you evaluated PlatformIO for your purposes and found it to be lacking, or that you are satisfied with the solution you have?
Side question, how are you running these CI scripts from your local machine prior to pushing code to github? On OSX, the only practical thing I could do was read through install.sh myself and copy those arguments to the command line -- I have to invoke a binary that's in a different path (capital-A Arduino at /Applications/Arduino.app/Contents/MacOS/Arduino, which is annoying). Is that how you do it?
Personally I use both PlatformIO and travis-ci-arduino. When I want to test that something compiles well on Arduino, I use travis-ci-arduino. I am not running these CI scripts from my local machine.
Hmm. Does this imply that you've noticed instances where PlatformIO passes but travis-ci-arduino fails?
No, but I didn't really compare.
It turns out that PlatformIO is a paid service and not a freely available program. I've submitted a pull request to Travis CI to update their docs accordingly.
FYI we just fixed some bugs, if you haven't had success - it should all be working now thx!
It's not that I haven't had success... it's that this script doesn't allow for unit testing. I am porting the CI idea from your install.sh script into a ruby gem. So far I have everything except the actual compilation command implemented in my sample script. Unit tests will be accomplished by writing some C++ libraries for Arduino mocks (allowing you to control the clock, fake serial ports, etc) that I'll bundle with the gem.
I'd love your comments in a week or so when I get this into a more stable state. Would a CI + unit testing script (inspired by the same "use one script everywhere" design of this repo) be of interest to you?
ive never used ruby so cannot test, but if you have a PR that doesn't change around the original install.sh plz submit a PR!
Unfortunately, mine (https://github.com/ianfixes/arduino_ci) is different from install.sh. It still makes a minimal impact on the library being tested, but as shown here:
# .travis.yml
sudo: false
language: ruby
script:
- bundle install
- bundle exec arduino_ci_remote.rbYou need to add a Gemfile:
source 'https://rubygems.org'
gem 'arduino_ci'And that's basically it -- a YAML file controls the rest. A working example of a CI-enabled Arduino project would look like this:
https://github.com/ianfixes/Arduino-Queue.h/tree/2018-01-25_ci
Most notably (and the point of this whole exercise), you can unit test your library with files like this:
#include <ArduinoUnitTests.h>
#include "../Queue.h"
unittest(empty_queue)
{
Queue<int> queue(5);
assertEqual(0, queue.count()); // initially empty
assertEqual(0, queue.peek()); // 0 happens to be the default int val
assertEqual(0, queue.pop()); // popping when empty also returns the default
assertEqual(0, queue.count()); // still empty
}
unittest(fill_and_empty)
{
Queue<int> queue(5);
queue.push(11);
queue.push(19);
queue.push(44);
assertEqual(3, queue.count());
assertEqual(11, queue.pop());
assertEqual(19, queue.pop());
assertEqual(44, queue.pop());
}
unittest_main()
Example output from a CI job (https://travis-ci.org/ianfixes/Arduino-Queue.h/builds/333540256#L522):
$ bundle exec arduino_ci_remote.rb
Downloading Arduino binary with wget
--2018-01-26 01:15:03-- https://downloads.arduino.cc/arduino-1.8.5-linux64.tar.xz
Resolving downloads.arduino.cc (downloads.arduino.cc)... 158.69.94.227
Connecting to downloads.arduino.cc (downloads.arduino.cc)|158.69.94.227|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 103807940 (99M) [application/octet-stream]
Saving to: ‘arduino-1.8.5-linux64.tar.xz’
100%[======================================>] 103,807,940 3.05MB/s in 33s
2018-01-26 01:15:36 (3.03 MB/s) - ‘arduino-1.8.5-linux64.tar.xz’ saved [103807940/103807940]
Extracting archive with tar
Installing library under test... ✓
Library installed at /home/travis/Arduino/libraries/Arduino-Queue.h... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Unit testing queue-tests.cpp...
TAP version 13
1..5
# Subtest: empty
ok 1 - assertEqual 0 == queue.count()
ok 2 - assertEqual 0 == queue.peek()
ok 3 - assertEqual 0 == queue.pop()
ok 4 - assertEqual 0 == queue.count()
1..4
ok 1 - empty
# Subtest: filling_and_emptying
ok 1 - assertEqual 0 == queue.count()
ok 2 - assertEqual testSize == queue.space()
(snip)
@ladyada I wanted to follow up on this.
What I wrote is a feature-complete replacement for your install.sh, using install.sh as my guide. My MVP was to support the same operations:
- Automatically download the Arduino executable
- Install dependent libraries
- Compile each example on each of several platforms
- Be able to customize which boards are used for compiling each example, since not all hardware supports each library
Then, I added the following features that you might find useful:
- Ability to write and run unit tests
- Install dependent libraries from GitHub or other sources
- Run CI tests on your local machine too (OSX, Windows, and Linux are supported)
- Ability to define additional platforms to run tests on
- Report out-of-bounds and use-after-free memory errors, if the compiler supports it
I've submitted adafruit/Adafruit_FONA#91 to show how my system works in practice, the results are here: https://travis-ci.org/adafruit/Adafruit_FONA/builds/350489990
Relevant portion:
...Unit testing comms.cpp ✓
Setting compiler warning level... ✓
Verifying FONA3G_setbaud.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying FONA3G_setbaud.ino... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Verifying FONA_SMS_Response.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying FONA_SMS_Response.ino... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Verifying FONAtest.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying FONAtest.ino... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Verifying FONAtest_KEY_mod.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying FONAtest_KEY_mod.ino... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Verifying GPS.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying GPS.ino... ✓
Switching to board for uno (arduino:avr:uno)... ✓
Verifying IncomingCall.ino... ✓
Switching to board for leonardo (arduino:avr:leonardo)... ✓
Verifying IncomingCall.ino... ✓
Are there any features I'm missing?
sounds cool - we've got 1000 github repos so not sure when we'll get to implementing this but next time we do a FONA sweep we will check it out :)
(thanks for writing it - it looks really nice!)
Assuming that I could write a script to automatically convert over each repo (foreach repo, look for .travis.yml and/or any skip-files, convert to equivalent arduino_ci config, submit pull request), what are the "must-have features" in your CI script? I noticed that you print out a JSON report at the end, and I'm interested in how you're consuming that.
Also, I'm happy to move this discussion if you can recommend a more appropriate forum.
(For future searchability I'm adding this link to the other issue: #20 )
/me thinks
i dont know if we're ready to completely redo our libraries just yet :)
can we merge in your project as an alternative install script? like people can choose which they prefer - i want to include it for others even if im not quite ready (i dont know ruby so maintenance would be tough)
Aah, I'm giving off the wrong impression here! What I'm really asking is "if you close your eyes and imagine the perfect CI script (and you could wave a magic wand to have it), what features would it have?"
Regarding an "alternative install script", that should be straightforward. I will set up a PR on https://github.com/adafruit/Adafruit-WS2801-Library that demonstrates both CI pieces running side by side and link it here.
ideal CI would be super fast, thats the most important thing we've found :) everything else is pretty easy to set up!
Can you enable CI on https://travis-ci.org/adafruit/Adafruit-WS2801-Library ? I'm setting up the demo in this PR:
enabked!
Out of curiosity, when you say that you want it to be "super fast", is that because your only way to test your changes is with Travis (and not your local machine)?
because it has to run on every commit and PR. if it takes a long time, and there's an error, the author has to check back and fix it. people will not run CI locally!
I've completed this work. adafruit/Adafruit-WS2801-Library#21 demonstrates both CI methods running in parallel jobs.
Speed considerations
The execution time seems to vary widely. In this job the two scripts take roughly the same amount of time to complete. In this job, arduino_ci was nearly 2 minutes (20%) faster.
You should check the logs to make sure I'm accurate on this, but I believe the same set of platforms and examples are being verified for both scripts. I temporarily disabled Travis' caching to make it a fair assessment.
Unit test considerations
Without adding significant time to the build, here is arduino_ci evaluating a unit test and finding a problem in the code, where travis-ci-arduino can't.
A detailed analysis of the problem is easier to see using the OSX version of g++, since (unlike Linux g++) it provides file/line/function name:
https://travis-ci.org/adafruit/Adafruit-WS2801-Library/jobs/431549500#L160
==17062==ERROR: AddressSanitizer: heap-use-after-free on address 0x6030000002b0 at pc 0x0001021c8cee bp 0x7ffeeda39f90 sp 0x7ffeeda39f88
READ of size 1 at 0x6030000002b0 thread T0
#0 0x1021c8ced in Adafruit_WS2801::show() Adafruit_WS2801.cpp:203
#1 0x1021c9876 in test_set_strip_values::task() strip.cpp:13
#2 0x1021cc770 in Test::test() ArduinoUnitTests.h:165
#3 0x1021cbe44 in Test::run(Test::ReporterTAP*) ArduinoUnitTests.h:137
#4 0x1021cb9e9 in Test::run_and_report(int, char**) ArduinoUnitTests.h:155
#5 0x1021cb928 in main strip.cpp:44
#6 0x7fff51921114 in start (libdyld.dylib:x86_64+0x1114)
0x6030000002b0 is located 0 bytes inside of 30-byte region [0x6030000002b0,0x6030000002ce)
freed by thread T0 here:
#0 0x102255a9d in wrap_free (libclang_rt.asan_osx_dynamic.dylib:x86_64+0x56a9d)
#1 0x1021c87dd in Adafruit_WS2801::~Adafruit_WS2801() Adafruit_WS2801.cpp:97
#2 0x1021c87f8 in Adafruit_WS2801::~Adafruit_WS2801() Adafruit_WS2801.cpp:96
It should be noted that jobs take considerably longer to run on OSX. I'd be glad to find a compiler flag that will offer the same support on a Linux host.
Hope this is useful!
thanks, we do use caching quite a bit to speed up CI's FYI!
Yup, disabling the cache was only to accurately measure the run times between the two scripts. I would expect to re-enable it along with any other pull request comments you have.