This merges the `node-8` branch. The heavy lift was by @Daniel15 with refactoring from me and a patch by @RedSparr0w.
* New API for registering services (#963)
* Disable Node 6 tests on node-8 branch (#1423)
* BaseService: Factor out methods _regex and _namedParamsForMatch (#1425)
- Adjust test grouping
- Rename data -> queryParams, text -> message
* BaseService tests: Use Chai (#1450)
* BaseService: make serviceData and badgeData explicit and declarative (#1451)
* fix isValidStyle test (#1544)
* Run tests in Node 9, not Node 6 (#1543)
Rather than depend on Shields production, use the GitHub auth info from CI. It's disorienting to have our own CI go down when production is down. It also makes it harder to review PRs when there are ops issues.
I don’t like that our build goes red on master all the time due to flaky service tests. I thought I’d look into other CI services that would make it possible to run the scheduled tests nightly without causing those messages to show up.
CircleCI, Heroku CI, and Codeship were obvious choices. Heroku CI wasn’t free and I didn’t have any experience with Codeship, so I looked into CircleCI. I’ve used their 1.0 system a lot though this was my first time on their 2.0 system. As with earlier versions, they’ve put a lot of work into making the build fast – perhaps more than any other CI system I’ve seen.
I had such good results, my goal shifted from scheduled daily builds (that don’t litter our commit history with red builds) to improving the CI experience as a whole.
This change made a big impact:
- Build logs load much, much faster. In the test I just ran, 22 seconds to < 2 seconds, a 90% improvement.
- Status of each step shows up right in the GitHub UI, which makes it much faster to see exactly what’s failed.
- Builds run about 50-75% faster on account of parallelism.
- GitHub service tests are fixed. This has been a long-standing issue.
- Ability to ssh into a build container to debug failures.
Here’s what I did:
- Created custom Docker images with our dependencies. To be honest, I’m not even sure these are necessary, only to install the greenkeeper-lockfile. We could get dejavu from npm. They make startup very fast.
- Created an npm-install stage which loads all dependencies into node_modules and caches them.
- Created separate stages for our main tests, service tests, and frontend tests, and stages to run the main tests and service tests in Node 6. These run in parallel, up to four at a time.
- Separated service test ID output from the service test results themselves. (I check these often during the PR process, when I confirm that service tests actually ran. Because the production Shields server caches the title, after updating it you can’t tell whether the update is taking effect.)
- Added a personal access token for the shields-ci user. This should actually fix the long-standing issue #979. CircleCI provides an option to “Pass secrets to builds from forked pull requests,” which means unlike Travis, they’ll give us enough rope to shoot ourselves in the foot.
- Schedule a daily build, which runs all the service tests.
Use the endpoint on img.shields.io added in #1114 to fetch the PR title.
This sidesteps the authentication requirement, and helps with #979 – except for PRs that need to run the Github service tests, which will need a separate solution.
The intended behavior of the bracketed [github], [bower], [discord] service names in the pull request title is to trigger the designated service tests. That way, affected services can be proven working during code review, without needing to run tests on a dev machine, nor running all the slow (and flaky) service tests.
Example pull request titles:
- [Travis] Fix timeout issues
- [Travis Sonar] Support user token authentication
- [CRAN CPAN CTAN] Add test coverage
The observed behavior is that, whenever bracketed service names are provided, all of the service tests run.
This is due to a Mocha limitation, which is that exclusive tests (it.only and describe.only) can only be applied synchronously. In other words, if we try to fetch the PR title and then add exclusive tests in the callback, all the tests will run anyway. This is true even when using _mocha --delay, as we are, and is true whether I use request or node-fetch.
Undoubtedly this could be fixed, though it's not worth it. The problem is obscure and therefore low priority for Mocha, which is quite backlogged.
And, there is an easy workaround, which is to generate the list of services to test in a separate process.
The pull request script test:services:pr is now split into two parts. First the :prepare script infers the pull request context, fetches the PR title, and writes the list of affected services to a file. Then the :run script reads the list of affected services and runs the appropriate tests.
In addition to sidestepping the Mocha bug, this setup makes it easier to reason about and debug these two steps of the test runner on a dev machine, and since I can't get pipefail to work – and want to be able to run the steps separately – I'm not using Node's built in pre scripts.
Overall, separating these concerns this makes the test runner easier to reason about.
Paul MelnikowandGitHub