I don’t like that our build goes red on master all the time due to flaky service tests. I thought I’d look into other CI services that would make it possible to run the scheduled tests nightly without causing those messages to show up.
CircleCI, Heroku CI, and Codeship were obvious choices. Heroku CI wasn’t free and I didn’t have any experience with Codeship, so I looked into CircleCI. I’ve used their 1.0 system a lot though this was my first time on their 2.0 system. As with earlier versions, they’ve put a lot of work into making the build fast – perhaps more than any other CI system I’ve seen.
I had such good results, my goal shifted from scheduled daily builds (that don’t litter our commit history with red builds) to improving the CI experience as a whole.
This change made a big impact:
- Build logs load much, much faster. In the test I just ran, 22 seconds to < 2 seconds, a 90% improvement.
- Status of each step shows up right in the GitHub UI, which makes it much faster to see exactly what’s failed.
- Builds run about 50-75% faster on account of parallelism.
- GitHub service tests are fixed. This has been a long-standing issue.
- Ability to ssh into a build container to debug failures.
Here’s what I did:
- Created custom Docker images with our dependencies. To be honest, I’m not even sure these are necessary, only to install the greenkeeper-lockfile. We could get dejavu from npm. They make startup very fast.
- Created an npm-install stage which loads all dependencies into node_modules and caches them.
- Created separate stages for our main tests, service tests, and frontend tests, and stages to run the main tests and service tests in Node 6. These run in parallel, up to four at a time.
- Separated service test ID output from the service test results themselves. (I check these often during the PR process, when I confirm that service tests actually ran. Because the production Shields server caches the title, after updating it you can’t tell whether the update is taking effect.)
- Added a personal access token for the shields-ci user. This should actually fix the long-standing issue #979. CircleCI provides an option to “Pass secrets to builds from forked pull requests,” which means unlike Travis, they’ll give us enough rope to shoot ourselves in the foot.
- Schedule a daily build, which runs all the service tests.
- Fix issue in Firefox 57 when run from static build
- Fix color parameter in dynamic badge maker
- Correctly apply maxAge in usage badges
- Follow the WHATWG lead and begin to standardize on URL, not URI (https://url.spec.whatwg.org/#goals)
Both of the depcheck's are throwing errors
OS: `Windows 10`
Node: `v8.7.0`
Current Output:
```cmd
npm run frontend-depcheck
> gh-badges@1.3.0 frontend-depcheck C:\Users\Dan\Documents\GitHub\shields
> check-node-version --node '>= 8.0'
C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:678
throw new TypeError('Invalid comparator: ' + comp);
^
TypeError: Invalid comparator: '
at Comparator.parse (C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:678:11)
at new Comparator (C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:662:8)
at C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:828:12
at Array.map (<anonymous>)
at Range.parseRange (C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:827:13)
at Range.<anonymous> (C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:769:17)
at Array.map (<anonymous>)
at new Range (C:\Users\Dan\Documents\GitHub\shields\node_modules\semver\semver.js:768:40)
at C:\Users\Dan\Documents\GitHub\shields\node_modules\check-node-version\index.js:119:32
at module.exports (C:\Users\Dan\Documents\GitHub\shields\node_modules\map-values\index.js:9:18)
npm ERR! Windows_NT 10.0.16299
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Dan\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "run" "frontend-depcheck"
npm ERR! node v8.7.0
npm ERR! npm v4.3.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! gh-badges@1.3.0 frontend-depcheck: `check-node-version --node '>= 8.0'`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the gh-badges@1.3.0 frontend-depcheck script 'check-node-version --node '>= 8.0''.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the gh-badges package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! check-node-version --node '>= 8.0'
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs gh-badges
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls gh-badges
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Dan\AppData\Roaming\npm-cache\_logs\2017-12-04T00_30_04_974Z-debug.log
```
```cmd
npm run server-depcheck
> gh-badges@1.3.0 server-depcheck C:\Users\Dan\Documents\GitHub\shields
> check-node-version --node '>= 6.0 < 9.0'
The system cannot find the file specified.
npm ERR! Windows_NT 10.0.16299
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Users\\Dan\\AppData\\Roaming\\npm\\node_modules\\npm\\bin\\npm-cli.js" "run" "server-depcheck"
npm ERR! node v8.7.0
npm ERR! npm v4.3.0
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! gh-badges@1.3.0 server-depcheck: `check-node-version --node '>= 6.0 < 9.0'`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the gh-badges@1.3.0 server-depcheck script 'check-node-version --node '>= 6.0 < 9.0''.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the gh-badges package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR! check-node-version --node '>= 6.0 < 9.0'
npm ERR! You can get information on how to open an issue for this project with:
npm ERR! npm bugs gh-badges
npm ERR! Or if that isn't available, you can get their info via:
npm ERR! npm owner ls gh-badges
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR! C:\Users\Dan\AppData\Roaming\npm-cache\_logs\2017-12-04T00_32_13_243Z-debug.log
```
Looks like its due to `'` being used around the version rather than `"`.
This PR Output:
```cmd
npm run frontend-depcheck
> gh-badges@1.3.0 frontend-depcheck C:\Users\Dan\Documents\GitHub\shields
> check-node-version --node ">= 8.0"
```
```cmd
npm run server-depcheck
> gh-badges@1.3.0 server-depcheck C:\Users\Dan\Documents\GitHub\shields
> check-node-version --node ">= 6.0 < 9.0"
```
- Periodically log github auth information
- Tokens are hashed which reduces the security risk inherent in the logs
- A consistent hash is used so tokens can be correlated across the three data structures and across the three servers
- Add an admin endpoint for github auth information
- Tokens are returned as-is to enable troubleshooting (e.g. comparing our reqRemaining to github’s)
While working on #1288 I configured frontend tests. I'm going in a different direction with that, though it seems worth keeping the test configuration. I added tests for one of the helper methods.
- Heroku reads the Node version from package.json. We're about to upgrade to Node 8 so this change to `engines` is only pre-emptive. It won't have any effect on the production servers.
- The production deploy scripts were missing the frontend files. This fixes that.
- This modifies the build setup to allow `BASE_URL=/`, which makes all requests relative to the page itself. That simplifies deploying the "debugging" frontend to the production servers, and makes it easy to host the frontend on Heroku.
I rewrote the frontend in React using a module bundler. It's matched feature-for-feature with the current frontend, with only slight changes in the styling. I did not fuss about making the styling identical; the badge popup looks particularly different.
This makes the front end much easier to develop. I'm really looking forward to implementing #701, to which this paves the way.
This makes light use of Next.js, which provides webpack config and dev/build tooling. We’ll probably replace it with create-react-app or our own webpack setup because unfortunately it comes with a lot of runtime overhead (the build is 400k).
Let’s open new issues for bugs and features, and track other follow-ups here: https://github.com/badges/shields/projects/1
The version in Shields goes back to last August. I reviewed the commits and didn't see any obvious incompatibilities. Hopefully @espadrine can weigh in!
Sazerac is a library for data-driven tests, where a series of tests asserts that the return value of a function matches the expected value. It provides nice syntax for tightening this up.
https://hackernoon.com/sazerac-data-driven-testing-for-javascript-e3408ac29d8c
This converts our tests to use it, and replaces some similar home-grown code.
I fixed one bug I encountered along the way: mikec/sazerac#12.
* Push frontend to production servers at /index.html
Local production builds will use local server instead of img.shields.io, to support local testing
* Restore https://img.shields.io to example URIs
* Add User Defined URL support
* Add test
* eslint fixes
* 100% coverage
* use JSONpath
* update try.html
* Update tests, restore prefix & suffix
* order dependencies alphabetically
* update jsonpath version dependency
* update url structure & move to query strings
* update error handling & remove xml refrences
* fix eslint errors
* update tests
* update dynamic badge generator
* url -> uri & decode uri
Allow an encoded url to be used.
needed for any uri that requires params in the address
* resolve conflicts
* update for new page generation
* check uri is defined
* add query params to `request-handler.js`
* eslint fixes
* add test for no uri specified
* move query params to be local to dynamic badge
* update tests
* dynamic badge gen use same base url as static badge
- Followup from #1163
- Retire try.html
- Separate build config for dev and production
- Move config for badge examples into the JS build
- Move the prod transform into npm scripts
- In the future this could be handled using a bundler plugin
- make website builds production build as before
- Run the production build in CI to make sure it’s working
- Build the frontend on Heroku
I developed this for #820 to provide the correct cache behavior when a service wants to use custom parameters from the query string.
For safety, only the declared parameters (and the global parameters) are provided to the service. Consequently, failure to declare a parameter results in the parameter not working at all (undesirable, but easy to debug) rather than indeterminate behavior that depends on the cache state (undesirable, but hard to debug).
This pull request sets us up to generate the badge examples dynamically from data and code.
Right now, try.html is still checked in, mostly for the benefit of reading this diff, though it should be removed on the next pass to avoid unnecessary complexity at merge time.
- Add tests to request-handler to prepare for some tweaks to caching for #820
- Clean up code in request-handler: renames, DRY, arrows, imports
- Allow for clean shutdown of `setInterval` code. This requires the ability to cancel autosaving.
- Upgrade to Mocha 4, and clean up so the process exits on its own (see mochajs/mocha#3044)
- Better encapsulate analytics
- `make setup` (i.e. `make`) in the dev tutorial seems an unnecessary step. Badges will load from my local server after a clone, as long as I open try.html.
This adds badges for Github issues and pull requests. You can display the state, title, username, number of comments, age, time since last update, and state of checks.
Provides an endpoint the Shields CI can use to fetch PR titles for #979 and resolves#1011.
The intended behavior of the bracketed [github], [bower], [discord] service names in the pull request title is to trigger the designated service tests. That way, affected services can be proven working during code review, without needing to run tests on a dev machine, nor running all the slow (and flaky) service tests.
Example pull request titles:
- [Travis] Fix timeout issues
- [Travis Sonar] Support user token authentication
- [CRAN CPAN CTAN] Add test coverage
The observed behavior is that, whenever bracketed service names are provided, all of the service tests run.
This is due to a Mocha limitation, which is that exclusive tests (it.only and describe.only) can only be applied synchronously. In other words, if we try to fetch the PR title and then add exclusive tests in the callback, all the tests will run anyway. This is true even when using _mocha --delay, as we are, and is true whether I use request or node-fetch.
Undoubtedly this could be fixed, though it's not worth it. The problem is obscure and therefore low priority for Mocha, which is quite backlogged.
And, there is an easy workaround, which is to generate the list of services to test in a separate process.
The pull request script test:services:pr is now split into two parts. First the :prepare script infers the pull request context, fetches the PR title, and writes the list of affected services to a file. Then the :run script reads the list of affected services and runs the appropriate tests.
In addition to sidestepping the Mocha bug, this setup makes it easier to reason about and debug these two steps of the test runner on a dev machine, and since I can't get pipefail to work – and want to be able to run the steps separately – I'm not using Node's built in pre scripts.
Overall, separating these concerns this makes the test runner easier to reason about.
