This implements the configuration mechanism I described in #2621. The heavy lifting is delegated to [node-config](https://github.com/lorenwest/node-config) with a minor assist from [dotenv](https://github.com/motdotla/dotenv).
`private/secret.json` has been replaced with environment variables and/or `config/local.yml`. See `doc/server-secrets.md`.
Because `server.js` was long a monolith, there are a bunch of shims in place to facilitate unit testing. A few of the test suites share port 1111 which means if one of them fails to set up, the port won't be freed and other unrelated tests will fail. Some of the tests which trigger server setup include timeouts which were added to give setup code time to run. In one the test suites, we actually modify `process.argv`, which seems completely gross.
This implements a few changes which improve this:
1. Separate the server from the server startup script, splitting out `lib/server.js`.
2. Inject config into the server and validate the config schema.
3. Inject config into the service test runner.
4. Use `portfinder`, a popular utility for grabbing open ports during testing.
5. Switch more of the setup code from callbacks to async-await.
Overall it leaves everything acting more reliably and looking rather cleaner, if in a few places more verbose.
It also fixes the root cause of #1455, a `setTimeout` in `rate-limit`. Off and on during development of this changeset, Mocha would decide not to exit, and that turned out to be the culprit.
Fix#1455
- The goal of this PR is:
- Consume the new service-definition format. (#2397)
- Make the frontend more readable.
- Behavior changes:
- I changed the **Image** field in the markup modal to show only the path.
- I added another click-to-select field below that shows the complete URL.
- This made it easier to suppress the live badge preview while it contains placeholders like `:user` or `:gem`, a minor tweak discussed at https://github.com/badges/shields/issues/2427#issuecomment-442972100.
- The search box now searches all categories, regardless of the current page. (This is an improvement, I would say.)
- I did not deliberately address performance, though I ripped out a bunch of anonymous functions and avoided re-filtering all the examples by category on every render, which I expect will not hurt. I haven't really tested this on a mobile connection and it'd be worth doing that.
- It would be great to have some tests of the components, though getting started with that seemed like a big project and I did not want to make this any larger than it already is.
It's a medium-sized refactor:
1. Replace `BadgeExamples`, `Category` and `Badge` component with a completely rewritten `BadgeExamples` component which renders a table of badges, and `CategoryHeading` and `CategoryHeadings` components.
2. Refactor `ExamplesPage` and `SearchResults` components into a new `Main` component.
3. Rewrite the data flow for `MarkupModal`. Rather than rely on unmounting and remounting the component to copy the badge URL into state, employ the `getDerivedStateFromProps` lifecycle method.
4. Remove `prepareExamples` and `all-badge-examples`.
5. Rewrite the `$suggest` schema to harmonize with the service definition format. It's not backward-compatible which means at deploy time there probably will be 10–20 minutes of downtime on that feature, between the first server deploy and the final gh-pages deploy. 🤷♂️ (We could leave the old version in place if it seems worth it.)
6. Added two new functions in `make-badge-url` with tests. I removed _most_ of the uses of the old functions, but there are some in parts of the frontend I didn't touch like the static and dynamic badge generators, and again I didn't want to make this any larger than it already is.
7. Fix a couple bugs in the service-definition export.
This starts the rewrite of the dynamic badges. I've pulled into BaseService an initial version of the query param validation from #2325.
I've extended from BaseJsonService to avoid duplicating the deserialization logic, though it means there is a bit of duplicated code among the three dynamic services. The way to unravel this would be to move the logic from `_requestJson` and friends from the base classes into functions so DynamicJson can inherit from BaseDynamic. Would that be worth it?
This introduces a regression of #1446 for this badge.
Close#2345
* Added matrix badge
* decreased the size of the matrix logo by more than 50%
* returning the size in fetch() instead of an object
* found another way to register a throwaway account (guest account). this one actually works on matrix.org, but I kept the old way as a backup method. also changed the POST from /members to /state because guest accounts didn't work with /members
* updated logo to a recolored version of the official logo
* Removed unnecessary comments.
Added documentation on how to create the badge URL.
Added a test that hits a real room to test for API compliance.
URLs are now obtained from getter functions.
Added JSON schema for the /state API request.
Improved state response filter.
Replaced example URL room ID to a dedicated testing room.
Made some error messages more helpful.
* correctly implemented requested changes
* changed color hex codes to constants
There's a lot going on in this PR, though it's all interdependent, so the only way I can see to break it up into smaller pieces would be serially.
1. I completely refactored the functions for managing cache headers. These have been added to `services/cache-headers.js`, and in some ways set the stage for the rest of this PR.
- There are ample higher-level test of the functionality via `request-handler`. Refactoring these tests was deferred. Cache headers were previously dealt with in three places:
- `request-handler.js`, for the dynamic badges. This function now calls `setCacheHeaders`.
- `base-static.js`, for the static badges. This method now calls the wordy `serverHasBeenUpSinceResourceCached` and `setCacheHeadersForStaticResource`.
- The bitFlip badge in `server.js`. 👈 This is what set all this in motion. This badge has been refactored to a new-style service based on a new `NoncachingBaseService` which does not use the Shields in-memory cache that the dynamic badges user.
- I'm open to clearer names for `NoncachingBaseService`, which is kind of terrible. Absent alternatives, I wrote a short essay of clarification in the docstring. 😝
2. In the process of writing `NoncachingBaseService`, I discovered it takes several lines of code to instantiate and invoke a service. These would be duplicated in three or four places in production code, and in lots and lots of tests. I kept the line that goes from regex to namedParams (for reasons) and moved the rest into a static method called `invoke()`, which instantiates and invokes the service. This _replaced_ the instance method `invokeHandler`.
- I gently reworked the unit tests to use `invoke` instead of `invokeHandler`– generally for the better.
- I made a small change to `BaseStatic`. Now it invokes `handle()` async as the dynamic badges do. This way it could use `BaseService.invoke()`.
3. There was logic in `request-handler` for processing environment variables, validating them, and setting defaults. This could have been lifted whole-hog to `services/cache-headers.js`, though I didn't do that. Instead I moved it to `server-config.js`. Ideally `server-config` is the only module that should access `process.env`. This puts the defaults and config validation in one place, decouples the config schema from the entire rest of the application, and significantly simplifies our ability to test different configs, particularly on small units of code. (We were doing this well enough before in `request-handler.spec`, though it required mutating the environment, which was kludgy.) Some of the `request-handler` tests could be rewritten at a higher level, with lower-level data-driven tests directly against `cache-headers`.
* Allow to run service tests for remote shields instance
* Allow to run service tests for remote shields instance - doc
* Load testedServerUrl from env variable
* Load 'skipIntercepted' flag from env variable
There's a lot of demand for the Gitlab badges (#541) and the PR has been lingering, so I thought I'd start off one of the simple ones as a new-style service. This one is SVG-based, so it shouldn't require the API-token logic which could use some more testing and will require us to create an app and configure it on our server.
We don't have any validation in place for `queryParams`. Probably this should be added to BaseService, though for the time being I extracted a helper function.
Thanks to @LVMBDV for getting this work started in #1838!
- Stop running daily service tests in the main repo (since they're now handled [over here](https://github.com/badges/daily-tests)
- Add coverage and separate daily tests badges with links to coveralls
- Update our coverage ignores
- Move scripts, which do not need coverage, into `scripts/`
- Split out coverage test for npm package
- Remove spurious env var
Ref: #1584#2314
This picks up @RedSparr0w's work in #1802.
1. The handler for the static badge is moved into its own service with a synchronous handler. Avoiding an async call may make the static badges slightly faster, though it may be worth profiling this if it turns out we want asynchronous static badges in the future. If it doesn't make a performance difference we could make this handler `async` like the others.
2. Most of the custom static-badge logic is in a BaseStaticBadge base class.
3. Rewrite the static Gitter badge to use BaseStaticBadge.
4. A bit of minor cleanup in related functions.
This simplifies and further optimizes text-width computation by computing the entire width table in advance, and serializing it in the style of QuickTextMeasurer (#1390). This entirely removes the need for PDFKit at runtime. This has the advantage of fixing #1305 – more generally: producing the same result everywhere – without having to deploy a copy of Verdana.
The lifting is delegated to these three libraries, which are housed in a monorepo: https://github.com/metabolize/anafanafo
I'd be happy to move it into the badges org if folks want to collaborate on maintaining them.
QuickTextMeasurer took kerning pairs into account, whereas this implementation does not. I was thinking kerning would be a necessary refinement, though this seems to work well enough.
I dropped in a binary-search package to traverse the data structure, in part to conserve space. This causes a moderate performance regression, though there is ample room for improving on that: https://github.com/badges/shields/pull/2311#issuecomment-439182704
When I deployed 5e99aad2de to s0, shortly after Sentry picked up an unhandled error. I'm not sure which of the legacy badges this is in.
The bug was introduced just now, in #2257. Oops!
A test would have caught this, but I don't think it's worth wrapping tests around this difficult-to-test code. It makes more sense I think, to refactor the remaining badges that use it, replace it with something using async/await (maybe based on [node-cache](https://www.npmjs.com/package/node-cache), and test that.
* move gh-badges files out of /lib
As far as possible, this is just moving files
around and updating paths however there are 2
functional changes in this commit:
- remove use of lib/register-chai-plugins.spec
in badge-cli.spec.js
- remove use of starRating()
in text-measurer.spec.js
* update service tests that use colorscheme.json
* split package.json in two
* clean up import
* don't hard-code path
* start a changelog
* put a license file in the package dir
* re-organise documentation 📚
* don't pack test files
* remove favicon from Makefile
* give package its own test command
* link the docs better in README
The NuGet badge examples are straggling in all-badge-examples. Rather than move them as is, I thought it made more sense to refactor the services and see if they could be generated. I didn't take that on here; this is a straight rewrite of the badges. The old implementations were fairly difficult to follow. The new implementations are complicated too, though I hope much more readable.
Though the NuGet behaviors could be consolidated into a single flag, I split `withTenant` and `withFeed` into separate flags, thinking naming the behaviors makes the implementations easier to understand. I defaulted these to true, thinking that really this is really a MyGet implementation which is generalized to NuGet. Though maybe it makes more sense to have the MyGet style as the default. Probably it doesn't matter much either way.
I added a helper class ServiceUrlBuilder to construct the Shields service URL. It's useful in this complex case where the URL must be built up conditionally. This might be useful in a couple other places.
I also wrote a new service to handle the Powershell badges. They've diverged a little bit from the Nuget v2. There's a bit of shared code which I factored out.
If the XML Nuget APIs are more reliable, we could consider switching everything else over to them, though for now I would like to get this merged and get #2078 fixed.
Fix#2078
The GitHub service family is the largest, and as yet untouched by our service rewrite. I thought I would start the process by tackling one service.
This pull request has a few things going on:
1. Rename pull-request-status to pull-request-check-state. We have another badge called pull request status. It seems like the checks are called one thing in the UI and another thing in the API, which is unfortunate. If other folks have strong feelings about the name, I’ll defer.
2. Move its tests and tighten up the syntax.
3. Move its badge examples including the doc string.
4. Add a new helper `errorMessagesFor` to use in the new services in place of `githubCheckErrorResponse`. It seems like we didn’t really use the `errorMessages` parameter to `githubCheckErrorResponse`, so I pared this down. I’m not sure if this is the function we’ll ultimately want, but it seems like a good place to start.
5. Pull fetch code I _know_ we use in other places into `github-common-fetch`. As in the PR I just opened for azure-devops, this takes a functional approach to the shared code, which is more direct, nimble, and easy to reason about than inheritance.
6. Create `GithubAuthService` which functions identically to BaseJsonService, except for one thing, which is that it uses the token pool. I accomplished this by adding a `_requestFetcher` property to BaseService, which is initialized to `sendAndCacheRequest` in the constructor, and can be overridden in subclasses. Since we weren’t using `_sendAndCacheRequest` directly except in BaseService and tests, I removed that property. I like this approach to patching in the GitHub auth because it’s very simple and creates no new API exposure. However, the way we’re doing the dependency injection feels a bit odd. Maybe the eventual refactor of request-handler would be a godo time to revisit this.
The GitHub requests go through many, many layers of indirection at this point. Later on it would be good to shave some of these off, perhaps once the legacy GitHub services have been converted, or when all the services are done and we can take another look at the base service hierarchy. The work in #2021 and #1205 is also related.
I went down a rabbit hole while trying to untangle the bug in the dockbit and bitrise examples https://github.com/badges/shields/pull/2234#pullrequestreview-169997546.
The URL generation code is spaghetti-like, with functions, many of which I wrote, with opaque names, doing similar but not identical things, and making slightly incompatible assumptions about the way query strings are handled.
I got a bit lost and need to take a step back.
Meanwhile, this is a small piece of work I did that’s worth keeping. It doesn’t scratch the surface of the tangle, but it does remove a bit of duplication.
It also makes a minor stylistic ES6 change in the handling of default arguments.
Ref: #2027
* Basic process metrics
* Enable Prometheus by an environment variable
* Code formatting
* Documentation for Prometheus metrics
* Link from README to documentation of Prometheus
* Link from README to documentation of Prometheus
* Link from README to documentation of Prometheus
* Separate module for metrics + tests
* Metrics limited by IP
* Metrics are forbidded for all requets by default
* Code refactoring
* allowedIps passed as a string to PrometheusMetrics
* Handle missing config
* METRICS_PROMETHEUS_ALLOWED_IPS added to documentation
* Log info about enabled metrics
* Unused code removed
* package-lock.json updated
* prom-client updated to 11.1.2
* Code refactoring
* Do not read IP address from X-Forwarder-For header
