Commit Graph

76 Commits

Author SHA1 Message Date
Pierre-Yves B
59dcdf24f3 Remove rate limiting functionality (#6513) 2021-05-20 08:02:44 +01:00
Pierre-Yves B
a51ef2996a Increase server request rate limits (#6511) 2021-05-16 22:25:39 +01:00
dependabot-preview[bot]
636ed87057 Build(deps-dev): bump prettier from 2.2.1 to 2.3.0, run [ciibestpractices eclipsemarketplacedownloads gemdownloads githubdeployments githubtotaldiscussions githubforks githubhacktoberfest githubissuessearch githubpackagejson githubsponsors githubtag jenkinscoverage node npm nuget scoop teamcitybuild testspace] (#6482)
* Build(deps-dev): bump prettier from 2.2.1 to 2.3.0

Bumps [prettier](https://github.com/prettier/prettier) from 2.2.1 to 2.3.0.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/2.2.1...2.3.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* chore: prettier

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Caleb Cartwright <caleb.cartwright@outlook.com>
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2021-05-15 08:49:20 +00:00
Pierre-Yves B
9c7baa26ab Deprecate [Bintray] service (#6423) 2021-04-24 18:05:59 +01:00
chris48s
22fb91fa36 use \r\n linebreaks when writing raw HTTP (#6386) 2021-04-19 18:29:40 +01:00
chris48s
7d452472de return a 408 Request Timeout response for timed out requests (#6371) 2021-04-11 21:29:13 +01:00
chris48s
b1fc492592 add setting which allows us to set a timeout on HTTP requests (#6364)
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2021-04-05 17:56:32 +00:00
chris48s
e0510ca761 Revert request --> got migration (for now) (#6277)
* Revert "send custom user agent when using got (#6256)"
This reverts commit fd7eddc7bb.

* Revert "Migrate request to got (part 1 of many) (#6160)"
This reverts commit 2359eb278b.

* install got as a prod dependency, allow npm 7
2021-03-17 20:10:38 +00:00
chris48s
9d404bc2b0 use got to push influx metrics (#6257) 2021-03-12 19:38:11 +00:00
chris48s
2359eb278b Migrate request to got (part 1 of many) (#6160)
* install new packages

* migrate request to got

* update dynamic json test

This is a behavioural difference between request and got

request will send the request, then we'll get a
`400 Bad Request` back and re-throw at as invalid

got will pick up that the URL is invalid and throw
`RequestError: URI malformed` before attempting to send it
which we'll re-throw as inaccessible

* fix OPM service

* fix wordpress querystring

Got doesn't natively support assmebling a querystring
from nested objects because it uses node's URLSearchParams
internally. Use qs and pass qs a string.

Wordpress is the only service that needs this,
so we could build the string manually in this case
if we don't want to take qs as a prod dependency.
It is mostly hard-coded values anyway.

* fix wercker

got overwrites any ?foo=bar in the URL string if
searchParams is also passed whereas request appends
see https://github.com/sindresorhus/got#url

* fix keybase

* add tests for got wrapper

* bootstrap global agent in server start
2021-03-09 20:03:52 +00:00
danielle
0d463b42e5 [Bitbucket] Server Adding Auth Tokens and Resolving Pull Request api … (#6076)
* [Bitbucket Server] Adding Auth Tokens and Resolving Pull Request api issue.  Added the bitbucket_server_username & bitbucket_server_password variables to the /core/server/server.js file.  Also changed the user variable in the /services/bitbucket/bitbucket/bitbucket-pull-request.service.js the api documentation (https://docs.atlassian.com/bitbucket-server/rest/5.16.0/bitbucket-rest.html#idm8287391664) defines the parameter as requiring the project name that the repository is contained in not the user name.

* [Bitbucket Server] Adding Auth Tokens and Resolving Pull Request api issue.  Added the bitbucket_server_username & bitbucket_server_password variables to the /core/server/server.js file.  Also changed the user variable in the /services/bitbucket/bitbucket/bitbucket-pull-request.service.js the api documentation (https://docs.atlassian.com/bitbucket-server/rest/5.16.0/bitbucket-rest.html#idm8287391664) defines the parameter as requiring the project name that the repository is contained in not the user name.

* Update bitbucket-pull-request.service.js

Updating the label as Bitbucket and Bitbucket Server have different parameters in the Api call.  Changing to compound name per @calebcartwright  recomendation.

* Update server.js

Removing code that was refactored and caught in my Pull Request

* Reversing commit back on bitbucket-pull-request.service.js to its previous state.

Co-authored-by: danielle <185722+danielle@users.noreply.github.com>
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2021-01-26 22:57:37 -06:00
Caleb Cartwright
2d9ac83bb4 fix: remove erroneous clearRequestCache call (#6067)
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2021-01-16 17:42:32 +00:00
dependabot-preview[bot]
e202b73c3c Build(deps): bump prom-client from 11.5.3 to 13.0.0 (#5962)
* Build(deps): bump prom-client from 11.5.3 to 13.0.0

Bumps [prom-client](https://github.com/siimon/prom-client) from 11.5.3 to 13.0.0.
- [Release notes](https://github.com/siimon/prom-client/releases)
- [Changelog](https://github.com/siimon/prom-client/blob/master/CHANGELOG.md)
- [Commits](https://github.com/siimon/prom-client/compare/v11.5.3...v13.0.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* deps: apply updates for prom version bump

* register.getMetricsAsJSON is async now

* PrometheusMetrics#metrics is async

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Caleb Cartwright <caleb.cartwright@outlook.com>
Co-authored-by: Marcin Mielnicki <marcin.mielnicki@gmail.com>
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2020-12-23 23:32:08 +00:00
Marcin Mielnicki
b748e62e0a Test for server running with metrics enabled (#5975)
* Test server running with metrics enabled

* Verify custom metric was send

* Extra code removed

* Tests moved to core/server/server.spec.js

* tickAsync value based on push interval

* Trigger Build

Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2020-12-23 18:52:08 +00:00
chris48s
8dd6da3cf9 Configure max-age header for frontend (#5775)
* upgrade scoutcamp

* set max age for static content, add tests

Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2020-11-12 20:04:20 +00:00
Paul Melnikow
1184938ed1 Align label and message props of coalesceBadge() with internal makeBadge() (#5719)
There is an internal `makeBadge()` function which is called from a few places in the server and from the public `makeBadge()` function which is a light wrapper. (Eventually we want to dogfood the public API: that's the work of #4950, and this helps with it by aligning the interfaces.)

Related to that is #3370, which is about aligning the `serviceData` schema (i.e. the result of `handle()`) with the public `makeBadge()` function.

A legacy quirk of the _private_ `makeBadge()` function is accepting a `text: ['label', 'message']` array instead of separate `{ label, message }` props like the rest of the codebase. `coalesceBadge()` has to translate from `{ label, message }` to `text: ['label', message']`. This removes that bit of indirection.

It also rewrites most of the tests of `coalesceBadge()` to use `.includes()`, providing IMO a slight improvement in readability.
2020-10-16 18:06:32 -04:00
Paul Melnikow
cbd8eba2c4 Fix REQUIRE_CLOUDFLARE for Heroku (#5712)
Ref #3027
2020-10-15 19:27:07 -04:00
Paul Melnikow
1fab1a7140 When configured, require requests to come from Cloudflare (#5666)
This is the code part of #3027, following [this article](https://www.viget.com/articles/heroku-cloudflare-the-right-way/) and using [this middleware](https://github.com/clive-io/cloudflare-middleware).

I pulled in the `addHandlerAtIndex()` function @chris48s wrote for #5574.

The middleware isn't perfect for scoutcamp, since it relies on `req.ip` which is something set by Express. However, the other solutions I found were either explicitly deprecated ([cloudflare-ip](https://www.npmjs.com/package/cloudflare-ip)) or relied on dynamically fetching the list of Cloudflare hosts ([cloudflare-ips](https://www.npmjs.com/package/cloudflare-ips)), which seems unnecessary as this list has not changed in several years.

I've left this off to start, so we can test it in production using an env var before we make it the production default.
2020-10-12 12:36:42 -04:00
Paul Melnikow
564a9145e9 Remove FsTokenPersistence (#5671)
Closes #5665
2020-10-10 21:59:52 +00:00
Paul Melnikow
f0b8480280 Inject shieldsSecret into GitHub token admin endpoint (#5631)
Ref #3393
2020-10-01 23:12:22 -04:00
chris48s
504015c0ba migrate hapi/joi to joi (#5624)
* update joi + joi-extension-semver

* @hapi/joi --> joi

Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2020-09-30 17:51:02 +00:00
Paul Melnikow
eb9743c25c Remove obsolete peer-to-peer GitHub token-sharing code (#5628)
Prevent tokens from being sent to the OVH servers. Instead, add them directly to the Redis DB.

Ref #3393
2020-09-29 21:22:30 -04:00
Paul Melnikow
bdc41670a9 Remove obsolete WebSocket logs monitor (#5629)
This never worked in prod (see #1460)
2020-09-29 21:05:55 -04:00
Pierre-Yves B
135b842946 Migrate [Discord] implementation to use bot token (#5346)
* Migrate [Discord] implementation to use bot token

* Rework authorization field creation

* Revert "Rework authorization field creation"

This reverts commit caf65bde5d.

* Add LGTM exclusion for hardcoded credentials
2020-07-24 18:04:12 +02:00
Pierre-Yves B
0fd557d7bb Implement [YouTube] badge (#5132)
* Implement [YouTube] badge

* Update production-hosting.md with account owner

* Add votes badge variant

* Add links to tests

* Switch to social badge style
2020-06-10 18:44:21 +02:00
Paul Melnikow
4582ea1c56 Avoid using racially linked language (#5190) 2020-06-09 02:28:53 +00:00
dependabot-preview[bot]
fc6390d62e Build(deps-dev): bump eslint-plugin-jsdoc from 24.0.6 to 25.4.1 (#5085)
* Build(deps-dev): bump eslint-plugin-jsdoc from 24.0.6 to 25.4.1

Bumps [eslint-plugin-jsdoc](https://github.com/gajus/eslint-plugin-jsdoc) from 24.0.6 to 25.4.1.
- [Release notes](https://github.com/gajus/eslint-plugin-jsdoc/releases)
- [Commits](https://github.com/gajus/eslint-plugin-jsdoc/compare/v24.0.6...v25.4.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* fix errors identified by new linting rules

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: chris48s <chris.shaw480@gmail.com>
Co-authored-by: repo-ranger[bot] <39074581+repo-ranger[bot]@users.noreply.github.com>
2020-05-17 15:00:21 +00:00
chris48s
13d75e0607 upgrade to prettier 2 (#5051)
* arrowParens: avoid
* remove trailingComma setting
2020-05-05 21:07:43 +01:00
Paul Melnikow
f0030a4025 Fill GitHub token pool on startup (#4987)
We're still seeing a few "Token pool is exhausted" errors at the moment the dynos are coming up. I'm wondering if this might help.

Ref #3771
2020-04-30 23:39:01 -04:00
Paul Melnikow
f9d5f57f49 Delay start until the server is ready (#4959)
Fixes #4958

Fixes, but only in Heroku, #3771, so let's keep that open.
2020-04-27 22:45:19 -04:00
Caleb Cartwright
4fe80bb150 Get data for [Discord] badges via OVH server proxies (#4956) 2020-04-27 09:39:57 -04:00
chris48s
3ba05cb184 📦 version 3 (#4756)
* Validate input to BadgeFactory.create() (#3875)

* validate input to create()

* remove deprecated properties (#3881)

* remove BadgeFactory class (#3884)

* Template literal templates (#4459)

- Remove use of the doT template library and move to generating SVG output using javascript template literals.
- Drop SVGO and mostly manually implement the optimisations.
- Add a bunch more tests

Co-authored-by: Paul Melnikow <github@paulmelnikow.com>

* drop raster support in package CLI (#4523)

* drop raster support in package CLI
* update docs

* rename gh-badges package to badge-maker

* rename gh-badges dir to badge-maker

* update relative imports and other refs to in parent dir

'gh-badges' --> 'badge-maker'

* update snyk service tests

This change is only tangentially related

We've used the shields repo as an example for these tests so
moving files around in our repo has a knock-on effect on them

* add missing type hints to dev style page

* write the changelog/migration guide for v3

* use extension in README CLI example

* update CLI help

whoops - missed this in #4523

* bump version

* update for self-hosting users

* README updates

* drop .format param from CLI, always output SVG

* Change text[] to label and message, Remove JSON output

- Change text[] to label and message
- Fix message only badge
- Remove JSON output format
- Update the docs

* update package-lock

* rename 'template' to 'style'

* handle invalid styles in coalesceBadge

* ensure makeBadge is passed a string for template in coalesceBadge()

issue #4925

* fix (logo/no label text/label color specified) case

issue #4926

* add example of (logo/no label text/label color specified) to style debug page

* update type defs

* padding fix for FTB style

Co-authored-by: Paul Melnikow <github@paulmelnikow.com>
2020-04-23 20:05:48 +01:00
Marcin Mielnicki
b64987d2dd Do not log errors in influx tests (#4931) 2020-04-22 16:58:11 -05:00
Marcin Mielnicki
e66cfa3c21 📈 PaaS-friendly metrics (#4874)
* prom-client JSON to InfluxDB line protocol converter

* Converts a metric with separate names

* prom-client JSON to InfluxDB line protocol (version 2) converter

* Server has instance id

* Read the instance id from an environment variable

* More unit tests for instance-metadata

* Log instance id

* Push influx metrics

* INSTANCE_ID with dyno metadata

* Prepare influx metrics in one place

* Influx metrics endpoint should return metrics

* More readable tests

* Env added to instance metadata

* hostname as an instance label value

* HEROKU_DYNO_ID as an instance id for heroku

* Instance env can be set by env variable

* HEROKU_APP_NAME as an instance env

* Log instance metadata as a JSON

* Typo fix

* Code refactoring in tests

* wait-for-expect dev dependency added

* Test for pushing metrics

* Test for pushing metrics

* Use basic authentication for pushing metrics

* intervalSeconds=2 for development env

* Using existing methods

* TODOs removed

* Schema for influx credentials

* Influx config removed from config files

* Require username and password when influx metrics are enabled

* Unused args removed

* pushing component should log errors

* Speed up tests

* should log error responses

* InstanceMetadata class replaces by simple object

* Influx metrics can be configuredd by env variables

* Use application label name instead of service

* Unused code removed

* Integration test for prom-client and converter

* metrics.influx.enabled configuration option added

* Improved influx configuration schema

* instanceMetadata validation

* Typo fix

* Default value for env

* metrics.infux.hostnameAsAInstanceId added

* should add hostname as an instance label when hostnameAsAInstanceId is enabled

* Default values for influx configuration

* flatMap is not available in Node.js 9.4

* Env vars removed from Procfile

* Better instance metadata values in tests

* Typo fix

* lodash.groupby added to prod dependencies

* Allow other keys in private config

* Missing test - should allow other private keys when influx metrics are enabled

* Missing test - should require private metrics config when influx configuration is enabled

* log.error instead of console.log

* metrics.influx.uri -> metrics.influx.url

* Unused arguments removed

* async removed

* promisify sendMetrics

* Allow to disable prometheus metrics

* Create test server with custom config

* 'metrics-influx' resource removed

* 'metrics-influx' resource removed

* Private config schema flattened out

* Extra code removed in Prometheus tests

* promisify moved outside of the class

* Do not throw errors from got in a specific test

* hostnameAliases added

* instanceIdFrom added

* instanceIdEnvVarName added

* envLabel added to schema

* instanceMetadata is not used by InfluxMetrics

* Instance metadata removed

* hostnameAsAnInstanceId removed

* A comment added

* waitForExpect removed

* Unused code removed
2020-04-19 20:03:00 +02:00
Pierre-Yves B
097bf87e41 Make it easier to benchmark and profile the code (#4780)
* Make it easier to benchmark and profile the code

* Remove unnecessary escape

* Clarify that the backend server is started without the frontend

* Add missing NODE_CONFIG_ENV environment variable

* Add error message when user has not included console.time statements

* Fix lint issue

* Handle multiple console.time statements

* Switch NODE_CONFIG_ENV to test

* Switch to const as variable never re-assigned
2020-03-22 20:45:53 +01:00
chris48s
d8831729cb Check request origin before sending credentials (#4729)
Co-authored-by: Caleb Cartwright <calebcartwright@users.noreply.github.com>
Co-authored-by: Paul Melnikow <github@paulmelnikow.com>
Co-authored-by: chris48s <chris48s@users.noreply.github.com>

Co-authored-by: Caleb Cartwright <calebcartwright@users.noreply.github.com>
Co-authored-by: Paul Melnikow <github@paulmelnikow.com>
Co-authored-by: chris48s <chris48s@users.noreply.github.com>
2020-03-04 20:42:27 +00:00
François Hodierne
0d8a2d5ca0 Update to eslint 6.8.0 [appveyor githubissuedetail packagist] (#4489)
* update to eslint 6.8.0 and related packages

* Fixes for no-prototype-builtins

* Updates for explicit-function-return-type

* Add ignores for no-explicit-any

* update to eslint 6.8.0 and related packages

* Fixes for no-prototype-builtins

* Updates for explicit-function-return-type

* Add ignores for no-explicit-any

* package: activate eslint-config-standard

* apply updated eslint configuration

* lint: apply eslint feedback after rebase

* Update lockfile

* Update lockfile

* Restore missing deps

* Update lockfile

* Re-add eslint-plugin-node

* Add eslint-plugin-standard and eslint-plugin-react-hooks

* Clean lint

Co-authored-by: Paul Melnikow <github@paulmelnikow.com>
2020-02-29 11:06:36 -06:00
Caleb Cartwright
27f1f5b792 fix: private config schema (#4689) 2020-02-26 17:32:21 -06:00
Marcin Mielnicki
22a85bfbcd A response size metric of an upstream service for dynamic badges like [dynamicyaml], (but not [jetbrains]) (#4434)
* Response size metric for all services

* Unused code removed

* Test for service response size metric

* All buckes of the service_response_bytes in a comment

* Register parameter in PrometheusMetrics is optional

* service response size metric enabled for dynamic badges

* Better test name

* JSDoc removed

* One import from one file

* Gather metrics in the background

* Revert saving response time metrics in the background
2020-01-04 17:48:24 +01:00
Hannes Nel
a583fa86c3 Changed configuration validation to allow using a named pipe for port. (#4392)
* Changed configuration validation to allow using a named pipe for port.

* Update core/server/server.js

Co-Authored-By: Caleb Cartwright <calebcartwright@users.noreply.github.com>

* Applied prettier.

* Update core/server/server.spec.js

Co-Authored-By: Caleb Cartwright <calebcartwright@users.noreply.github.com>

* Updated spec for port setting to allow only numbers or formatted named pipes.
2019-12-12 08:42:59 +00:00
dependabot-preview[bot]
0f332413df Build(deps-dev): bump prettier from 1.18.2 to 1.19.1 (#4328)
* Build(deps-dev): bump prettier from 1.18.2 to 1.19.1

Bumps [prettier](https://github.com/prettier/prettier) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/1.18.2...1.19.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Build(deps-dev): bump prettier from 1.18.2 to 1.19.1

Bumps [prettier](https://github.com/prettier/prettier) from 1.18.2 to 1.19.1.
- [Release notes](https://github.com/prettier/prettier/releases)
- [Changelog](https://github.com/prettier/prettier/blob/master/CHANGELOG.md)
- [Commits](https://github.com/prettier/prettier/compare/1.18.2...1.19.1)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* chore: apply new prettier formatting

* chore: try prettier, again
2019-11-15 19:33:40 -06:00
Caleb Cartwright
5f3a893eb4 refactor: revert cors header injection #4171 (#4253) 2019-10-26 22:29:56 +00:00
Tagan Hoyle
b8d0ec4238 Add CORS Header to every request (#4171)
* Try enable universal cors

* Move handle from registerErrorHandlers because it isn't and error handler

* Add test for cors headers

Also add link to issue
2019-10-13 18:51:15 -05:00
Andy Li
4b9053d2d5 add [twitch] user status badge (#3683)
* add /twitch/status/:user badge

* update comments

* use a proper schema for the Twitch API calls

* use a token to make Twitch api calls

* fix handling of rate-limit error and bad token error

* [twitch] get a token as soon as creating a Twitch service

* [twitch] start both requests to users and stream before awaiting

* [twitch] set a timeout to replace the token before it expires

* [twitch] use authHelper

* [twitch] skip tests when no credentials

* [twitch] add one more status test

* twitch: do not check whether a user exists
2019-09-09 16:41:32 -05:00
Paul Melnikow
926e837457 Exclude more in-use GitHub IPs from rate limiting and add metrics (#3950)
Fix issue where badges loaded through GitHub intermittently generated 502's on camo.

Ref https://github.com/badges/shields/issues/3874#issuecomment-527904731
2019-09-04 11:30:32 -04:00
Paul Melnikow
b7a29f20ef Add a response-time metric (#3948)
* Refactor existing metrics support into MetricHelper

This completes the refactor done at https://github.com/badges/shields/pull/3662#issuecomment-509011229 in anticipation of adding more metrics support, such as response size of an upstream service, or response time.

* Clean up

* Renames

* Add response time metrics

This adds around 30 new metrics to cover response times at a fairly granular level. We may be able to shrink the number of buckets with time, though I think using 30 metrics is probably okay given that I think may become our most important metric.

* Fix
2019-09-03 22:19:24 +00:00
Paul Melnikow
e33cb30a9a Fix regex match in notfound route (#3790)
Fixes #3789
2019-07-25 17:18:02 -05:00
Paul Melnikow
a120e5df9b Upgrade to new Sentry SDKs; fix error reporting detail (#3764) 2019-07-24 18:58:22 -05:00
Paul Melnikow
cfbd2c30df SVG by default (#3717)
Make cleaner badge URLs by omitting the `.svg` extension.

Closes #2674
2019-07-24 12:57:39 -05:00
Paul Melnikow
28b8836595 Unify the ESLint config (#3743)
Adding TypeScript to the frontend and a `.d.ts` file to `core` (see #3742) has multiplied out the different combinations of lint rules. ESLint has support for file-pattern-based overrides, which we've used in some places, but we've also maintained a separate eslintrc for `frontend/`.

This merges the config together, with the strategy of putting all the rules at the top level except where they conflict, and applying settings to exactly the files where they should apply.

This introduces a few new errors in the server but they are true positives – hoisting and lowercase class names – things we don't really need to be doing).
2019-07-22 15:06:38 -05:00