This commit disables the adding of new Nitter sources, because Nitter is
deprecated 😞 and adding new sources is not working anymore. Nitter
sources are also not updated anymore, because the update for Nitter
sources isn't working as well. To ensure that the Nitter sources are not
updated anymore the sources with type `nitter` are skipped in the
scheduler.
The Nitter source isn't removed completly, because we would also have to
cleanup the database and their might be items, which are bookmarked by
users.
See https://github.com/zedeus/nitter/issues/1155#issuecomment-1913361757
Reddit and Nitter are heavlily rate limited from time to time, to not
effect our premium users, by these limits, we decided to increase the
interval in which the Reddit and Nitter sources are updated for users
which are on the free tier.
The log level is now configurable, via the `FEEDDECK_LOG_LEVEL`
environment variable. The environment variable can have the following
values: `debug`, `info`, `warning` or `error`.
With this change we also only log the response when getting and parsing
the feed fails when the log level is set to `debug`, also when the
actual message is still an error. This should reduce the noice in the
logs a lot and allows us to specially turn this on while debugging.
It is now possible to add and update sources via client side scraping.
For that a new edge function `add-or-update-source-v1` was added and the
old `add-source-v1` function was deprecated.
The new function accepts a new `feedData` field, which can contain the
feed for a source. If the field is provided we will not try to get the
feed for a source within our edge function and instead use the provided
data.
Currently this function is only used to add a Reddit source. Later we
plan to extend it for other sources and want to use it to update source
via the app, when the source provider makes heavy use of rate limiting.
This commit refactors the existing tools, by moving the tools logic to a
new `tools.ts` file, so that the main `cmd.ts` file remains clear.
Besides that we also add a new tool `get-feed` which can be used to run
the `getFeed` function from the command line. The function is called
with a source and returns the generated source and items, as they are
saved in the database by the `add-source-v1` Supabase edge function.
This commit improve the error handling for the edge function to add a
new source and the worker, so that we get more insights why a request
fails. A user will now also get a more detailed error why a source could
not be added.
