It is now possible to add and update sources via client side scraping.
For that a new edge function `add-or-update-source-v1` was added and the
old `add-source-v1` function was deprecated.
The new function accepts a new `feedData` field, which can contain the
feed for a source. If the field is provided we will not try to get the
feed for a source within our edge function and instead use the provided
data.
Currently this function is only used to add a Reddit source. Later we
plan to extend it for other sources and want to use it to update source
via the app, when the source provider makes heavy use of rate limiting.
This commit improve the error handling for the edge function to add a
new source and the worker, so that we get more insights why a request
fails. A user will now also get a more detailed error why a source could
not be added.
This commit updates all used Deno modules to their latest version.
Since some of the used modules / functions were deprecated we had to
adjust our encrypt / descrypt functions and the generation of the source
and item ids, where we have to use a new md5 function.
This commit adds tests for all available sources.
This commit also fixes the parsing of Atom feeds for the RSS source,
where the `dc:date` field must be used for the `publishedAt` field.
For the item ingestion logic we are now also looking at the published
date for all entries in a RSS feed. If the published date is older then
the last update date of a source the item will be skipped. This way we
should reduce the number of duplicated items, where an entry was updated
in the RSS feed, so that it had a new id and was written again to the
database and not only updated.
Improve the media handling within the app. We do not save the media
files for items to the Supabase storage anymore. The source icons are
now only saved in the Supabase storage, the usage of an url as source
icon is not possible anymore.
The media files for items are directly retrieved from the corresponding
url or for the web version from the "image-proxy-v1" Supabase function.
If the image is retireved from the Supabase function we cache the image
in the browser via the cache control headers. If the image is directly
retrieved from it's url it's cached by the "CachedNetworkImage" widget.
