It is now possible to add and update sources via client side scraping.
For that a new edge function `add-or-update-source-v1` was added and the
old `add-source-v1` function was deprecated.
The new function accepts a new `feedData` field, which can contain the
feed for a source. If the field is provided we will not try to get the
feed for a source within our edge function and instead use the provided
data.
Currently this function is only used to add a Reddit source. Later we
plan to extend it for other sources and want to use it to update source
via the app, when the source provider makes heavy use of rate limiting.
This commit refactors the existing tools, by moving the tools logic to a
new `tools.ts` file, so that the main `cmd.ts` file remains clear.
Besides that we also add a new tool `get-feed` which can be used to run
the `getFeed` function from the command line. The function is called
with a source and returns the generated source and items, as they are
saved in the database by the `add-source-v1` Supabase edge function.
This commit improve the error handling for the edge function to add a
new source and the worker, so that we get more insights why a request
fails. A user will now also get a more detailed error why a source could
not be added.
This commit adds tests for all available sources.
This commit also fixes the parsing of Atom feeds for the RSS source,
where the `dc:date` field must be used for the `publishedAt` field.
Users can now provide the URL of a website instead of the url of a RSS
feed via the input field for the RSS source.
This is possible because we are now trying to get and parse the RSS feed
for the provided url as usual, but if this operation fails, we try to
parse the text as html, so that we can check if it contains a
"<link type="application/rss+xml" href="RSS_FEED_URL">" tag. If this is
the case we are using this value to try to get and parse the RSS feed
again. If it fails again we are returning an error as before.
This commit improves / simplifies the media handling within the app.
Until now we always had to provide the type of the media file (item
media / source icon) when we wanted to display it. This is now not
necessary anymore. Instead we always display the image from it's
original path when the url starts with "http://" or "https://".
Additionally we also check the platform to proxy the image request on
the web. If the image doesn't start with "http://" or "https://" we
always try to display the image from the Supabase storage.
For this we also adjusted the corresponding Deno function, so that we
check if the image starts with "http://" or "https://" before we upload
it to the Supabase storage. If this is the case we upload the source
icon to the Supabase storage. If the upload fails, we will use the
original path of the icon.
Last but not least this commit also introduces our own
"CachedNetworkImage" widget, which wraps the original
"CachedNetworkImage" widget. Our own widget will ensure that we use the
correct url for the image request, so that we do not have to use this
function all over the app anymore. Later this widget can also be used to
introduce our own cache manager.
Improve the media handling within the app. We do not save the media
files for items to the Supabase storage anymore. The source icons are
now only saved in the Supabase storage, the usage of an url as source
icon is not possible anymore.
The media files for items are directly retrieved from the corresponding
url or for the web version from the "image-proxy-v1" Supabase function.
If the image is retireved from the Supabase function we cache the image
in the browser via the cache control headers. If the image is directly
retrieved from it's url it's cached by the "CachedNetworkImage" widget.
