- Set `search_path` for all database functions, as this is recommended
by the Supabase security advisor
- Fix `items_delete` functions: The function was never working, which
caused that we saved to many items for a source, which slowed down our
performance
- Revoke execution rights for users for the `sources_delete_files` and
`items_delete` functions
- Add E2E test, to check if users can call the database functions
Until now we had a `find` command to delete existing edge functions
before copying them for the self hosting set up. This command is now
removed, because we also moved the `main` function to the existing
functions directory and just not push it as function to Supabase.
Instead of using an `import_map.json` file to define the versions for
dependencies, they are now defined directly within the import. Since the
`import_map.json` file should not be used anymore and instead a
`deno.json` file per function should be used, we decided to define them
directly with the code. The overhead compared to a `deno.json` file per
function shouldn't be that large and it makes using functions in a
self-hosted setup easier.
It is now possible to re-order sources in a column by dragging them into
a new position. To achieve this a new `position` field was added to the
`sources` column, which contains the index of the source in the column.
We also added a new `updateSourcePositions` function, which is used to
sort the sources locally and update `position` field in the database
afterwards. The dragging is handled via a `ReorderableListView` widget.
Last but not least the selection of sources from the database was
changed, so order them by the `position` column and if the column is
`null` by the `createdAt` data as it was before.
Until now, we always checked the `description` field of an item first to
get our description value. If the `description` field is present we
directly used it and never checked the `content` field.
Now we are checking the `content` field first and afterwards the
`description` field. This way by default the `content` field is used as
our description. This should be more common to get the full article text
when both fields are present in an RSS feed.
When both fields are present and the `description` field is longer then
the `content` field, we will still prefer the `description` field.
* Update Deno Version to 1.45.2
Update Deno Version in the Docker image to version 1.45.2 and update the
`@supabase/supabase-js` dependency specified in the `import_map.json`
file.
**NOTES:**
- The current recommended approach for managing dependencies is using a
`deno.json` file, which is currently not working locally and returns
the following error:
```
serving the request with supabase/functions/add-or-update-source-v1
worker boot error: failed to create the graph: Relative import path "@supabase/supabase-js" not prefixed with / or ./ or ../
at file:///Users/ricoberger/Documents/GitHub/feeddeck/feeddeck/supabase/functions/add-or-update-source-v1/index.ts:1:30
worker boot error: failed to create the graph: Relative import path "@supabase/supabase-js" not prefixed with / or ./ or ../
at file:///Users/ricoberger/Documents/GitHub/feeddeck/feeddeck/supabase/functions/add-or-update-source-v1/index.ts:1:30
InvalidWorkerCreation: worker boot error: failed to create the graph: Relative import path "@supabase/supabase-js" not prefixed with / or ./ or ../
at file:///Users/ricoberger/Documents/GitHub/feeddeck/feeddeck/supabase/functions/add-or-update-source-v1/index.ts:1:30
at async UserWorker.create (ext:sb_user_workers/user_workers.js:139:15)
at async Object.handler (file:///root/index.ts:157:22)
at async respond (ext:sb_core_main_js/js/http.js:197:14) {
name: "InvalidWorkerCreation"
}
```
- When using Deno v2 the dependencies via the `deno.json` file are
working. To enable Deno v2 the following must be set in the
`config.toml` file:
```
[edge_runtime]
deno_version = 2
```
- Deno v2 is currently only supported locally, see
https://supabase.com/blog/supabase-edge-functions-deploy-dashboard-deno-2-1#deno-21-preview
- Once Deno v2 is supported in the hosted platform, we should switch
from the `import_map.json` to `deno.json` to manage dependencies and
update the existing dependencies.
* Fix Errors
- Replace `err.toString()` with just `err` when logging errors.
- Add new `blobToFile` function, to upload files to the Supabase
storage, otherwise we receive the following error (https://github.com/feeddeck/feeddeck/actions/runs/14546547197/job/40812707871):
```
error: TS2345 [ERROR]: Argument of type 'Blob' is not assignable to parameter of type 'FileBody'.
Type 'Blob' is missing the following properties from type 'File': lastModified, name, webkitRelativePath
file,
~~~~
at file:///home/runner/work/feeddeck/feeddeck/supabase/functions/_shared/feed/utils/uploadFile.ts:66:9
```
This commit disables the adding of new Nitter sources, because Nitter is
deprecated 😞 and adding new sources is not working anymore. Nitter
sources are also not updated anymore, because the update for Nitter
sources isn't working as well. To ensure that the Nitter sources are not
updated anymore the sources with type `nitter` are skipped in the
scheduler.
The Nitter source isn't removed completly, because we would also have to
cleanup the database and their might be items, which are bookmarked by
users.
See https://github.com/zedeus/nitter/issues/1155#issuecomment-1913361757
X (Twitter) was never supported and the code was never used to add a
source. We kept it in the hope that it will be possible again to use X
within the app. Since this isn't the case it is time to remove the code.
Until now it could happen that the items for a column could not be
retrieved from the database, because of the set query timeout. In this
case we received the following error:
```
PostgrestException(message: canceling statement due to statement timeout, code: 57014, details: Internal Server Error, hint: null)
```
To fix this issue we added an `items_columnId_idx` index, so that the
items for a column are retrieved fast.
We also added some other useful indexes to improve the overall
performance of our queries.
Last but not least we also added an index to the `userId` columns as it
is recommended by Supabase to improve the performance of our RLS
policies.
A 4chan source can now also be added via the RSS source type in the
Flutter app. If a user provides the RSS feed of an 4chan board, we check
if the url is related to 4chan and if this is the case, we use the 4chan
instead of the RSS source type, similar to how it is handled for other
sources like Medium, Reddit, etc.
This commit adds support for 4chan. This means that 4chan can be
selected as a new source type. When the 4chan source is selected a user
can select a board from which he wants to get the RSS feed.
Reddit and Nitter are heavlily rate limited from time to time, to not
effect our premium users, by these limits, we decided to increase the
interval in which the Reddit and Nitter sources are updated for users
which are on the free tier.
We do not remove HTML tags from the description of a RSS feed item
anymore, because:
- This didn't worked for all feeds, because we first removed the tags
and then unescape the data, which didn't make sense
- We render the HTML to Makrdown in the frontend so that the description
can contain HTML tags and is still properly rendered and in some case
better rendered then before
The log level is now configurable, via the `FEEDDECK_LOG_LEVEL`
environment variable. The environment variable can have the following
values: `debug`, `info`, `warning` or `error`.
With this change we also only log the response when getting and parsing
the feed fails when the log level is set to `debug`, also when the
actual message is still an error. This should reduce the noice in the
logs a lot and allows us to specially turn this on while debugging.
It is now possible to add and update sources via client side scraping.
For that a new edge function `add-or-update-source-v1` was added and the
old `add-source-v1` function was deprecated.
The new function accepts a new `feedData` field, which can contain the
feed for a source. If the field is provided we will not try to get the
feed for a source within our edge function and instead use the provided
data.
Currently this function is only used to add a Reddit source. Later we
plan to extend it for other sources and want to use it to update source
via the app, when the source provider makes heavy use of rate limiting.
This commit refactors the existing tools, by moving the tools logic to a
new `tools.ts` file, so that the main `cmd.ts` file remains clear.
Besides that we also add a new tool `get-feed` which can be used to run
the `getFeed` function from the command line. The function is called
with a source and returns the generated source and items, as they are
saved in the database by the `add-source-v1` Supabase edge function.
This commit improve the error handling for the edge function to add a
new source and the worker, so that we get more insights why a request
fails. A user will now also get a more detailed error why a source could
not be added.
If an RSS feed contains a video within the `attachments` field, the video
will now be added to the `options` field of the item. In the Flutter
code we then check if the video field is present in the options and show
the video instead of an image in the details view of the item.
This commit updates all used Deno modules to their latest version.
Since some of the used modules / functions were deprecated we had to
adjust our encrypt / descrypt functions and the generation of the source
and item ids, where we have to use a new md5 function.
Until now we only checked if a website contained a RSS feed which can be
used for FeedDeck. Now we are also checking if the website contains a
Atom or RDF feed when no RSS feed was found.
For this we are checking `link` tag with the `type="application/atom+xml"`
attribute or a link tag with the `type="application/rdf+xml"` attribute.
This commit adds tests for all available sources.
This commit also fixes the parsing of Atom feeds for the RSS source,
where the `dc:date` field must be used for the `publishedAt` field.
This commit uses the "Continuous Integration" GitHub Action, to run
tests for the Deno code, which is used by the Supabase functions and our
Docker containers.
This commit also adds a first test so that the `deno test` command does
not fail.
This commit adds support to add Lemmy RSS feeds to FeedDeck. A user can
provide the url of an Lemmy instance, the url of a community or of an
user.
The special thing of the Lemmy source in opposite to the normal RSS
source is, that we parse the provided link form a feed item, to check if
it contains a image, video or YouTube url, to apply some special
formatting.
