It is now possible to add and update sources via client side scraping. For that a new edge function add-or-update-source-v1 was added and the old add-source-v1 function was deprecated.
The new function accepts a new feedData field, which can contain the feed for a source. If the field is provided we will not try to get the feed for a source within our edge function and instead use the provided data.
Currently this function is only used to add a Reddit source. Later we plan to extend it for other sources and want to use it to update source via the app, when the source provider makes heavy use of rate limiting.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.
## 📋 Pull Request Information
**Original PR:** https://github.com/feeddeck/feeddeck/pull/118
**Author:** [@ricoberger](https://github.com/ricoberger)
**Created:** 1/30/2024
**Status:** ✅ Merged
**Merged:** 1/30/2024
**Merged by:** [@ricoberger](https://github.com/ricoberger)
**Base:** `main` ← **Head:** `core-add-client-side-scraping-of-sources`
---
### 📝 Commits (1)
- [`a8f590c`](https://github.com/feeddeck/feeddeck/commit/a8f590c4383b29d9d5e9a6b6decb1af3520fc2a3) [core] Add Client Side Scraping of Sources
### 📊 Changes
**39 files changed** (+583 additions, -54 deletions)
<details>
<summary>View changed files</summary>
📝 `.github/workflows/continuous-delivery.yaml` (+2 -0)
📝 `CONTRIBUTING.md` (+1 -0)
📝 `app/lib/repositories/app_repository.dart` (+22 -10)
➕ `app/lib/utils/get_feed.dart` (+54 -0)
📝 `app/lib/widgets/source/add/add_source_reddit.dart` (+19 -0)
📝 `app/pubspec.lock` (+2 -2)
📝 `app/pubspec.yaml` (+1 -0)
📝 `supabase/functions/_cmd/scheduler/scheduler.ts` (+0 -1)
📝 `supabase/functions/_shared/feed/feed.ts` (+76 -17)
📝 `supabase/functions/_shared/feed/github.ts` (+3 -2)
📝 `supabase/functions/_shared/feed/googlenews.ts` (+2 -0)
📝 `supabase/functions/_shared/feed/googlenews_test.ts` (+2 -0)
📝 `supabase/functions/_shared/feed/lemmy.ts` (+6 -1)
📝 `supabase/functions/_shared/feed/lemmy_test.ts` (+3 -0)
📝 `supabase/functions/_shared/feed/mastodon.ts` (+6 -1)
📝 `supabase/functions/_shared/feed/mastodon_test.ts` (+2 -0)
📝 `supabase/functions/_shared/feed/medium.ts` (+6 -1)
📝 `supabase/functions/_shared/feed/medium_test.ts` (+2 -0)
📝 `supabase/functions/_shared/feed/nitter.ts` (+2 -0)
📝 `supabase/functions/_shared/feed/nitter_test.ts` (+2 -0)
_...and 19 more files_
</details>
### 📄 Description
It is now possible to add and update sources via client side scraping. For that a new edge function `add-or-update-source-v1` was added and the old `add-source-v1` function was deprecated.
The new function accepts a new `feedData` field, which can contain the feed for a source. If the field is provided we will not try to get the feed for a source within our edge function and instead use the provided data.
Currently this function is only used to add a Reddit source. Later we plan to extend it for other sources and want to use it to update source via the app, when the source provider makes heavy use of rate limiting.
<!--
Keep PR title verbose enough and add prefix telling about what source it touches e.g "[rss] Add feature xyz" or if the
the PR is not realated to a source use "[core]", e.g. "[core] Fix xyz".
If you add a breaking change within your PR you should add ":warning:" to the title,
e.g. ":warning: [core] My breaking change"
-->
<!--
Description of what have been changed. Please also reference an issue, when available.
-->
---
<sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
📋 Pull Request Information
Original PR: https://github.com/feeddeck/feeddeck/pull/118
Author: @ricoberger
Created: 1/30/2024
Status: ✅ Merged
Merged: 1/30/2024
Merged by: @ricoberger
Base:
main← Head:core-add-client-side-scraping-of-sources📝 Commits (1)
a8f590c[core] Add Client Side Scraping of Sources📊 Changes
39 files changed (+583 additions, -54 deletions)
View changed files
📝
.github/workflows/continuous-delivery.yaml(+2 -0)📝
CONTRIBUTING.md(+1 -0)📝
app/lib/repositories/app_repository.dart(+22 -10)➕
app/lib/utils/get_feed.dart(+54 -0)📝
app/lib/widgets/source/add/add_source_reddit.dart(+19 -0)📝
app/pubspec.lock(+2 -2)📝
app/pubspec.yaml(+1 -0)📝
supabase/functions/_cmd/scheduler/scheduler.ts(+0 -1)📝
supabase/functions/_shared/feed/feed.ts(+76 -17)📝
supabase/functions/_shared/feed/github.ts(+3 -2)📝
supabase/functions/_shared/feed/googlenews.ts(+2 -0)📝
supabase/functions/_shared/feed/googlenews_test.ts(+2 -0)📝
supabase/functions/_shared/feed/lemmy.ts(+6 -1)📝
supabase/functions/_shared/feed/lemmy_test.ts(+3 -0)📝
supabase/functions/_shared/feed/mastodon.ts(+6 -1)📝
supabase/functions/_shared/feed/mastodon_test.ts(+2 -0)📝
supabase/functions/_shared/feed/medium.ts(+6 -1)📝
supabase/functions/_shared/feed/medium_test.ts(+2 -0)📝
supabase/functions/_shared/feed/nitter.ts(+2 -0)📝
supabase/functions/_shared/feed/nitter_test.ts(+2 -0)...and 19 more files
📄 Description
It is now possible to add and update sources via client side scraping. For that a new edge function
add-or-update-source-v1was added and the oldadd-source-v1function was deprecated.The new function accepts a new
feedDatafield, which can contain the feed for a source. If the field is provided we will not try to get the feed for a source within our edge function and instead use the provided data.Currently this function is only used to add a Reddit source. Later we plan to extend it for other sources and want to use it to update source via the app, when the source provider makes heavy use of rate limiting.
🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.