github-starred/feeddeck
2
0
Fork 0
mirror of https://github.com/feeddeck/feeddeck.git synced 2026-03-09 07:02:01 -05:00
Code Issues 5 Packages Projects Releases 8 Wiki Activity

[PR #118] [MERGED] [core] Add Client Side Scraping of Sources #119

New Issue
Closed
opened 2025-10-31 16:57:42 -05:00 by GiteaMirror · 0 comments
GiteaMirror commented 2025-10-31 16:57:42 -05:00
Owner
Copy Link

📋 Pull Request Information

Original PR: https://github.com/feeddeck/feeddeck/pull/118
Author: @ricoberger
Created: 1/30/2024
Status: ✅ Merged
Merged: 1/30/2024
Merged by: @ricoberger

Base: main ← Head: core-add-client-side-scraping-of-sources

📝 Commits (1)

  • a8f590c [core] Add Client Side Scraping of Sources

📊 Changes

39 files changed (+583 additions, -54 deletions)

View changed files

📝 .github/workflows/continuous-delivery.yaml (+2 -0)
📝 CONTRIBUTING.md (+1 -0)
📝 app/lib/repositories/app_repository.dart (+22 -10)
➕ app/lib/utils/get_feed.dart (+54 -0)
📝 app/lib/widgets/source/add/add_source_reddit.dart (+19 -0)
📝 app/pubspec.lock (+2 -2)
📝 app/pubspec.yaml (+1 -0)
📝 supabase/functions/_cmd/scheduler/scheduler.ts (+0 -1)
📝 supabase/functions/_shared/feed/feed.ts (+76 -17)
📝 supabase/functions/_shared/feed/github.ts (+3 -2)
📝 supabase/functions/_shared/feed/googlenews.ts (+2 -0)
📝 supabase/functions/_shared/feed/googlenews_test.ts (+2 -0)
📝 supabase/functions/_shared/feed/lemmy.ts (+6 -1)
📝 supabase/functions/_shared/feed/lemmy_test.ts (+3 -0)
📝 supabase/functions/_shared/feed/mastodon.ts (+6 -1)
📝 supabase/functions/_shared/feed/mastodon_test.ts (+2 -0)
📝 supabase/functions/_shared/feed/medium.ts (+6 -1)
📝 supabase/functions/_shared/feed/medium_test.ts (+2 -0)
📝 supabase/functions/_shared/feed/nitter.ts (+2 -0)
📝 supabase/functions/_shared/feed/nitter_test.ts (+2 -0)

...and 19 more files

📄 Description

It is now possible to add and update sources via client side scraping. For that a new edge function add-or-update-source-v1 was added and the old add-source-v1 function was deprecated.

The new function accepts a new feedData field, which can contain the feed for a source. If the field is provided we will not try to get the feed for a source within our edge function and instead use the provided data.

Currently this function is only used to add a Reddit source. Later we plan to extend it for other sources and want to use it to update source via the app, when the source provider makes heavy use of rate limiting.

🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.

## 📋 Pull Request Information **Original PR:** https://github.com/feeddeck/feeddeck/pull/118 **Author:** [@ricoberger](https://github.com/ricoberger) **Created:** 1/30/2024 **Status:** ✅ Merged **Merged:** 1/30/2024 **Merged by:** [@ricoberger](https://github.com/ricoberger) **Base:** `main` ← **Head:** `core-add-client-side-scraping-of-sources` --- ### 📝 Commits (1) - [`a8f590c`](https://github.com/feeddeck/feeddeck/commit/a8f590c4383b29d9d5e9a6b6decb1af3520fc2a3) [core] Add Client Side Scraping of Sources ### 📊 Changes **39 files changed** (+583 additions, -54 deletions) <details> <summary>View changed files</summary> 📝 `.github/workflows/continuous-delivery.yaml` (+2 -0) 📝 `CONTRIBUTING.md` (+1 -0) 📝 `app/lib/repositories/app_repository.dart` (+22 -10) ➕ `app/lib/utils/get_feed.dart` (+54 -0) 📝 `app/lib/widgets/source/add/add_source_reddit.dart` (+19 -0) 📝 `app/pubspec.lock` (+2 -2) 📝 `app/pubspec.yaml` (+1 -0) 📝 `supabase/functions/_cmd/scheduler/scheduler.ts` (+0 -1) 📝 `supabase/functions/_shared/feed/feed.ts` (+76 -17) 📝 `supabase/functions/_shared/feed/github.ts` (+3 -2) 📝 `supabase/functions/_shared/feed/googlenews.ts` (+2 -0) 📝 `supabase/functions/_shared/feed/googlenews_test.ts` (+2 -0) 📝 `supabase/functions/_shared/feed/lemmy.ts` (+6 -1) 📝 `supabase/functions/_shared/feed/lemmy_test.ts` (+3 -0) 📝 `supabase/functions/_shared/feed/mastodon.ts` (+6 -1) 📝 `supabase/functions/_shared/feed/mastodon_test.ts` (+2 -0) 📝 `supabase/functions/_shared/feed/medium.ts` (+6 -1) 📝 `supabase/functions/_shared/feed/medium_test.ts` (+2 -0) 📝 `supabase/functions/_shared/feed/nitter.ts` (+2 -0) 📝 `supabase/functions/_shared/feed/nitter_test.ts` (+2 -0) _...and 19 more files_ </details> ### 📄 Description It is now possible to add and update sources via client side scraping. For that a new edge function `add-or-update-source-v1` was added and the old `add-source-v1` function was deprecated. The new function accepts a new `feedData` field, which can contain the feed for a source. If the field is provided we will not try to get the feed for a source within our edge function and instead use the provided data. Currently this function is only used to add a Reddit source. Later we plan to extend it for other sources and want to use it to update source via the app, when the source provider makes heavy use of rate limiting. <!-- Keep PR title verbose enough and add prefix telling about what source it touches e.g "[rss] Add feature xyz" or if the the PR is not realated to a source use "[core]", e.g. "[core] Fix xyz". If you add a breaking change within your PR you should add ":warning:" to the title, e.g. ":warning: [core] My breaking change" --> <!-- Description of what have been changed. Please also reference an issue, when available. --> --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>
GiteaMirror added the pull-request label 2025-10-31 16:57:42 -05:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
enhancement
pull-request
Mirrored from GitHub Pull Request
No Label pull-request
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/feeddeck#119
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?