Removing broken links from lists #252

New Issue

GiteaMirror · 2025-11-06T15:10:19-06:00

GiteaMirror commented

2025-11-06 15:10:19 -06:00

Originally created by @basharovV on GitHub (Jun 23, 2020).

I've got a list of every dead or broken link in every awesome list, along with the error, which is probably useful to the list maintainers.

Here is the output from my cleanup script, which checks every URL in every awesome list:

Total urls: 47941  Valid urls: 41299 Invalid urls: 6642  HTTP errors: 2044  Other errors: 2406

That's 14% of all links being unavailable...

If you maintain an awesome list, feel free to find the file for your list and check if it has any dead links.

Originally created by @basharovV on GitHub (Jun 23, 2020). I've got a list of [every dead or broken link in every awesome list](https://github.com/basharovV/StumbleUponAwesome/tree/master/extension/data/broken-urls/awesome), along with the error, which is probably useful to the list maintainers. Here is the output from my cleanup script, which checks every URL in every awesome list: ``` Total urls: 47941 Valid urls: 41299 Invalid urls: 6642 HTTP errors: 2044 Other errors: 2406 ``` That's 14% of all links being unavailable... If you maintain an awesome list, feel free to find the file for your list and check if it has any dead links.

GiteaMirror commented

2025-11-06 15:10:20 -06:00

@bernardoduarte commented on GitHub (May 14, 2021):

Here's a tool that could help check if your lists links are alive: https://github.com/dkhamsing/awesome_bot

I use it on my awesome-version-managers list .travis.yml config for CI automation.

@bernardoduarte commented on GitHub (May 14, 2021): Here's a tool that could help check if your lists links are alive: https://github.com/dkhamsing/awesome_bot I use it on my [awesome-version-managers](https://github.com/bernardoduarte/awesome-version-managers) list [`.travis.yml`](https://github.com/bernardoduarte/awesome-version-managers/blob/main/.travis.yml) config for CI automation.

GiteaMirror commented

2025-11-06 15:10:20 -06:00

@riderx commented on GitHub (Nov 21, 2022):

in my repo i use this github action who check link and lint

name: Check Lint

on:
  workflow_dispatch:
  push:
    branches:
    - main
  schedule:
  # Run every first day at 00:00 AM every month.
    - cron: "0 0 * * *"
  pull_request:
    branches: [main]
    paths:
      - 'readme.md'

jobs:
  main:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0
      - name: Check links
        uses: gaurav-nelson/github-action-markdown-link-check@v1
        with:
          retry-after: 10
          use-quiet-mode: 'yes'
          use-verbose-mode: 'yes'
          config-file: '.github/workflows/markdown.links.config.json'
      - name: Awesome linter
        run: npx awesome-lint

@riderx commented on GitHub (Nov 21, 2022): in my repo i use this github action who check link and lint ``` name: Check Lint on: workflow_dispatch: push: branches: - main schedule: # Run every first day at 00:00 AM every month. - cron: "0 0 * * *" pull_request: branches: [main] paths: - 'readme.md' jobs: main: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 with: fetch-depth: 0 - name: Check links uses: gaurav-nelson/github-action-markdown-link-check@v1 with: retry-after: 10 use-quiet-mode: 'yes' use-verbose-mode: 'yes' config-file: '.github/workflows/markdown.links.config.json' - name: Awesome linter run: npx awesome-lint ```

GiteaMirror commented

2025-11-06 15:10:20 -06:00

@d-wasserman commented on GitHub (Nov 23, 2022):

This was really helpful @riderx. The only change I made was I run the awesome linter first, then the check links.

@d-wasserman commented on GitHub (Nov 23, 2022): This was really helpful @riderx. The only change I made was I run the awesome linter first, then the check links.

GiteaMirror commented

2025-11-06 15:10:20 -06:00

@GovLeena commented on GitHub (May 29, 2024):

🙋‍♀️

@GovLeena commented on GitHub (May 29, 2024): 🙋‍♀️

GiteaMirror commented

2025-11-06 15:10:21 -06:00

@koushik-drmgr commented on GitHub (Oct 2, 2024):

Hey use my Turbo C code for this..I hope this may helpful for you

#include <stdio.h>
#include <string.h>

// Simulated function to "check" if a URL is valid
int check_url(char *url) {
// In a real scenario, network operations would be done here.
// For now, simulate that any URL containing "valid" is good, others are broken.

if (strstr(url, "valid") != NULL) {
    return 1; // Valid URL
}
return 0; // Invalid URL

}

// Function to process URLs from a file and generate a report
void process_urls(const char *input_file, const char *output_file) {
FILE *infile, *outfile;
char url[200]; // Buffer to store each URL (assuming max 200 chars)
int status;

// Open the input file in read mode
infile = fopen(input_file, "r");
if (infile == NULL) {
    printf("Error: Could not open input file '%s'\n", input_file);
    return;
}

// Open the output file in write mode
outfile = fopen(output_file, "w");
if (outfile == NULL) {
    printf("Error: Could not open output file '%s'\n", output_file);
    fclose(infile);
    return;
}

// Write header to the output file
fprintf(outfile, "URL, Status\n");

// Read each URL from the input file and check its status
while (fgets(url, sizeof(url), infile) != NULL) {
    // Remove newline character from URL, if present
    if ((strlen(url) > 0) && (url[strlen(url) - 1] == '\n')) {
        url[strlen(url) - 1] = '\0';
    }

    // Simulate the URL check
    status = check_url(url);

    // Write the result to the output file
    if (status == 1) {
        fprintf(outfile, "%s, Valid\n", url);
    } else {
        fprintf(outfile, "%s, Invalid\n", url);
    }
}

// Close both input and output files
fclose(infile);
fclose(outfile);

// Notify the user that the process is complete
printf("URL check completed. Results saved to '%s'\n", output_file);

}

int main() {
// File paths (adjust as needed)
char input_file[] = "urls.txt"; // Input file containing URLs (one per line)
char output_file[] = "report.txt"; // Output file to store the results

// Display a welcome message
printf("===== URL Checker Program =====\n");
printf("Processing URLs from '%s'...\n", input_file);

// Process the URLs and generate the report
process_urls(input_file, output_file);

// Display completion message
printf("All done!\n");

return 0;

}

@koushik-drmgr commented on GitHub (Oct 2, 2024): Hey use my Turbo C code for this..I hope this may helpful for you #include <stdio.h> #include <string.h> // Simulated function to "check" if a URL is valid int check_url(char *url) { // In a real scenario, network operations would be done here. // For now, simulate that any URL containing "valid" is good, others are broken. if (strstr(url, "valid") != NULL) { return 1; // Valid URL } return 0; // Invalid URL } // Function to process URLs from a file and generate a report void process_urls(const char *input_file, const char *output_file) { FILE *infile, *outfile; char url[200]; // Buffer to store each URL (assuming max 200 chars) int status; // Open the input file in read mode infile = fopen(input_file, "r"); if (infile == NULL) { printf("Error: Could not open input file '%s'\n", input_file); return; } // Open the output file in write mode outfile = fopen(output_file, "w"); if (outfile == NULL) { printf("Error: Could not open output file '%s'\n", output_file); fclose(infile); return; } // Write header to the output file fprintf(outfile, "URL, Status\n"); // Read each URL from the input file and check its status while (fgets(url, sizeof(url), infile) != NULL) { // Remove newline character from URL, if present if ((strlen(url) > 0) && (url[strlen(url) - 1] == '\n')) { url[strlen(url) - 1] = '\0'; } // Simulate the URL check status = check_url(url); // Write the result to the output file if (status == 1) { fprintf(outfile, "%s, Valid\n", url); } else { fprintf(outfile, "%s, Invalid\n", url); } } // Close both input and output files fclose(infile); fclose(outfile); // Notify the user that the process is complete printf("URL check completed. Results saved to '%s'\n", output_file); } int main() { // File paths (adjust as needed) char input_file[] = "urls.txt"; // Input file containing URLs (one per line) char output_file[] = "report.txt"; // Output file to store the results // Display a welcome message printf("===== URL Checker Program =====\n"); printf("Processing URLs from '%s'...\n", input_file); // Process the URLs and generate the report process_urls(input_file, output_file); // Display completion message printf("All done!\n"); return 0; }

GiteaMirror commented

2025-11-06 15:10:21 -06:00

@good1400 commented on GitHub (Mar 2, 2025):

👍

@good1400 commented on GitHub (Mar 2, 2025): 👍

GiteaMirror commented

2025-11-06 15:10:21 -06:00

@Catgotyourtongue commented on GitHub (Aug 31, 2025):

Jailbreak @basharovV

@Catgotyourtongue commented on GitHub (Aug 31, 2025): - Jailbreak @basharovV

GiteaMirror commented

2025-11-06 15:10:21 -06:00

@Angel-Mathew007 commented on GitHub (Sep 4, 2025):

Comment suggestion:

Hi @basharovV and everyone,
Thanks for this valuable cleanup effort! I’d love to help maintainers by contributing to fixing broken links in some Awesome lists.
Also, if anyone needs assistance setting up automated link checking workflows or tools like awesome_bot, feel free to reach out. Happy to help!

@Angel-Mathew007 commented on GitHub (Sep 4, 2025): Comment suggestion: Hi @basharovV and everyone, Thanks for this valuable cleanup effort! I’d love to help maintainers by contributing to fixing broken links in some Awesome lists. Also, if anyone needs assistance setting up automated link checking workflows or tools like awesome_bot, feel free to reach out. Happy to help!