Originally created by @awesome-bot on GitHub (Dec 18, 2015).
Original GitHub issue: https://github.com/awesomedata/awesome-public-datasets/issues/130
Hello, I wrote a tool that can validate README links (valid URLs, not duplicate). It can be run when someone submits a pull request.
It is currently being used by
- https://github.com/vsouza/awesome-ios
- https://github.com/matteocrippa/awesome-swift
- https://github.com/dkhamsing/open-source-ios-apps
Examples
- https://travis-ci.org/matteocrippa/awesome-swift/builds/96526196 ok ✅
- https://travis-ci.org/matteocrippa/awesome-swift/builds/96722421 link redirected / rename 🔴
- https://travis-ci.org/dkhamsing/open-source-ios-apps/builds/96763135 bad link / project deleted 🔴
- https://travis-ci.org/dkhamsing/open-source-ios-apps/builds/95754715 dupe 🔴
If you are interested, connect this repo to https://travis-ci.org/ and add a [`.travis.yml`](https://github.com/dkhamsing/open-source-ios-apps/blob/master/.travis.yml) file to the project.
See https://github.com/dkhamsing/awesome_bot for options, more information
Feel free to leave a comment :smile:
<!-- gh-comment-id:166504714 -->
@caesar0301 commented on GitHub (Dec 22, 2015):
Thanks @awesome-bot ! I also added it to my [anther project pcaptools](https://github.com/caesar0301/awesome-pcaptools).
Cool!
I reviewed the results and there are some issues.. Some links may have to be white listed, let me know if you have any questions
<!-- gh-comment-id:166525514 -->
@awesome-bot commented on GitHub (Dec 22, 2015):
Cool!
I reviewed the results and there are some issues.. Some links may have to be white listed, let me know if you have any questions
I added some links with issues into the white list. But I found the bot returned inconsistent results when I run travis multiple times. And most are failure. Some links reported as failures are accessible when I check them manually. I donnt know why but am confused.
<!-- gh-comment-id:166890029 -->
@caesar0301 commented on GitHub (Dec 23, 2015):
I added some links with issues into the white list. But I found the bot returned inconsistent results when I run travis multiple times. And most are failure. Some links reported as failures are accessible when I check them manually. I donnt know why but am confused.
Other ones should be white listed if you know they do work but somehow the bot is misreporting them
Does that make sense?
<!-- gh-comment-id:167044407 -->
@awesome-bot commented on GitHub (Dec 24, 2015):
So I am reviewing https://travis-ci.org/caesar0301/awesome-public-datasets/builds/98612497 which is actually passing :white_check_mark:
Is that not ok?
Sites that are returning 404 should actually be removed from the README
1. 404 http://lib.stat.cmu.edu/datasets/
2. 404 http://archive.org/details/2011-05-calufa-twitter-sql
3. 404 http://www.datawrangling.com/some-datasets-available-on-the-web
4. 404 http://www.stats4stem.org/data-sets.html
Other ones should be white listed if you know they do work but somehow the bot is misreporting them
Does that make sense?
Thanks for your checking. I still can not explain my previous observations, maybe resulting from instability of networks of Travis services. I will keep using awesome_bot and feed back to you if there are any other problems.
I think its better to keep these 404 links in the list, because I am not sure they are removed permanently or temporally. Even more, others may find alternative sources for these 404 data.
<!-- gh-comment-id:167079002 -->
@caesar0301 commented on GitHub (Dec 24, 2015):
Thanks for your checking. I still can not explain my previous observations, maybe resulting from instability of networks of Travis services. I will keep using awesome_bot and feed back to you if there are any other problems.
I think its better to keep these 404 links in the list, because I am not sure they are removed permanently or temporally. Even more, others may find alternative sources for these 404 data.
<!-- gh-comment-id:167116577 -->
@awesome-bot commented on GitHub (Dec 24, 2015):
Cool
FYI, I think 404 or permanent but yeah maybe the projects still exist, just at a different URL.. https://en.m.wikipedia.org/wiki/HTTP_404
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @awesome-bot on GitHub (Dec 18, 2015).
Original GitHub issue: https://github.com/awesomedata/awesome-public-datasets/issues/130
Hello, I wrote a tool that can validate README links (valid URLs, not duplicate). It can be run when someone submits a pull request.
It is currently being used by
Examples
If you are interested, connect this repo to https://travis-ci.org/ and add a
.travis.ymlfile to the project.See https://github.com/dkhamsing/awesome_bot for options, more information
Feel free to leave a comment 😄
@caesar0301 commented on GitHub (Dec 22, 2015):
Thanks @awesome-bot ! I also added it to my anther project pcaptools.
@awesome-bot commented on GitHub (Dec 22, 2015):
Cool!
I reviewed the results and there are some issues.. Some links may have to be white listed, let me know if you have any questions
@caesar0301 commented on GitHub (Dec 23, 2015):
I added some links with issues into the white list. But I found the bot returned inconsistent results when I run travis multiple times. And most are failure. Some links reported as failures are accessible when I check them manually. I donnt know why but am confused.
@awesome-bot commented on GitHub (Dec 24, 2015):
Ok let me take a look
@awesome-bot commented on GitHub (Dec 24, 2015):
So I am reviewing https://travis-ci.org/caesar0301/awesome-public-datasets/builds/98612497 which is actually passing ✅
Is that not ok?
Sites that are returning 404 should actually be removed from the README
Other ones should be white listed if you know they do work but somehow the bot is misreporting them
Does that make sense?
@caesar0301 commented on GitHub (Dec 24, 2015):
Thanks for your checking. I still can not explain my previous observations, maybe resulting from instability of networks of Travis services. I will keep using awesome_bot and feed back to you if there are any other problems.
I think its better to keep these 404 links in the list, because I am not sure they are removed permanently or temporally. Even more, others may find alternative sources for these 404 data.
@awesome-bot commented on GitHub (Dec 24, 2015):
Cool
FYI, I think 404 or permanent but yeah maybe the projects still exist, just at a different URL.. https://en.m.wikipedia.org/wiki/HTTP_404