Hi, dear all
First, thank all authors of awesome list, it makes the world better.
I don't know that if there are similar projects for these.
I am trying to exact awesome lists into JSON in my leisure time(still working on it) and build a simple website for searching awesome. Hope it's a helpful thing :p
I am looking forward to someone to PR any awesome list in JSON format like this because the format of awesome lists are different so that it's not so easy to write a general program to extract them into JSON I think.
Originally created by @lockys on GitHub (Nov 26, 2015).
Original GitHub issue: https://github.com/sindresorhus/awesome/issues/427
Hi, dear all
First, thank all authors of awesome list, it makes the world better.
I don't know that if there are similar projects for these.
I am trying to exact awesome lists into JSON in my leisure time(still working on it) and build a simple website for searching awesome. Hope it's a helpful thing :p
~~I am looking forward to someone to PR any awesome list in JSON format like this because the format of awesome lists are different so that it's not so easy to write a general program to extract them into JSON I think.~~
It's not an issue actually and can be closed undoubtedly.
if you get interested, below is the repos:
https://github.com/lockys/awesome-search
the JSON repo:
https://github.com/lockys/awesome.json
site:
https://awesomelists.top
tks.
We've previously discussed about this, but you're actually the first one who started working on it! So, thanks!
<!-- gh-comment-id:159968815 -->
@inputsh commented on GitHub (Nov 26, 2015):
We've previously discussed about this, but you're actually the first one who started working on it! So, thanks!
The problem I have with this JSON idea is that this list itself receives pull requests on a daily basis. It's pretty much impossible to even estimate the amount of changes happening on each and every list included here.
So, may I suggest that instead of manually adding JSON files you find some solution using GitHub's API? It might not be easy, but it definitely is necessary because manually implementing each and every change is a mission impossible.
<!-- gh-comment-id:159972563 -->
@inputsh commented on GitHub (Nov 26, 2015):
The problem I have with this JSON idea is that this list itself receives pull requests on a daily basis. It's pretty much impossible to even estimate the amount of changes happening on each and every list included here.
So, may I suggest that instead of manually adding JSON files you find some solution using GitHub's API? It might not be easy, but it definitely _is_ necessary because manually implementing each and every change is a mission impossible.
Should be possible to do some best-effort parsing of the Markdown. It's just sections and links, even though there are slight differences. I think that's a much better approach than trying to convince lots of people to manually maintain a JSON file.
<!-- gh-comment-id:160048266 -->
@sindresorhus commented on GitHub (Nov 27, 2015):
Should be possible to do some best-effort parsing of the Markdown. It's just sections and links, even though there are slight differences. I think that's a much better approach than trying to convince lots of people to manually maintain a JSON file.
@alejandronanez Thank for your constructive opinions.
@sindresorhus Yes, I agree with you. Writing a general program to parse them will be definitely better and more efficient than convincing people to contribute. Also, thank you for taking time to reply 👍
<!-- gh-comment-id:160056785 -->
@lockys commented on GitHub (Nov 27, 2015):
@alejandronanez Thank for your constructive opinions.
@sindresorhus Yes, I agree with you. Writing a general program to parse them will be definitely better and more efficient than convincing people to contribute. Also, thank you for taking time to reply :+1:
@dthree and @wooorm have some experience with parsing Markdown and might be able to point you in the right direction.
<!-- gh-comment-id:160057859 -->
@sindresorhus commented on GitHub (Nov 27, 2015):
@dthree and @wooorm have some experience with parsing Markdown and might be able to point you in the right direction.
@lockys Cool project 👍 I’m building a tool called mdast which I think could help. mdast exposes a syntax tree for markdown.
I hacked this script together:
varfs=require('fs');varmdast=require('mdast');vartoString=require('mdast-util-to-string');vartoc=/^((table of )?contents|packages)$/i;mdast().use(function(){returnfunction(ast){varsections={};varsectionName;varsection;varcontents;ast.children.forEach(function(node){if(node.type==='heading'&&node.depth!==1){// Ignore first-level headings.
contents=toString(node).trim();// Ignore TOC lists.
if(!toc.test(contents)){sectionName=contents;section=[];}}elseif(section&&node.type==='list'){// set the section (only when followed by a list).
sections[sectionName]=section;node.children.forEach(function(item){// get the link in the paragraph in the list-item.
varhead=item.children[0].children[0];contents=toString(item).split(/-/);section.push({'name':contents[0].trim(),'description':contents.slice(1).join('').trim()||null,'url':head&&head.type==='link'?head.href:null,'cate':sectionName});});}});console.log(JSON.stringify(sections,null,2));}}).process(fs.readFileSync('readme.md','utf-8'));
In the sindresorhus/awesome-nodejs repo, it produces your desired output (with descriptions added on top 😉):
{"Mad science":[{"name":"webtorrent","description":"Streaming torrent client for Node.js and the browser.","url":"https://github.com/feross/webtorrent","cate":"Mad science"},{"name":"GitTorrent","description":"Peertopeer network of Git repositories being shared over BitTorrent.","url":"https://github.com/cjb/GitTorrent","cate":"Mad science"},...{"name":"RequireBin","description":"Shareable JavaScript programs powered by npm and browserify.","url":"http://requirebin.com","cate":"Tools"},{"name":"Tonic","description":"Embed a Node.js environment on any website.","url":"http://blog.tonicdev.com/2015/09/30/embedded-tonic.html","cate":"Tools"}]}
Now, I hope this helps you. I suggest checking out how mdast works (there are a lot of docs). Good luck, and ping me if you have any Qs 😄
<!-- gh-comment-id:160111301 -->
@wooorm commented on GitHub (Nov 27, 2015):
Thanks for the ping, @sindresorhus!
@lockys Cool project :+1: I’m building a tool called [**mdast**](https://github.com/wooorm/mdast) which I think could help. mdast exposes a [syntax tree](https://github.com/wooorm/mdast/blob/master/doc/mdastnode.7.md) for markdown.
I hacked this script together:
``` js
var fs = require('fs');
var mdast = require('mdast');
var toString = require('mdast-util-to-string');
var toc = /^((table of )?contents|packages)$/i;
mdast().use(function () {
return function (ast) {
var sections = {};
var sectionName;
var section;
var contents;
ast.children.forEach(function (node) {
if (node.type === 'heading' && node.depth !== 1) { // Ignore first-level headings.
contents = toString(node).trim();
// Ignore TOC lists.
if (!toc.test(contents)) {
sectionName = contents;
section = [];
}
} else if (section && node.type === 'list') {
// set the section (only when followed by a list).
sections[sectionName] = section;
node.children.forEach(function (item) {
// get the link in the paragraph in the list-item.
var head = item.children[0].children[0];
contents = toString(item).split(/-/);
section.push({
'name': contents[0].trim(),
'description': contents.slice(1).join('').trim() || null,
'url': head && head.type === 'link' ? head.href : null,
'cate': sectionName
});
});
}
});
console.log(JSON.stringify(sections, null, 2));
}
}).process(fs.readFileSync('readme.md', 'utf-8'));
```
In the [`sindresorhus/awesome-nodejs`](https://github.com/sindresorhus/awesome-nodejs) repo, it produces your desired output (with `description`s added on top :wink:):
``` json
{
"Mad science": [
{
"name": "webtorrent",
"description": "Streaming torrent client for Node.js and the browser.",
"url": "https://github.com/feross/webtorrent",
"cate": "Mad science"
},
{
"name": "GitTorrent",
"description": "Peertopeer network of Git repositories being shared over BitTorrent.",
"url": "https://github.com/cjb/GitTorrent",
"cate": "Mad science"
},
...
{
"name": "RequireBin",
"description": "Shareable JavaScript programs powered by npm and browserify.",
"url": "http://requirebin.com",
"cate": "Tools"
},
{
"name": "Tonic",
"description": "Embed a Node.js environment on any website.",
"url": "http://blog.tonicdev.com/2015/09/30/embedded-tonic.html",
"cate": "Tools"
}
]
}
```
Now, I hope this helps you. I suggest checking out how mdast works (there are a lot of [docs](https://github.com/wooorm/mdast/tree/master/doc)). Good luck, and ping me if you have any Qs :smile:
@sindresorhus thanks for finding people who experiencing in markdown 😄
@wooorm mdast is pretty remarkable and helpful for me! It really reduces the pain of parsing markdown. I will try that! Thanks for your kindness and help 💯
<!-- gh-comment-id:160155649 -->
@lockys commented on GitHub (Nov 27, 2015):
@sindresorhus thanks for finding people who experiencing in markdown :smile:
@wooorm `mdast` is pretty remarkable and helpful for me! It really reduces the pain of parsing markdown. I will try that! Thanks for your kindness and help :100:
I have a coupon for a free .me domain that I could give to you for this project.
Unfortunately, awesome.me is currently taken, so if anyone has any ideas...
<!-- gh-comment-id:160158948 -->
@inputsh commented on GitHub (Nov 27, 2015):
I have a coupon for a free `.me` domain that I could give to you for this project.
Unfortunately, `awesome.me` is currently taken, so if anyone has any ideas...
<!-- gh-comment-id:160160327 -->
@alejandronanez commented on GitHub (Nov 27, 2015):
@aleksandar-todorovic
http://awesomethings.me
I like http://awesomelists.me too :)
@aleksandar-todorovic
wow, it's good. I love http://awesomelists.me as well.
I don't know how to express my gratitude, ha!
Thank for your generosity 😃
<!-- gh-comment-id:160165625 -->
@lockys commented on GitHub (Nov 27, 2015):
@aleksandar-todorovic
wow, it's good. I love `http://awesomelists.me` as well.
I don't know how to express my gratitude, ha!
Thank for your generosity :smiley:
Don't need to thank me, thank the people from Montenegro that gave me a coupon to their domain for free. I'm already using one (r3bl.me, so I had no idea what to do with it. 😄
Fun fact: .me is actually the biggest exporting product of the country of Montenegro.
<!-- gh-comment-id:160174855 -->
@inputsh commented on GitHub (Nov 27, 2015):
Up and running on [awesomelists.me](http://awesomelists.me).
Don't need to thank me, thank the people from Montenegro that gave me a coupon to their domain for free. I'm already using one ([r3bl.me](https://r3bl.me), so I had no idea what to do with it. :smile:
Fun fact: `.me` is actually the biggest exporting product of the country of Montenegro.
👍 on using mdast. @wooorm did an excellent job and it's been working perfectly for me in complicated markdown parsing.
<!-- gh-comment-id:160177384 -->
@dthree commented on GitHub (Nov 27, 2015):
:+1: on using `mdast`. @wooorm did an excellent job and it's been working perfectly for me in complicated markdown parsing.
<!-- gh-comment-id:160432603 -->
@Sljux commented on GitHub (Nov 29, 2015):
Great work!
On Sun, Nov 29, 2015 at 3:34 PM, Deejavu notifications@github.com wrote:
> Wow amazing work bro
>
> —
> Reply to this email directly or view it on GitHub
> https://github.com/sindresorhus/awesome/issues/427#issuecomment-160419528
> .
Actually, I talked to the guy who runs aweso.me, and he said he might be open to routing it over, as long as he still controls the domain. Just an option!
<!-- gh-comment-id:163360572 -->
@RichardLitt commented on GitHub (Dec 9, 2015):
Actually, I talked to the guy who runs aweso.me, and he said he might be open to routing it over, as long as he still controls the domain. Just an option!
@RichardLitt Thank you :)
We have decided https://awesomelists.top as our final domain name.
BTW, Awesome Search is better than the previous version for now.
Also, we set a cron job to crawl every awesome list including the awesome repo every day.
Therefore, it will be up-to-date I think!
Take a look if you get interetsed :-)
<!-- gh-comment-id:166657352 -->
@lockys commented on GitHub (Dec 22, 2015):
@RichardLitt Thank you :)
We have decided https://awesomelists.top as our final domain name.
BTW, `Awesome Search` is better than the previous version for now.
Also, we set a cron job to crawl every awesome list including the awesome repo every day.
Therefore, it will be up-to-date I think!
Take a look if you get interetsed :-)
@lockys #470 maybe of use for finding list not on the main awsome list, but as @sindresorhus said, they may be incomplete or such
<!-- gh-comment-id:171386116 -->
@pokeball99 commented on GitHub (Jan 13, 2016):
@lockys #470 maybe of use for finding list not on the main awsome list, but as @sindresorhus said, they may be incomplete or such
@pokeball99, thanks you for telling, I have checked your issue before.
But I would use sindresorhus/awesome as only source.
The reason is that the lists which @sindresorhus decided to put on awesome would be a more mature lists as you mentioned.
<!-- gh-comment-id:171677571 -->
@lockys commented on GitHub (Jan 14, 2016):
@pokeball99, thanks you for telling, I have checked your issue before.
But I would use `sindresorhus/awesome` as only source.
The reason is that the lists which @sindresorhus decided to put on awesome would be a more mature lists as you mentioned.
Hey, if anybody is still interested in this, I took my own crack at it! https://www.rtfmanual.io/awesome/. Used the x-ray package from Node to scrape the site. Happy to share the key code snippet if anybody is interested. It is nice and hacky. Inspired by this
<!-- gh-comment-id:236782359 -->
@kbk0125 commented on GitHub (Aug 2, 2016):
Hey, if anybody is still interested in this, I took my own crack at it! https://www.rtfmanual.io/awesome/. Used the [x-ray package](https://github.com/lapwinglabs/x-ray) from Node to scrape the site. Happy to share the key code snippet if anybody is interested. It is nice and hacky. Inspired by [this](https://bl.ocks.org/mbostock/7607535)
For searching, a simple Ctrl-F or Cmd-F would work. It might not be UI friendly, but I don't think people looking for these kind of resources would mind. Let me know what you think and what kind of tradeoffs is worth it. Thanks
<!-- gh-comment-id:269742287 -->
@veggiemonk commented on GitHub (Dec 30, 2016):
I don't know if you might be interested in this, for awesome-docker i've done it like this:
```html
<script src="https://cdnjs.cloudflare.com/ajax/libs/fetch/2.0.1/fetch.min.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/showdown/1.5.4/showdown.min.js"></script>
<script>
const converter = new showdown.Converter();
const text = fetch('https://raw.githubusercontent.com/veggiemonk/awesome-docker/master/README.md')
.then(res => res.text())
.catch(err => console.error(err))
.then( text => {
document.getElementById('md').innerHTML = converter.makeHtml(text);
});
</script>
```
For searching, a simple` Ctrl-F` or `Cmd-F` would work. It might not be UI friendly, but I don't think people looking for these kind of resources would mind. Let me know what you think and what kind of tradeoffs is worth it. Thanks
<!-- gh-comment-id:467585452 -->
@sindresorhus commented on GitHub (Feb 26, 2019):
We have a section for Awesome related tools now. Submit what you make there. https://github.com/sindresorhus/awesome#related :)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Originally created by @lockys on GitHub (Nov 26, 2015).
Original GitHub issue: https://github.com/sindresorhus/awesome/issues/427
Hi, dear all
First, thank all authors of awesome list, it makes the world better.
I don't know that if there are similar projects for these.
I am trying to exact awesome lists into JSON in my leisure time(still working on it) and build a simple website for searching awesome. Hope it's a helpful thing :p
I am looking forward to someone to PR any awesome list in JSON format like this because the format of awesome lists are different so that it's not so easy to write a general program to extract them into JSON I think.It's not an issue actually and can be closed undoubtedly.
if you get interested, below is the repos:
https://github.com/lockys/awesome-search
the JSON repo:
https://github.com/lockys/awesome.json
site:
https://awesomelists.top
tks.
@inputsh commented on GitHub (Nov 26, 2015):
We've previously discussed about this, but you're actually the first one who started working on it! So, thanks!
@parro-it commented on GitHub (Nov 26, 2015):
Cool project!
@inputsh commented on GitHub (Nov 26, 2015):
The problem I have with this JSON idea is that this list itself receives pull requests on a daily basis. It's pretty much impossible to even estimate the amount of changes happening on each and every list included here.
So, may I suggest that instead of manually adding JSON files you find some solution using GitHub's API? It might not be easy, but it definitely is necessary because manually implementing each and every change is a mission impossible.
@sindresorhus commented on GitHub (Nov 27, 2015):
Should be possible to do some best-effort parsing of the Markdown. It's just sections and links, even though there are slight differences. I think that's a much better approach than trying to convince lots of people to manually maintain a JSON file.
@lockys commented on GitHub (Nov 27, 2015):
@alejandronanez Thank for your constructive opinions.
@sindresorhus Yes, I agree with you. Writing a general program to parse them will be definitely better and more efficient than convincing people to contribute. Also, thank you for taking time to reply 👍
@sindresorhus commented on GitHub (Nov 27, 2015):
@dthree and @wooorm have some experience with parsing Markdown and might be able to point you in the right direction.
@wooorm commented on GitHub (Nov 27, 2015):
Thanks for the ping, @sindresorhus!
@lockys Cool project 👍 I’m building a tool called mdast which I think could help. mdast exposes a syntax tree for markdown.
I hacked this script together:
In the
sindresorhus/awesome-nodejsrepo, it produces your desired output (withdescriptions added on top 😉):Now, I hope this helps you. I suggest checking out how mdast works (there are a lot of docs). Good luck, and ping me if you have any Qs 😄
@lockys commented on GitHub (Nov 27, 2015):
@sindresorhus thanks for finding people who experiencing in markdown 😄
@wooorm
mdastis pretty remarkable and helpful for me! It really reduces the pain of parsing markdown. I will try that! Thanks for your kindness and help 💯@inputsh commented on GitHub (Nov 27, 2015):
I have a coupon for a free
.medomain that I could give to you for this project.Unfortunately,
awesome.meis currently taken, so if anyone has any ideas...@gepser commented on GitHub (Nov 27, 2015):
@aleksandar-todorovic what about http://awesomelists.me/?
@inputsh commented on GitHub (Nov 27, 2015):
@gepser sounds good (and it's available). What do you think @lockys ?
@alejandronanez commented on GitHub (Nov 27, 2015):
@aleksandar-todorovic
http://awesomethings.me
I like http://awesomelists.me too :)
@lockys commented on GitHub (Nov 27, 2015):
@aleksandar-todorovic
wow, it's good. I love
http://awesomelists.meas well.I don't know how to express my gratitude, ha!
Thank for your generosity 😃
@inputsh commented on GitHub (Nov 27, 2015):
Up and running on awesomelists.me.
Don't need to thank me, thank the people from Montenegro that gave me a coupon to their domain for free. I'm already using one (r3bl.me, so I had no idea what to do with it. 😄
Fun fact:
.meis actually the biggest exporting product of the country of Montenegro.@dthree commented on GitHub (Nov 27, 2015):
👍 on using
mdast. @wooorm did an excellent job and it's been working perfectly for me in complicated markdown parsing.@lockys commented on GitHub (Nov 28, 2015):
Thanks all you guys for helping this project 👍
@ghost commented on GitHub (Nov 29, 2015):
Wow amazing work bro
@Sljux commented on GitHub (Nov 29, 2015):
Great work!
On Sun, Nov 29, 2015 at 3:34 PM, Deejavu notifications@github.com wrote:
@RichardLitt commented on GitHub (Dec 9, 2015):
Actually, I talked to the guy who runs aweso.me, and he said he might be open to routing it over, as long as he still controls the domain. Just an option!
@lockys commented on GitHub (Dec 22, 2015):
@RichardLitt Thank you :)
We have decided https://awesomelists.top as our final domain name.
BTW,
Awesome Searchis better than the previous version for now.Also, we set a cron job to crawl every awesome list including the awesome repo every day.
Therefore, it will be up-to-date I think!
Take a look if you get interetsed :-)
@pokeball99 commented on GitHub (Jan 13, 2016):
@lockys #470 maybe of use for finding list not on the main awsome list, but as @sindresorhus said, they may be incomplete or such
@lockys commented on GitHub (Jan 14, 2016):
@pokeball99, thanks you for telling, I have checked your issue before.
But I would use
sindresorhus/awesomeas only source.The reason is that the lists which @sindresorhus decided to put on awesome would be a more mature lists as you mentioned.
@kbk0125 commented on GitHub (Aug 2, 2016):
Hey, if anybody is still interested in this, I took my own crack at it! https://www.rtfmanual.io/awesome/. Used the x-ray package from Node to scrape the site. Happy to share the key code snippet if anybody is interested. It is nice and hacky. Inspired by this
@veggiemonk commented on GitHub (Dec 30, 2016):
I don't know if you might be interested in this, for awesome-docker i've done it like this:
For searching, a simple
Ctrl-ForCmd-Fwould work. It might not be UI friendly, but I don't think people looking for these kind of resources would mind. Let me know what you think and what kind of tradeoffs is worth it. Thanks@sindresorhus commented on GitHub (Feb 26, 2019):
We have a section for Awesome related tools now. Submit what you make there. https://github.com/sindresorhus/awesome#related :)