External markup renderer [$15] #117

Closed
opened 2025-11-02 03:09:40 -06:00 by GiteaMirror · 22 comments
Owner

Originally created by @plemp on GitHub (Dec 11, 2016).

Originally assigned to: @lunny on GitHub.

See gogits/gogs#211 and gogits/gogs#2097.

Most Python-related projects use reStructuredText .rst instead of markdown files, for README.rst and documentation. Sometimes other markup languages are used, or markdown with extensions currently not supported by Gitea. GitHub supports the following markups: https://github.com/github/markup/.

I propose the following feature:

[markup]
; List of file extensions that should be rendered by an external command
FILE_EXTENSIONS = .rst,.rest,.restx
; External command to render all matching extensions
RENDER_COMMAND = "rst2html.py --no-raw"

If, for example, the file README.rst exists, its first 1024 bytes will be passed into stdin of rst2html.py --no-raw and stdout will be displayed as html for the file preview.

An optional %s will be replaced by the matched file extension (so that one can write a script that handles both .rst and .asciidoc).

As such a feature would be important to me, I am willing to implement this proposal, assuming there is a good chance it will be merged into master. Let me know if I can start working on it.


There is a $15 open bounty on this issue. Add to the bounty at Bountysource.

Originally created by @plemp on GitHub (Dec 11, 2016). Originally assigned to: @lunny on GitHub. See gogits/gogs#211 and gogits/gogs#2097. Most Python-related projects use reStructuredText `.rst` instead of markdown files, for `README.rst` and documentation. Sometimes other markup languages are used, or markdown with extensions currently not supported by Gitea. GitHub supports the following markups: https://github.com/github/markup/. I propose the following feature: ```ini [markup] ; List of file extensions that should be rendered by an external command FILE_EXTENSIONS = .rst,.rest,.restx ; External command to render all matching extensions RENDER_COMMAND = "rst2html.py --no-raw" ``` If, for example, the file `README.rst` exists, its first 1024 bytes will be passed into stdin of `rst2html.py --no-raw` and stdout will be displayed as html for the file preview. An optional `%s` will be replaced by the matched file extension (so that one can write a script that handles both `.rst` and `.asciidoc`). As such a feature would be important to me, I am willing to implement this proposal, assuming there is a good chance it will be merged into master. Let me know if I can start working on it. <bountysource-plugin> --- There is a **[$15 open bounty](https://www.bountysource.com/issues/39957080-external-markup-renderer?utm_campaign=plugin&utm_content=tracker%2F47456670&utm_medium=issues&utm_source=github)** on this issue. Add to the bounty at [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F47456670&utm_medium=issues&utm_source=github). </bountysource-plugin>
GiteaMirror added the type/featureissue/bounty labels 2025-11-02 03:09:41 -06:00
Author
Owner

@tboerger commented on GitHub (Dec 11, 2016):

Depending on external tools for rendering content is not really our goal

@tboerger commented on GitHub (Dec 11, 2016): Depending on external tools for rendering content is not really our goal
Author
Owner

@plemp commented on GitHub (Dec 11, 2016):

The default can be empty, then it's not a dependency and more like plugin interface.

@plemp commented on GitHub (Dec 11, 2016): The default can be empty, then it's not a dependency and more like plugin interface.
Author
Owner

@lunny commented on GitHub (Dec 11, 2016):

We could find a go lib to do that

@lunny commented on GitHub (Dec 11, 2016): We could find a go lib to do that
Author
Owner

@lunny commented on GitHub (Dec 12, 2016):

It seems there isn't any mature Golang library to do that. So maybe give a config like @plemp said, but default is closed currently.

@lunny commented on GitHub (Dec 12, 2016): It seems there isn't any mature Golang library to do that. So maybe give a config like @plemp said, but default is closed currently.
Author
Owner

@bkcsoft commented on GitHub (Dec 12, 2016):

IMO adding external renderers isn't a bad idea, could be made with a simple list.
Have a simple schema to identify arguments, basically injecting .InFile and .OutFile

Example:

EXTERNAL_RENDER = .rst:/usr/bin/rst2html5 {{ .InFile }} > {{ .OutFile }}

given

renderParam := struct {
  InFile  string
  OutFile string
}{"foo.rst", "/tmp/gitea-1337/foo.html"}

would generate /usr/bin/rst2html5 foo.rst > /tmp/gitea-1337/foo.html

@bkcsoft commented on GitHub (Dec 12, 2016): IMO adding external renderers isn't a bad idea, could be made with a simple list. Have a simple schema to identify arguments, basically injecting `.InFile` and `.OutFile` Example: ``` EXTERNAL_RENDER = .rst:/usr/bin/rst2html5 {{ .InFile }} > {{ .OutFile }} ``` given ``` renderParam := struct { InFile string OutFile string }{"foo.rst", "/tmp/gitea-1337/foo.html"} ``` would generate `/usr/bin/rst2html5 foo.rst > /tmp/gitea-1337/foo.html`
Author
Owner

@tboerger commented on GitHub (Dec 16, 2016):

I would prefer to integrate a real plugin system instead of increasing the current config more and more

@tboerger commented on GitHub (Dec 16, 2016): I would prefer to integrate a real plugin system instead of increasing the current config more and more
Author
Owner

@Solido commented on GitHub (Mar 11, 2017):

Well I guess I'm just here to vote for .asciidoc the rich format and feature is offers have make it a defacto tools in our product chain.

@Solido commented on GitHub (Mar 11, 2017): Well I guess I'm just here to vote for **.asciidoc** the rich format and feature is offers have make it a defacto tools in our product chain.
Author
Owner

@joneepenk commented on GitHub (Mar 16, 2017):

@tboerger maybe external renderer is better choice, because asciidoc is an extensible format. the renderer itself should have a plugin system.

@joneepenk commented on GitHub (Mar 16, 2017): @tboerger maybe external renderer is better choice, because asciidoc is an extensible format. the renderer itself should have a plugin system.
Author
Owner

@lannocc commented on GitHub (May 26, 2017):

I recently did a quick-and-dirty hack on Gogs to add some basic .rst support for our own use, but maybe you will also find it useful:

https://github.com/AlphaGriffin/gogs

@lannocc commented on GitHub (May 26, 2017): I recently did a quick-and-dirty hack on Gogs to add some basic .rst support for our own use, but maybe you will also find it useful: https://github.com/AlphaGriffin/gogs
Author
Owner

@lunny commented on GitHub (May 26, 2017):

@lannocc great! I will try to merge it on v1.3

@lunny commented on GitHub (May 26, 2017): @lannocc great! I will try to merge it on v1.3
Author
Owner

@lunny commented on GitHub (Sep 17, 2017):

After #2525 merged, I will try to add reStructuredText support based on github.com/hhatto/gorst and https://github.com/AlphaGriffin/gogs. And I think that should be easy since that a new document type only implementation markup.Parser.

@lunny commented on GitHub (Sep 17, 2017): After #2525 merged, I will try to add reStructuredText support based on github.com/hhatto/gorst and https://github.com/AlphaGriffin/gogs. And I think that should be easy since that a new document type only implementation `markup.Parser`.
Author
Owner

@lunny commented on GitHub (Sep 17, 2017):

And for asciidoc, since github.com/VonC/asciidocgo could be imported.

@lunny commented on GitHub (Sep 17, 2017): ~And for asciidoc, since github.com/VonC/asciidocgo could be imported.~
Author
Owner

@lunny commented on GitHub (Sep 21, 2017):

@plemp, It seems there is no golang lib to handle asciidoc well. don't know how to use github.com/VonC/asciidocgo.

@lunny commented on GitHub (Sep 21, 2017): @plemp, It seems there is no golang lib to handle asciidoc well. don't know how to use `github.com/VonC/asciidocgo`.
Author
Owner

@lenisko commented on GitHub (Oct 20, 2017):

Any update on this one?
@lunny https://godoc.org/github.com/VonC/asciidocgo documentation looks good to me

@lenisko commented on GitHub (Oct 20, 2017): Any update on this one? @lunny https://godoc.org/github.com/VonC/asciidocgo documentation looks good to me
Author
Owner

@lunny commented on GitHub (Oct 20, 2017):

@lenisko #2570 will support external render commands.

@lunny commented on GitHub (Oct 20, 2017): @lenisko #2570 will support external render commands.
Author
Owner

@tantegerda1 commented on GitHub (Nov 15, 2017):

@lunny: Bountysource keeps telling me "Your application has been suspended" and throws various {"error":"Internal server error."} responses, keeping me from releasing the bounty. The mails I got from them uniquely say that if I "don't respond" you will get the bounty paid out at 2017-11-21. Sorry for that delay!
Could you check back at that date and - please - get in touch with me[1], should Bountysource fail to pay the bounty.
Thank you - and all contributors - for your continuous work on Gitea!


[1] via GitHub or accounts+github@netztechniker.at

@tantegerda1 commented on GitHub (Nov 15, 2017): @lunny: Bountysource keeps telling me "Your application has been suspended" and throws various `{"error":"Internal server error."}` responses, keeping me from releasing the bounty. The mails I got from them uniquely say that if I "don't respond" you will get the bounty paid out at 2017-11-21. Sorry for that delay! Could you check back at that date and - please - get in touch with me[1], should Bountysource fail to pay the bounty. Thank you - and all contributors - for your continuous work on Gitea! --- [1] via GitHub or `accounts+github@netztechniker.at`
Author
Owner

@lunny commented on GitHub (Nov 15, 2017):

@tantegerda1 OK. Thanks for your bounty.

@lunny commented on GitHub (Nov 15, 2017): @tantegerda1 OK. Thanks for your bounty.
Author
Owner

@James-E-A commented on GitHub (Sep 21, 2018):

Getting this working was as simple as

  1. Installing docutils (didn't even need root for that!)
  2. Ensuring PATH contains $HOME/.local/bin
  3. Appending a small snippet to app.ini:
[markup.restructuredtext]
ENABLED = true
FILE_EXTENSIONS = .rst
RENDER_COMMAND = rst2html.py
IS_INPUT_FILE = false
  1. Restart the server 👍…and DONE! Readme.rst renders just as prettily on my selfhost as on GitHub! 🎉

However, what I am wondering is: isn't this a security nightmare*? How quick+easy would it be for some sufficiently clever person to monkey into existence a Readme.rst that, when uploaded, would allow doing things not otherwise doable on the server?

*—in a situation involving potentially untrusted users having write access to repositories

@James-E-A commented on GitHub (Sep 21, 2018): Getting this working was as simple as 1. Installing `docutils` (didn't even need root for that!) 2. Ensuring `PATH` contains `$HOME/.local/bin` 3. Appending a small snippet to `app.ini`: ```` [markup.restructuredtext] ENABLED = true FILE_EXTENSIONS = .rst RENDER_COMMAND = rst2html.py IS_INPUT_FILE = false ```` 4. Restart the server 👍…and DONE! `Readme.rst` renders just as prettily on my selfhost as on GitHub! 🎉 However, what I am wondering is: isn't this a security *nightmare*\*? How quick+easy would it be for some sufficiently clever person to monkey into existence a `Readme.rst` that, when uploaded, would allow _doing things not otherwise doable_ on the server? *\*—in a situation involving potentially untrusted users having write access to repositories*
Author
Owner

@James-E-A commented on GitHub (Sep 21, 2018):

Although, a small P.S. to the previous comment:

  1. Installing…
  2. Ensuring…
  3. Adding a small snippet…
  4. Restart…!

Is the above explicitly documented anywhere beyond thread #2570 in GitHub's walled-garden forum, associated with (but not documented in) commit 62d0a4d882 ?

@James-E-A commented on GitHub (Sep 21, 2018): Although, a small P.S. to the previous comment: >1. Installing… >2. Ensuring… >3. Adding a small snippet… >4. Restart…! Is the above explicitly documented _anywhere_ beyond thread #2570 in GitHub's walled-garden forum, associated with (but not documented in) commit 62d0a4d8829f214af4f9c00ecf8a81907d86ef06 ?
Author
Owner

@Shados commented on GitHub (Nov 21, 2018):

@JamesTheAwesomeDude it is definitely a security problem in that situation. I wouldn't say it is a complete nightmare, however, because you can write a wrapper for your external renderer that takes one or more steps to isolate it from your server environment, e.g.:

  • Run it in a firejail
  • Run it in an ephemeral KVM VM
  • Run it on another computer entirely

You're still left with the problem of the returned html being untrusted, but you can at least prevent attacks on the external renderer(s) from being able to perform arbitrary actions on the server.

@Shados commented on GitHub (Nov 21, 2018): @JamesTheAwesomeDude it is definitely a security problem in that situation. I wouldn't say it is a complete nightmare, however, because you can write a wrapper for your external renderer that takes one or more steps to isolate it from your server environment, e.g.: - Run it in a firejail - Run it in an ephemeral KVM VM - Run it on another computer entirely You're still left with the problem of the returned html being untrusted, but you can at least prevent attacks on the external renderer(s) from being able to perform arbitrary actions on the server.
Author
Owner

@James-E-A commented on GitHub (Dec 14, 2018):

@Shados Just to expand on that a bit,

since rst2html.py is seemingly only having stdio used, I guess it'd pretty simple to set up something like the way FastCGI servers work, just assign the wrapper/offloader script to RENDER_COMMAND.

But I really am concerned at the scarcity of documentation on this; as I mentioned before, you've got to manually hunt down the information yourself to make this work..I might have to stick something in the Wiki, maybe...


Although, re:"the problem of the returned html being untrusted",

It does look like Gitea (and Github itself, for that matter) actually sanitizes it pretty violently...for instance,

.. note:: HTML is sanitized

renders to the browser merely to:

<div><p>Note</p><p>HTML is sanitized</p></div>

with absolutely no styling at all, not even an indication that the word "Note" is a header for the following content, no indent for the note itself....anything

(same with warning, etc)

@James-E-A commented on GitHub (Dec 14, 2018): @Shados Just to expand on that a bit, since `rst2html.py` is seemingly only having stdio used, I guess it'd pretty simple to set up something like the way FastCGI servers work, just assign the wrapper/offloader script to `RENDER_COMMAND`. But I really am concerned at the scarcity of documentation on this; as I mentioned before, you've got to manually hunt down the information yourself to make this work..I might have to stick something in the Wiki, maybe... ----- Although, re:"the problem of the returned html being untrusted", It does look like Gitea (and Github itself, for that matter) actually sanitizes it pretty violently...for instance, .. note:: HTML is sanitized renders to the browser merely to: <div><p>Note</p><p>HTML is sanitized</p></div> with absolutely no styling at all, not even an indication that the word "Note" is a header for the following content, no indent for the note itself....anything (same with `warning`, etc)
Author
Owner

@Shados commented on GitHub (Dec 17, 2018):

@JamesTheAwesomeDude yes, output from markup modules (external renderer included) are passed through a sanitizer in an attempt at solving that part of the problem.

Of course, if you want the output to be useful, you would likely need to relax the sanitizer slightly to allow more class names (whatever ones your renderer is using), and then separately supply a manually-audited or created stylesheet that defines those classes.

You're still left with two possible avenues of attack, however:

  • Attacks on the sanitizer; it's running as part of the gitea server so a successful attack on it or its parser could potentially allow an attacker to access anything gitea can
  • Attacks on browsers viewing the sanitized html; I don't know what someone could possibly do with just HTML and a set of known-good classes, so maybe this is not worth worrying about
@Shados commented on GitHub (Dec 17, 2018): @JamesTheAwesomeDude yes, output from markup modules (external renderer included) are passed through a [sanitizer](https://github.com/go-gitea/gitea/blob/master/modules/markup/sanitizer.go) in an attempt at solving that part of the problem. Of course, if you want the output to be useful, you would likely need to relax the sanitizer slightly to allow more class names (whatever ones your renderer is using), and then separately supply a manually-audited or created stylesheet that defines those classes. You're still left with two possible avenues of attack, however: - Attacks on the sanitizer; it's running as part of the gitea server so a successful attack on it or its parser could potentially allow an attacker to access anything gitea can - Attacks on browsers viewing the sanitized html; I don't know what someone could possibly do with just HTML and a set of known-good classes, so maybe this is not worth worrying about
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#117