Create file in a repository with Rest API (content, encoding and special chars escape) #3774

New Issue

GiteaMirror · 2025-11-02T05:24:46-06:00

GiteaMirror commented

2025-11-02 05:24:46 -06:00

Originally created by @fraballi on GitHub (Aug 13, 2019).

What is the correct encoding and format for using this endpoint:
http://localhost:3000/api/swagger#/repository/repoCreateFile

I got this error (without base64 conversion):
[{"classification":"DeserializationError","message":"invalid character '\n' in string literal"}]

Then I converted to base64 the file content and got another error:
Error in char 0

Also, how to escape correctly those special chars during file content extraction?

Originally created by @fraballi on GitHub (Aug 13, 2019). What is the correct encoding and format for using this endpoint: http://localhost:3000/api/swagger#/repository/repoCreateFile I got this error (without base64 conversion): [{"classification":"DeserializationError","message":"invalid character '\\n' in string literal"}] Then I converted to base64 the file content and got another error: Error in char 0 Also, how to escape correctly those special chars during file content extraction?

GiteaMirror added the type/question issue/stale issue/needs-feedback modifies/api labels 2025-11-02 05:24:46 -06:00

GiteaMirror closed this issue

2025-11-02 05:24:46 -06:00

GiteaMirror commented

2025-11-02 05:24:47 -06:00

@richmahn commented on GitHub (Aug 26, 2019):

@fraballi Should be base64. Can you paste the encoding here?

@richmahn commented on GitHub (Aug 26, 2019): @fraballi Should be base64. Can you paste the encoding here?

GiteaMirror commented

2025-11-02 05:24:47 -06:00

@fraballi commented on GitHub (Aug 26, 2019):

Many thanks, on your reply.
I had many issues with encoding and JSON deserialization. Ending chars and
line breaks fails for Go/JSON deserializer API, even if a base64 content is
sent to your REST API (Google/Stackoverflow answers)

Steps:

$my_file="the-file.yml"
*1) File content: *file_content=$(iconv -c -f (file -b --mime-encoding
my_file) -t UTF-8 $my_file)
2) Base64 conversion: base64_content=$(echo -n $file_content | base64)

Choice)
*I decided to use git push (so I had more than one file). *
Do you have any API feature that uploads a folder into a repository?

;-)

El lun., 26 ago. 2019 a las 14:21, Richard Mahn (notifications@github.com)
escribió:

@fraballi https://github.com/fraballi Should be base64. Can you paste
the encoding here?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/go-gitea/gitea/issues/7844?email_source=notifications&email_token=AA4GYYVPXECE3KYWPX6XUJTQGQNLXA5CNFSM4ILOC442YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FGXGY#issuecomment-524970907,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4GYYVHQOEZYLUHO65WLODQGQNLXANCNFSM4ILOC44Q
.

@fraballi commented on GitHub (Aug 26, 2019): *Many thanks, on your reply.* *I had many issues with encoding and JSON deserialization. Ending chars and line breaks fails for Go/JSON deserializer API, even if a base64 content is sent to your REST API (Google/Stackoverflow answers)* *Steps:* *$my_file="the-file.yml"* *1) File content: *file_content=$(iconv -c -f $(file -b --mime-encoding $ *my_file*) -t UTF-8 $*my_file*) 2) Base64 conversion: base64_content=$(echo -n $file_content | base64) *Choice)* *I decided to use git push (so I had more than one file). * *Do you have any API feature that uploads a folder into a repository?* *;-)* El lun., 26 ago. 2019 a las 14:21, Richard Mahn (<notifications@github.com>) escribió: > @fraballi <https://github.com/fraballi> Should be base64. Can you paste > the encoding here? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/go-gitea/gitea/issues/7844?email_source=notifications&email_token=AA4GYYVPXECE3KYWPX6XUJTQGQNLXA5CNFSM4ILOC442YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FGXGY#issuecomment-524970907>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AA4GYYVHQOEZYLUHO65WLODQGQNLXANCNFSM4ILOC44Q> > . >

GiteaMirror commented

2025-11-02 05:24:47 -06:00

@richmahn commented on GitHub (Aug 26, 2019):

Kind of confused by your reply, but I did my own testing and it seems like I had to run base64 on a UTF-16 string, not UTF-8. For example, this content fails:

VGhpcyBpcyBzb21lCgpjb250ZW50IHdpdGggbmV3IGxpbmVzIGFuZCBcInF1b3RlZCBxdW90ZXNcIgoKYW5kIHNvbWUgY2hpbmVzZSDkvaDlpb0KCg===

but this encoding in base64 works:

//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA=

Not sure if that is a bug or what should be expected. Here is my whole payload to the POST http://localhost:3000/api/v1/repos/{owner}/{repo}/contents/test.txt API:

  "message": "initial commit of test_special_characters.txt",
  "committer": {
    "name": "Jill",
    "email": "jill@gmail.com"
  },
  "author": {
  	"name": "John",
  	"email": "john@gmail.com"
  },
  "branch": "master",
  "content": "//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA="
}

That content string is base64 encoding of a UTF-16 string. You can see that decoded here: https://bg.door43.org/richmahn/test_this/src/branch/master/test_special_characters.txt

@richmahn commented on GitHub (Aug 26, 2019): Kind of confused by your reply, but I did my own testing and it seems like I had to run base64 on a UTF-16 string, not UTF-8. For example, this content fails: ```VGhpcyBpcyBzb21lCgpjb250ZW50IHdpdGggbmV3IGxpbmVzIGFuZCBcInF1b3RlZCBxdW90ZXNcIgoKYW5kIHNvbWUgY2hpbmVzZSDkvaDlpb0KCg===``` but this encoding in base64 works: ```//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA=``` Not sure if that is a bug or what should be expected. Here is my whole payload to the POST `http://localhost:3000/api/v1/repos/{owner}/{repo}/contents/test.txt` API: ```{ "message": "initial commit of test_special_characters.txt", "committer": { "name": "Jill", "email": "jill@gmail.com" }, "author": { "name": "John", "email": "john@gmail.com" }, "branch": "master", "content": "//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA=" } ``` That content string is base64 encoding of a UTF-16 string. You can see that decoded here: https://bg.door43.org/richmahn/test_this/src/branch/master/test_special_characters.txt

GiteaMirror commented

2025-11-02 05:24:47 -06:00

@fraballi commented on GitHub (Aug 26, 2019):

Thanks, that seems to be the issue: UTF-16 replacing UTF-8.
Which conversion function shall be used for UTF-16?

I missed REST part in my previous thread, but that's it. Using "content"
part in a JSON payload.

Thanks for the help.

El lun., 26 ago. 2019 a las 14:38, Richard Mahn (notifications@github.com)
escribió:

Kind of confused by your reply, but I did my own testing and it seems like
I had to run base64 on a UTF-16 string, not UTF-8. For example, this
content fails:

VGhpcyBpcyBzb21lCgpjb250ZW50IHdpdGggbmV3IGxpbmVzIGFuZCBcInF1b3RlZCBxdW90ZXNcIgoKYW5kIHNvbWUgY2hpbmVzZSDkvaDlpb0KCg===

but this encoding in base64 works:

//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA=

Not sure if that is a bug or what should be expected. Here is my whole
payload to the POST
http://localhost:3000/api/v1/repos/{owner}/{repo}/contents/test.txt API:

"message": "initial commit of test_special_characters.txt",
"committer": {
"name": "Jill",
"email": "jill@localhost.com"
},
"author": {
"name": "Rich",
"email": "richmahn@gmail.com"
},
"branch": "master",
"content": "//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA="
}

That content string is base64 encoding of a UTF-16 string. You can see
that decoded here:
https://bg.door43.org/richmahn/test_this/src/branch/master/test_special_characters.txt

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/go-gitea/gitea/issues/7844?email_source=notifications&email_token=AA4GYYQD6D6ID7VHVXL4JELQGQPJTA5CNFSM4ILOC442YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FILMI#issuecomment-524977585,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA4GYYUDNXBATPIELY3FADDQGQPJTANCNFSM4ILOC44Q
.

@fraballi commented on GitHub (Aug 26, 2019): Thanks, that seems to be the issue: UTF-16 replacing UTF-8. Which conversion function shall be used for UTF-16? I missed REST part in my previous thread, but that's it. Using "content" part in a JSON payload. Thanks for the help. El lun., 26 ago. 2019 a las 14:38, Richard Mahn (<notifications@github.com>) escribió: > Kind of confused by your reply, but I did my own testing and it seems like > I had to run base64 on a UTF-16 string, not UTF-8. For example, this > content fails: > > > VGhpcyBpcyBzb21lCgpjb250ZW50IHdpdGggbmV3IGxpbmVzIGFuZCBcInF1b3RlZCBxdW90ZXNcIgoKYW5kIHNvbWUgY2hpbmVzZSDkvaDlpb0KCg=== > > but this encoding in base64 works: > > > //5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA= > > Not sure if that is a bug or what should be expected. Here is my whole > payload to the POST > http://localhost:3000/api/v1/repos/{owner}/{repo}/contents/test.txt API: > > "message": "initial commit of test_special_characters.txt", > "committer": { > "name": "Jill", > "email": "jill@localhost.com" > }, > "author": { > "name": "Rich", > "email": "richmahn@gmail.com" > }, > "branch": "master", > "content": "//5UAGgAaQBzACAAaQBzACAAcwBvAG0AZQAKAAoAYwBvAG4AdABlAG4AdAAgAHcAaQB0AGgAIABuAGUAdwAgAGwAaQBuAGUAcwAgAGEAbgBkACAAXAAiAHEAdQBvAHQAZQBkACAAcQB1AG8AdABlAHMAXAAiAAoACgBhAG4AZAAgAHMAbwBtAGUAIABjAGgAaQBuAGUAcwBlACAAYE99WQoACgA=" > } > > That content string is base64 encoding of a UTF-16 string. You can see > that decoded here: > https://bg.door43.org/richmahn/test_this/src/branch/master/test_special_characters.txt > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/go-gitea/gitea/issues/7844?email_source=notifications&email_token=AA4GYYQD6D6ID7VHVXL4JELQGQPJTA5CNFSM4ILOC442YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FILMI#issuecomment-524977585>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AA4GYYUDNXBATPIELY3FADDQGQPJTANCNFSM4ILOC44Q> > . >

GiteaMirror commented

2025-11-02 05:24:47 -06:00

@guillep2k commented on GitHub (Aug 26, 2019):

Kind of confused by your reply, but I did my own testing and it seems like I had to run base64 on a UTF-16 string, not UTF-8. For example, this content fails:

I'm not familiar with this API endpoint, but it does look like a bug to me:

I think UTF-16 is a weird default. Except specific reasons to use UTF-16 (e.g. Asian languages), it is generally not much recommended. UTF-8 should be the default, IMO.
If a file is uploaded, encoding should not be an issue! What if I wish to upload a binary file?
If encoding is important (I don't see why it should) the user should be able to add a content-type field to the request JSON.

@guillep2k commented on GitHub (Aug 26, 2019): > > > Kind of confused by your reply, but I did my own testing and it seems like I had to run base64 on a UTF-16 string, not UTF-8. For example, this content fails: I'm not familiar with this API endpoint, but it does look like a bug to me: * I think UTF-16 is a weird default. Except specific reasons to use UTF-16 (e.g. Asian languages), it is generally not much recommended. UTF-8 should be the default, IMO. * If a file is uploaded, encoding should not be an issue! What if I wish to upload a binary file? * If encoding is important (I don't see why it should) the user should be able to add a `content-type` field to the request JSON.

GiteaMirror commented

2025-11-02 05:24:48 -06:00

@richmahn commented on GitHub (Sep 4, 2019):

I agree. I thought it was odd I had to use UTF-16 to encode it into base64 for our decoder to work. Apparently it works out of the box that way? I'll keep a bookmark on this issue and see if I can get the encoding to work with UTF-8

@richmahn commented on GitHub (Sep 4, 2019): I agree. I thought it was odd I had to use UTF-16 to encode it into base64 for our decoder to work. Apparently it works out of the box that way? I'll keep a bookmark on this issue and see if I can get the encoding to work with UTF-8

GiteaMirror commented

2025-11-02 05:24:48 -06:00

@stale[bot] commented on GitHub (Nov 3, 2019):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

@stale[bot] commented on GitHub (Nov 3, 2019): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

GiteaMirror commented

2025-11-02 05:24:48 -06:00

@zeeshan1112 commented on GitHub (Jul 14, 2020):

@richmahn How did you make it work with UTF-16 encoding?
String bytesEncodedContent = Base64.getEncoder().encodeToString(content.getBytes(StandardCharsets.UTF_16));
This did not help. I get characters as � in github viewer for special characters.
Any suggestions?

@zeeshan1112 commented on GitHub (Jul 14, 2020): @richmahn How did you make it work with UTF-16 encoding? `String bytesEncodedContent = Base64.getEncoder().encodeToString(content.getBytes(StandardCharsets.UTF_16));` This did not help. I get characters as � in github viewer for special characters. Any suggestions?

GiteaMirror commented

2025-11-02 05:24:48 -06:00

@richmahn commented on GitHub (Aug 13, 2020):

@zeeshan1112 I believe I was just using an online base64 decoder to make the payload to Gitea. I don't think I was actually doing it in an app, but I could be mistaken as it was a year ago.

@richmahn commented on GitHub (Aug 13, 2020): @zeeshan1112 I believe I was just using an online base64 decoder to make the payload to Gitea. I don't think I was actually doing it in an app, but I could be mistaken as it was a year ago.

GiteaMirror commented

2025-11-02 05:24:49 -06:00

@stale[bot] commented on GitHub (Aug 22, 2021):

This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

@stale[bot] commented on GitHub (Aug 22, 2021): This issue has been automatically marked as stale because it has not had recent activity. I am here to help clear issues left open even if solved or waiting for more insight. This issue will be closed if no further activity occurs during the next 2 weeks. If the issue is still valid just add a comment to keep it alive. Thank you for your contributions.

GiteaMirror referenced this issue

2025-11-02 12:25:46 -06:00

[PR #3774] [MERGED] Backport: fix: show Clipboard button if disable HTTP of git protocol #3773 #17077

Sign in to join this conversation.

Branches Tags

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/gitea#3774