github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-07-28 01:33:48 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Web Upload Treats UTF-16LE XML Files as Binary Despite .gitattributes #14197

New Issue
Open
opened 2025-11-02 11:06:01 -06:00 by GiteaMirror · 11 comments
GiteaMirror commented 2025-11-02 11:06:01 -06:00
Owner
Copy Link
Copy Source

Originally created by @kool2zero on GitHub (Feb 26, 2025).

Description

Description

When uploading a UTF-16 Little Endian (without BOM) XML file to a Gitea repository via the web interface, the file is treated as binary, and textual diffs are not displayed. Committing the same file via the command line (or VS Code, which uses the command line) works correctly, with textual diffs shown as expected. This indicates that the issue is specific to the web upload process.

Issue Replicated in Demo

Steps to Reproduce

  1. Create a new Gitea repository.

  2. Create a .gitattributes file in the root of the repository with the following content:

    *.xml text working-tree-encoding=UTF-16LE eol=crlf

  3. Create an XML file (e.g., test.xml) encoded in UTF-16 Little Endian without a Byte Order Mark (BOM). You can use a text editor like Notepad++ or VS Code to ensure the correct encoding. A simple example file:

    <?xml version="1.0" encoding="UTF-16"?>
<root>
    <element attribute="value1"/>
</root>

  4. Commit and push the .gitattributes and test.xml files using the Git command line:

    git add .gitattributes test.xml
git commit -m "Initial commit"
git push origin main  # Or your branch name

  5. Modify the XML file locally. Change the attribute value (e.g., from "value1" to "value2"). Save the file (still as UTF-16LE, no BOM).

  6. Upload via the Web UI:

    • Go to your repository in Gitea's web interface.
    • Click Add File then Upload File
    • Select the file to be uploaded.
    • Add a title to the commit and hit the Commit Changes button.

  7. View the diff: Go to the commit history and view the diff for the uploaded test.xml file.

Expected Behavior

The diff should be displayed as a textual diff, showing the change in the attribute value (e.g., "value1" to "value2").

Actual Behavior

The diff is displayed as a binary diff ("Binary files a/test.xml and b/test.xml differ").

Context

  • Gitea Version: 1.23.4
  • Operating System (Gitea Server): Windows Server 2019
  • Git Version (Server): git version 2.48.1.windows.1
  • Git Version (Client): git version 2.47.1.windows.1
  • Browser: Chrome 133
  • app.ini git section:
[git]
MAX_GIT_DIFF_LINES = 200000
MAX_GIT_DIFF_LINE_CHARACTERS = 5000
ENABLE_CHARSET_DETECTION = true

### Gitea Version

1.23.4

### Can you reproduce the bug on the Gitea demo site?

Yes

### Log Gist

https://gist.github.com/kool2zero/ab06322362cdafb7370db3ae3ebfca78

### Screenshots

After creating new branches and commiting via VS Code and the Web.

Initiating a pull request from the VSCode branch:

![Image](https://github.com/user-attachments/assets/c397c8ae-584f-47c8-84a7-1c5e23c63872)

Diff shows properly:

![Image](https://github.com/user-attachments/assets/3ce3cabd-024e-4292-9c36-0a9e26d77c8c)

Trying to initiate the pull request from the UploadFromWeb Branch:

![Image](https://github.com/user-attachments/assets/f2ff698c-cb96-46aa-a312-c6697f734c20)

Shows it as a binary file difference:

![Image](https://github.com/user-attachments/assets/a54b7114-eac2-4912-9deb-1695919ebf47)


### Git Version

git version 2.48.1.windows.1

### Operating System

Windows 2019

### How are you running Gitea?

I am using the Windows executable run via NSSM.

### Database

PostgreSQL
Originally created by @kool2zero on GitHub (Feb 26, 2025). ### Description ## Description When uploading a UTF-16 Little Endian (without BOM) XML file to a Gitea repository via the web interface, the file is treated as binary, and textual diffs are not displayed. Committing the *same* file via the command line (or VS Code, which uses the command line) works correctly, with textual diffs shown as expected. This indicates that the issue is specific to the web upload process. [Issue Replicated in Demo](https://demo.gitea.com/Jim-Geraci/Test-UTF-16-Encoded-Files) ## Steps to Reproduce 1. **Create a new Gitea repository.** 2. **Create a `.gitattributes` file** in the root of the repository with the following content: ``` *.xml text working-tree-encoding=UTF-16LE eol=crlf ``` 3. **Create an XML file** (e.g., `test.xml`) encoded in UTF-16 Little Endian *without* a Byte Order Mark (BOM). You can use a text editor like Notepad++ or VS Code to ensure the correct encoding. A simple example file: ```xml <?xml version="1.0" encoding="UTF-16"?> <root> <element attribute="value1"/> </root> ``` 4. **Commit and push the `.gitattributes` and `test.xml` files** using the Git command line: ```bash git add .gitattributes test.xml git commit -m "Initial commit" git push origin main # Or your branch name ``` 5. **Modify the XML file locally.** Change the attribute value (e.g., from "value1" to "value2"). Save the file (still as UTF-16LE, no BOM). 6. **Upload via the Web UI**: * Go to your repository in Gitea's web interface. * Click `Add File` then `Upload File` * Select the file to be uploaded. * Add a title to the commit and hit the `Commit Changes` button. 7. **View the diff:** Go to the commit history and view the diff for the uploaded `test.xml` file. ## Expected Behavior The diff should be displayed as a textual diff, showing the change in the attribute value (e.g., "value1" to "value2"). ## Actual Behavior The diff is displayed as a binary diff ("Binary files a/test.xml and b/test.xml differ"). ## Context * **Gitea Version:** 1.23.4 * **Operating System (Gitea Server):** Windows Server 2019 * **Git Version (Server):** git version 2.48.1.windows.1 * **Git Version (Client):** git version 2.47.1.windows.1 * **Browser:** Chrome 133 * **app.ini git section**: ```ini [git] MAX_GIT_DIFF_LINES = 200000 MAX_GIT_DIFF_LINE_CHARACTERS = 5000 ENABLE_CHARSET_DETECTION = true ### Gitea Version 1.23.4 ### Can you reproduce the bug on the Gitea demo site? Yes ### Log Gist https://gist.github.com/kool2zero/ab06322362cdafb7370db3ae3ebfca78 ### Screenshots After creating new branches and commiting via VS Code and the Web. Initiating a pull request from the VSCode branch: ![Image](https://github.com/user-attachments/assets/c397c8ae-584f-47c8-84a7-1c5e23c63872) Diff shows properly: ![Image](https://github.com/user-attachments/assets/3ce3cabd-024e-4292-9c36-0a9e26d77c8c) Trying to initiate the pull request from the UploadFromWeb Branch: ![Image](https://github.com/user-attachments/assets/f2ff698c-cb96-46aa-a312-c6697f734c20) Shows it as a binary file difference: ![Image](https://github.com/user-attachments/assets/a54b7114-eac2-4912-9deb-1695919ebf47) ### Git Version git version 2.48.1.windows.1 ### Operating System Windows 2019 ### How are you running Gitea? I am using the Windows executable run via NSSM. ### Database PostgreSQL
GiteaMirror added the type/bug label 2025-11-02 11:06:01 -06:00
GiteaMirror commented 2025-11-02 11:06:02 -06:00
Author
Owner
Copy Link
Copy Source

@delvh commented on GitHub (Feb 26, 2025):

Image
?

@delvh commented on GitHub (Feb 26, 2025): ![Image](https://github.com/user-attachments/assets/507ce596-dc67-46cf-b828-ab934a9668f2) ?
GiteaMirror commented 2025-11-02 11:06:02 -06:00
Author
Owner
Copy Link
Copy Source

@kool2zero commented on GitHub (Feb 26, 2025):

Check the diff between the two auxiliary branches during a pull request. When the file is pushed via vs code, the diff seems to work. When the file is uploaded via the web interface, it does not. The file is treated as binary...

@kool2zero commented on GitHub (Feb 26, 2025): Check the diff between the two auxiliary branches during a pull request. When the file is pushed via vs code, the diff seems to work. When the file is uploaded via the web interface, it does not. The file is treated as binary...
GiteaMirror commented 2025-11-02 11:06:02 -06:00
Author
Owner
Copy Link
Copy Source

@wxiaoguang commented on GitHub (Feb 27, 2025):

When the file is uploaded via the web interface

IIRC the web interface only supports UTF-8

@wxiaoguang commented on GitHub (Feb 27, 2025): > When the file is uploaded via the web interface IIRC the web interface only supports UTF-8
GiteaMirror commented 2025-11-02 11:06:02 -06:00
Author
Owner
Copy Link
Copy Source

@kool2zero commented on GitHub (Feb 27, 2025):

It seems like the change was made 2 years ago to not assume the encoding, so I thought that it should work if the file is being uploaded as is. I'm assuming the web interface is using the API to upload the file.
#18368 #25828

@kool2zero commented on GitHub (Feb 27, 2025): It seems like the change was made 2 years ago to not assume the encoding, so I thought that it should work if the file is being uploaded as is. I'm assuming the web interface is using the API to upload the file. #18368 #25828
GiteaMirror commented 2025-11-02 11:06:02 -06:00
Author
Owner
Copy Link
Copy Source

@lunny commented on GitHub (Feb 27, 2025):

When the file is uploaded via the web interface

IIRC the web interface only supports UTF-8

Is that related? He is just uploading the file with no modification in the web interface.

It seems like the change was made 2 years ago to not assume the encoding, so I thought that it should work if the file is being uploaded as is. I'm assuming the web interface is using the API to upload the file. #18368 #25828

I don't believe they are related. Removing the guessing of encoding is right which is consistent how a file pushed from git client.

@lunny commented on GitHub (Feb 27, 2025): > > When the file is uploaded via the web interface > > IIRC the web interface only supports UTF-8 Is that related? He is just uploading the file with no modification in the web interface. > It seems like the change was made 2 years ago to not assume the encoding, so I thought that it should work if the file is being uploaded as is. I'm assuming the web interface is using the API to upload the file. [#18368](https://github.com/go-gitea/gitea/issues/18368) [#25828](https://github.com/go-gitea/gitea/pull/25828) I don't believe they are related. Removing the guessing of encoding is right which is consistent how a file pushed from git client.
GiteaMirror commented 2025-11-02 11:06:03 -06:00
Author
Owner
Copy Link
Copy Source

@kool2zero commented on GitHub (Feb 27, 2025):

Basically my use case is to add a code review prior to loading the file into the target application. So I would like to have a user go into the web interface, create a branch, upload their change, then initiate a pull request. Using workflows, I would then take the file and load it into the target system. The application expects UTF-16.

I am trying to remove the need for them to use separate git software.

As of right now, my workaround is to have them:

  • download and install github desktop and git and clone the repository.
  • Create a branch in GH Desktop
  • Copy the file to their cloned folder.
  • Push the change to the branch.
  • Log into gitea and navigate to their branch
  • Initiate the pull request.

Let me know if this makes sense or you need additional clarification.

@kool2zero commented on GitHub (Feb 27, 2025): Basically my use case is to add a code review prior to loading the file into the target application. So I would like to have a user go into the web interface, create a branch, upload their change, then initiate a pull request. Using workflows, I would then take the file and load it into the target system. The application expects UTF-16. I am trying to remove the need for them to use separate git software. As of right now, my workaround is to have them: - download and install github desktop and git and clone the repository. - Create a branch in GH Desktop - Copy the file to their cloned folder. - Push the change to the branch. - Log into gitea and navigate to their branch - Initiate the pull request. Let me know if this makes sense or you need additional clarification.
GiteaMirror commented 2025-11-02 11:06:03 -06:00
Author
Owner
Copy Link
Copy Source

@wxiaoguang commented on GitHub (Feb 28, 2025):

When the file is uploaded via the web interface

IIRC the web interface only supports UTF-8

Is that related? He is just uploading the file with no modification in the web interface.

Why not related? The web interface only supports UTF-8.

Image

@wxiaoguang commented on GitHub (Feb 28, 2025): > > > When the file is uploaded via the web interface > > > > > > IIRC the web interface only supports UTF-8 > > Is that related? He is just uploading the file with no modification in the web interface. Why not related? The web interface only supports UTF-8. ![Image](https://github.com/user-attachments/assets/c61005d0-fa30-4a64-9c79-098b99c2b6f8)
GiteaMirror commented 2025-11-02 11:06:03 -06:00
Author
Owner
Copy Link
Copy Source

@wxiaoguang commented on GitHub (Feb 28, 2025):

BOM/Encoding problem is the same as EOL problem.

See my proposal:

Think about some files in a git repo:

  • A file without BOM
  • A file with UTF-8 BOM
  • A file with UTF-16 BOM
  • A file with "\n"
  • A file with "\r\n"

By existing approach, they could all be edited by MonacoEditor(web editor). The real problem is when "saving" the files, backend code does incorrect assumption: it only recognizes UTF-8 or "\n". So UTF-16 BOM and "\r\n" files are all changed unexpectedly.

So, the complete approach could be like this IMO: when saving, the backend code should first detect the file kind (BOM, EOL), then convert the submitted text to the expected format (correct BOM, correct EOL), then everything should work well.

@wxiaoguang commented on GitHub (Feb 28, 2025): BOM/Encoding problem is the same as EOL problem. See my proposal: * https://github.com/go-gitea/gitea/pull/28935#issuecomment-1912115491 * https://github.com/go-gitea/gitea/pull/28119#issuecomment-1912129780 ---- > Think about some files in a git repo: > > * A file without BOM > * A file with UTF-8 BOM > * A file with UTF-16 BOM > * A file with "\n" > * A file with "\r\n" > > By existing approach, they could all be edited by MonacoEditor(web editor). The real problem is when "saving" the files, backend code does incorrect assumption: it only recognizes UTF-8 or "\n". So UTF-16 BOM and "\r\n" files are all changed unexpectedly. > > So, the complete approach could be like this IMO: when saving, the backend code should first detect the file kind (BOM, EOL), then convert the submitted text to the expected format (correct BOM, correct EOL), then everything should work well.
GiteaMirror commented 2025-11-02 11:06:04 -06:00
Author
Owner
Copy Link
Copy Source

@lunny commented on GitHub (Feb 28, 2025):

When the file is uploaded via the web interface

IIRC the web interface only supports UTF-8

Is that related? He is just uploading the file with no modification in the web interface.

Why not related? The web interface only supports UTF-8.

Image

Oh. Yes, it's related. If they use uploading file feature, that should not be affected.

@lunny commented on GitHub (Feb 28, 2025): > > > > When the file is uploaded via the web interface > > > > > > > > > IIRC the web interface only supports UTF-8 > > > > > > Is that related? He is just uploading the file with no modification in the web interface. > > Why not related? The web interface only supports UTF-8. > > ![Image](https://github.com/user-attachments/assets/c61005d0-fa30-4a64-9c79-098b99c2b6f8) Oh. Yes, it's related. If they use `uploading file` feature, that should not be affected.
GiteaMirror commented 2025-11-02 11:06:04 -06:00
Author
Owner
Copy Link
Copy Source

@kool2zero commented on GitHub (Feb 28, 2025):

Oh. Yes, it's related. If they use uploading file feature, that should not be affected.

Sorry for the confusion, I had used copilot to help make sure I got all of the information I needed as I was trying to figure out where the process was breaking down and I skimmed the part where it said upload via web ui. I am not copying and pasting, I am using the upload file feature.

I have updated the initial problem with the above. I apologize for the confusion, purely my bad.

  1. Upload via the Web UI:
    • Go to your repository in Gitea's web interface.
    • Click Add File then Upload File
    • Select the file to be uploaded.
    • Add a title to the commit and hit the Commit Changes button.

Image

Image

@kool2zero commented on GitHub (Feb 28, 2025): > Oh. Yes, it's related. If they use `uploading file` feature, that should not be affected. Sorry for the confusion, I had used copilot to help make sure I got all of the information I needed as I was trying to figure out where the process was breaking down and I skimmed the part where it said upload via web ui. I am not copying and pasting, I am using the upload file feature. I have updated the initial problem with the above. I apologize for the confusion, purely my bad. > 6. **Upload via the Web UI**: > * Go to your repository in Gitea's web interface. > * Click `Add File` then `Upload File` > * Select the file to be uploaded. > * Add a title to the commit and hit the `Commit Changes` button. ![Image](https://github.com/user-attachments/assets/bf6f23a2-2b0e-4be3-b35c-81435c954c20) ![Image](https://github.com/user-attachments/assets/9affafae-1fbb-4e86-ba77-a5432375687a)
GiteaMirror commented 2025-11-02 11:06:04 -06:00
Author
Owner
Copy Link
Copy Source

@kool2zero commented on GitHub (Apr 10, 2025):

Any thoughts?

@kool2zero commented on GitHub (Apr 10, 2025): Any thoughts?
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No labels type/bug
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
GiteaMirror ninjasurge (Kaehlen Rog)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#14197