github-starred/gitea
2
0
Fork 0
mirror of https://github.com/go-gitea/gitea.git synced 2026-03-13 02:57:44 -05:00
Code Issues 2.6k Packages Projects Releases 100 Wiki Activity

Web deployment with static content #9849

New Issue
Closed
opened 2025-11-02 08:51:14 -06:00 by GiteaMirror · 16 comments
GiteaMirror commented 2025-11-02 08:51:14 -06:00
Owner
Copy Link

Originally created by @KB3HNS on GitHub (Nov 17, 2022).

Description

Hello. I need some clarification.
I've built my own version of gitea for Debian 11 (bullseye) including PAM and excluding the built-ins. I'm currently running a local instance of APACHE 2 with appropriate mod_proxy and mod_proxy_http. I am able to confirm that apache is correctly passing requests to gitea. However, all requests are resulting in a 500 internal server error. Even when manually running gitea with the --verbose option there is very little information to go on as shown below (redacting some potentially sensitive information):

2022/11/17 00:01:12 cmd/web.go:217:listen() [I] [6375c018-5] Listen: http://0.0.0.0:3000/git
2022/11/17 00:01:12 cmd/web.go:221:listen() [I] [6375c018-5] AppURL(ROOT_URL): https://www.XXXXXX.END/git/
2022/11/17 00:01:12 ...s/graceful/server.go:61:NewServer() [I] [6375c018-5] Starting new Web server: tcp:0.0.0.0:3000 on PID: 2274289
2022/11/17 00:01:30 ...s/context/context.go:232:HTML() [E] [6375c02a] Render failed: html/template: "install" is undefined
2022/11/17 00:01:30 ...eb/routing/logger.go:99:func1() [I] [6375c02a] router: completed GET / for xxx.xxx.xxx.xxx, 500 Internal Server Error in 0.5ms @ install/install.go:90(install.Install)

I have copied the contents of ${BUILD_DIR}/"public" to /var/www-ssl/git-static, based on this documentation and the resulting gitea executable to /usr/local/bin. I have also set up an appropriate account to run under and systemd startup script based on these instructions.

My current app.ini:

[server]
STATIC_URL_PREFIX = /git-static
ROOT_URL = https://www.XXXXXX.END/git

If I run the application in the build directory it executes fine. However, the build directory resides on a dedicated build server and the build environment is purposefully not available on the production server (both the build system and the production server are running identically patched versions of Debian 11 - only that the production server does not have the ability to compile software for security reasons).
I desire to avoid building in the static content as this server has a heavy load and is somewhat stretched for resources as it is. PAM authentication is required as I have already implemented a system-wide user database and do not desire to maintain a separate authentication system.

Gitea Version

1.17.3 built with GNU Make 4.3, go1.19.1 : pam

Can you reproduce the bug on the Gitea demo site?

No

Log Gist

Logs inline

Screenshots

-N/A- see logs

Git Version

2.30.2

Operating System

Linux xxxxxx 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux

How are you running Gitea?

See description.

Database

MySQL

Originally created by @KB3HNS on GitHub (Nov 17, 2022). ### Description Hello. I need some clarification. I've built my own version of gitea for Debian 11 (bullseye) including PAM and _excluding_ the built-ins. I'm currently running a local instance of APACHE 2 with appropriate mod_proxy and mod_proxy_http. I am able to confirm that apache is correctly passing requests to gitea. However, all requests are resulting in a 500 internal server error. Even when manually running gitea with the --verbose option there is very little information to go on as shown below (redacting some potentially sensitive information): ``` 2022/11/17 00:01:12 cmd/web.go:217:listen() [I] [6375c018-5] Listen: http://0.0.0.0:3000/git 2022/11/17 00:01:12 cmd/web.go:221:listen() [I] [6375c018-5] AppURL(ROOT_URL): https://www.XXXXXX.END/git/ 2022/11/17 00:01:12 ...s/graceful/server.go:61:NewServer() [I] [6375c018-5] Starting new Web server: tcp:0.0.0.0:3000 on PID: 2274289 2022/11/17 00:01:30 ...s/context/context.go:232:HTML() [E] [6375c02a] Render failed: html/template: "install" is undefined 2022/11/17 00:01:30 ...eb/routing/logger.go:99:func1() [I] [6375c02a] router: completed GET / for xxx.xxx.xxx.xxx, 500 Internal Server Error in 0.5ms @ install/install.go:90(install.Install) ``` I have copied the contents of ${BUILD_DIR}/"public" to /var/www-ssl/git-static, based on [this documentation](https://docs.gitea.io/en-us/reverse-proxies/#nginx-and-serve-static-resources-directly) and the resulting gitea executable to /usr/local/bin. I have also set up an appropriate account to run under and systemd startup script based on [these instructions](https://computingforgeeks.com/install-gitea-git-service-on-debian/). My current app.ini: ``` [server] STATIC_URL_PREFIX = /git-static ROOT_URL = https://www.XXXXXX.END/git ``` If I run the application in the build directory it executes fine. However, the build directory resides on a dedicated build server and the build environment is purposefully not available on the production server (both the build system and the production server are running identically patched versions of Debian 11 - only that the production server does not have the ability to compile software for security reasons). I desire to avoid building in the static content as this server has a heavy load and is somewhat stretched for resources as it is. PAM authentication is required as I have already implemented a system-wide user database and do not desire to maintain a separate authentication system. ### Gitea Version 1.17.3 built with GNU Make 4.3, go1.19.1 : pam ### Can you reproduce the bug on the Gitea demo site? No ### Log Gist Logs inline ### Screenshots -N/A- see logs ### Git Version 2.30.2 ### Operating System Linux xxxxxx 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64 GNU/Linux ### How are you running Gitea? See description. ### Database MySQL
GiteaMirror added the type/question label 2025-11-02 08:51:14 -06:00
GiteaMirror commented 2025-11-02 08:51:14 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 17, 2022):

Further clarification, command line to execute gitea:
/usr/local/bin/gitea web -c /etc/gitea/app.ini

@KB3HNS commented on GitHub (Nov 17, 2022): Further clarification, command line to execute gitea: `/usr/local/bin/gitea web -c /etc/gitea/app.ini`
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

After doing some searching, I believe that this is related to #18110

@KB3HNS commented on GitHub (Nov 18, 2022): After doing some searching, I believe that this is related to #18110
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@clarfonthey commented on GitHub (Nov 18, 2022):

Are you sure that the JS you're using matches the version of gitea correctly? Since my issue was having stale extracts and it not matching what the frontend was actually looking for.

@clarfonthey commented on GitHub (Nov 18, 2022): Are you sure that the JS you're using matches the version of gitea correctly? Since my issue was having stale extracts and it not matching what the frontend was actually looking for.
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

Are you sure that the JS you're using matches the version of gitea correctly? Since my issue was having stale extracts and it not matching what the frontend was actually looking for.

In my case, yes as I directly copied the public folder into my www-ssl directory (renamed, of course).

@KB3HNS commented on GitHub (Nov 18, 2022): > Are you sure that the JS you're using matches the version of gitea correctly? Since my issue was having stale extracts and it not matching what the frontend was actually looking for. In my case, yes as I directly copied the public folder into my www-ssl directory (renamed, of course).
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

You are doing something wrong, unrelated to 18110

Please follow the guideline strictly, for example: the document https://docs.gitea.io/en-us/install-from-source/ has told you TAGS="bindata" make build

You may need TAGS="bindata pam" make build to build with PAM

@wxiaoguang commented on GitHub (Nov 18, 2022): You are doing something wrong, unrelated to 18110 Please follow the guideline strictly, for example: the document https://docs.gitea.io/en-us/install-from-source/ has told you `TAGS="bindata" make build` You may need `TAGS="bindata pam" make build` to build with PAM
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

As per my description: adding bindata is not desired. I would much prefer to have the static content hosted separately so it will be cached by my CDN and therefore won't require server resources to provide. According to documentation, bindata is an optional flag.

@KB3HNS commented on GitHub (Nov 18, 2022): As per my description: adding `bindata` is not desired. I would much prefer to have the static content hosted separately so it will be cached by my CDN and therefore won't require server resources to provide. According to documentation, `bindata` is an optional flag.
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

I can not understand why "bindata" is not desired (you didn't mention in your description either Hmm, I didn't understand that before). It's only optional for experienced users who clearly know what they want and fully understand how Gitea assets work.

If you want to play with bindata by yourself, please also put the template files in the correct places, following to the document.

For a quick and stable solution: just use bindata. You can still serve these static resources via CDN (just proxy the static assets to Gitea instance, it's very fast and will be cached soon)

@wxiaoguang commented on GitHub (Nov 18, 2022): I can not understand why "bindata" is not desired (~~you didn't mention in your description either~~ Hmm, I didn't understand that before). It's only optional for experienced users who clearly know what they want and fully understand how Gitea assets work. If you want to play with `bindata` by yourself, please also put the template files in the correct places, following to the document. For a quick and stable solution: just use `bindata`. You can still serve these static resources via CDN (just proxy the static assets to Gitea instance, it's very fast and will be cached soon)
GiteaMirror commented 2025-11-02 08:51:15 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

Another choice is that you still build Gitea with bindata, then copy the JS/CSS resources to your filesystem as your plan (I won't recommend so, you will run into other problems if you upgrade Gitea but forgot the update the resources).

bindata is not only for JS/CSS, it also contains templates, which is not related to your CDN, but used by Gitea internally.

@wxiaoguang commented on GitHub (Nov 18, 2022): Another choice is that you still build Gitea with `bindata`, then copy the JS/CSS resources to your filesystem as your plan (I won't recommend so, you will run into other problems if you upgrade Gitea but forgot the update the resources). `bindata` is not only for JS/CSS, it also contains templates, which is not related to your CDN, but used by Gitea internally.
GiteaMirror commented 2025-11-02 08:51:16 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

Primary issue is that the current server is a VPS with limited memory and CPU resources. It is shared between 3 total developers and has very low code velocity. I would prefer for static assets to be served directly by apache and not proxied through an additional application - this will allow cloudflare to properly cache them while continuing to forward dynamic content through to the application. I am running many other services on the same web server - all using FCGI for HTTP/2 and as a result apache has a 512MB/instance overhead so I am limited to the total number of apache instances that I can support (3). An instance that is being used to forward a request for what amounts to a static file is an instance that could have been put to better use generating dynamic content.

To get around the 500 error, I did to a build with bindata to prove out the issue - again not desired. There was no setting of STATIC_URL_PREFIX that would allow the control panel (first page prior to database provisioning) to properly load without throwing the pop-up exception referred to in the referenced issue, hence why I do believe this to be a legitimate bug.

@KB3HNS commented on GitHub (Nov 18, 2022): Primary issue is that the current server is a VPS with limited memory and CPU resources. It is shared between 3 total developers and has very low code velocity. I would prefer for static assets to be served directly by apache and not proxied through an additional application - this will allow cloudflare to properly cache them while continuing to forward dynamic content through to the application. I am running many other services on the same web server - all using FCGI for HTTP/2 and as a result apache has a 512MB/instance overhead so I am limited to the total number of apache instances that I can support (3). An instance that is being used to forward a request for what amounts to a static file is an instance that could have been put to better use generating dynamic content. To get around the 500 error, I did to a build with bindata to prove out the issue - again not desired. There was no setting of `STATIC_URL_PREFIX` that would allow the control panel (first page prior to database provisioning) to properly load without throwing the pop-up exception referred to in the referenced issue, hence why I do believe this to be a legitimate bug.
GiteaMirror commented 2025-11-02 08:51:16 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

I see, if you want to save memory heavily, you should look into the bindata details to separate the assets carefully and put bindata files in your deployment manually. I doubt that it won't save too much memory ....

ps: I proposed a PR to fix the document: #21853

@wxiaoguang commented on GitHub (Nov 18, 2022): I see, if you want to save memory heavily, you should look into the `bindata` details to separate the assets carefully and put bindata files in your deployment manually. I doubt that it won't save too much memory .... ps: I proposed a PR to fix the document: #21853
GiteaMirror commented 2025-11-02 08:51:16 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

At the moment, there are 3 kinds of bindata assets:

  1. modules/public/bindata.go (the static web assets)
  2. modules/options/bindata.go (the options, see the options directory)
  3. modules/templates/bindata.go (the backend templates)

When Gitea is running, only 1 can be put in other servers (to be served by other HTTP sites)

2 & 3 are necessary for Gitea, either bundle them into Gitea binary, or put them in Gitea's working directory (IIRC some documents about customizing may have mentioned that)

@wxiaoguang commented on GitHub (Nov 18, 2022): At the moment, there are 3 kinds of bindata assets: 1. modules/public/bindata.go (the static web assets) 2. modules/options/bindata.go (the options, see the options directory) 3. modules/templates/bindata.go (the backend templates) When Gitea is running, only 1 can be put in other servers (to be served by other HTTP sites) 2 & 3 are necessary for Gitea, either bundle them into Gitea binary, or put them in Gitea's working directory (IIRC some documents about customizing may have mentioned that)
GiteaMirror commented 2025-11-02 08:51:16 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

Ok. Thanks for the PR and clarification there.

At this point not nearly that important but; it would be nice to know what content can be put at STATIC_URL_PREFIX. It seems that whatever I put there (even with bindata added as a tag) the system throws the web error. At this point, system is working with PAM authentication and the option commented out in app.ini.

Further note - this is for reference only, perhaps someone in the future will find this helpful:
This works on "Debian 11 AKA BullsEye" but I imagine that it will work for any other debian based system (Ubuntu, Mint, etc). The correct "PAM Service Name" for using the normal Linux Password authentication mechanism is common-session-noninteractive.

@KB3HNS commented on GitHub (Nov 18, 2022): Ok. Thanks for the PR and clarification there. At this point not nearly that important but; it would be nice to know what content can be put at `STATIC_URL_PREFIX`. It seems that whatever I put there (even with `bindata` added as a tag) the system throws the web error. At this point, system is working with PAM authentication and the option commented out in app.ini. Further note - this is for reference only, perhaps someone in the future will find this helpful: This works on "Debian 11 AKA BullsEye" but I imagine that it will work for any other debian based system (Ubuntu, Mint, etc). The correct "PAM Service Name" for using the normal Linux Password authentication mechanism is `common-session-noninteractive`.
GiteaMirror commented 2025-11-02 08:51:17 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

it would be nice to know what content can be put at STATIC_URL_PREFIX

I think they are the JS/CSS files (under the http://gitea/assets, which is called as public in source code, i.e the kind 1)

The correct "PAM Service Name" for using the normal Linux Password authentication mechanism is common-session-noninteractive.

Do you mean it could be documented, or be changed in source code?

ps: I do not use STATIC_URL_PREFIX nor PAM 😂, if you could help to improve the document and code, I can help to review and approve. Or you can tell me how to improve the document and code, then I will propose PRs.

@wxiaoguang commented on GitHub (Nov 18, 2022): > it would be nice to know what content can be put at STATIC_URL_PREFIX I think they are the JS/CSS files (under the `http://gitea/assets`, which is called as `public` in source code, i.e the kind 1) > The correct "PAM Service Name" for using the normal Linux Password authentication mechanism is common-session-noninteractive. Do you mean it could be documented, or be changed in source code? ---- ps: I do not use STATIC_URL_PREFIX nor PAM 😂, if you could help to improve the document and code, I can help to review and approve. Or you can tell me how to improve the document and code, then I will propose PRs.
GiteaMirror commented 2025-11-02 08:51:17 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 18, 2022):

At the moment, there are 3 kinds of bindata assets:

1. modules/public/bindata.go (the static web assets)

2. modules/options/bindata.go (the options, see the options directory)

3. modules/templates/bindata.go (the backend templates)

When Gitea is running, only 1 can be put in other servers (to be served by other HTTP sites)

2 & 3 are necessary for Gitea, either bundle them into Gitea binary, or put them in Gitea's working directory (IIRC some documents about customizing may have mentioned that)

So.. About that #3 above... Is it possible that, the STATIC_URL_PREFIX is being incorrectly applied to items in this bucket? That actually would align with the behavior that I was seeing after adding bindata.1

Do you mean it could be documented, or be changed in source code? ...

It's getting a bit late here in Central US and I have to get up early for work tomorrow. I'll fork and put together a PR to try and improve some of that documentation around PAM - give me a few days. Can't help on the STATIC_URL_PREFIX yet because I can't reproduce a working system.

  1. Along with that PR, I may also try to see if I can find some time to investigate the prefix issue myself to see if I can better see what's going on. Haven't screwed with GO much at all, I'm a C/C++/Python/Assembly guy so will be a bit of a learning curve there. But your comment plus my addiction to regular expression hunting may be just enough to be able to triage this a bit. ↩︎

@KB3HNS commented on GitHub (Nov 18, 2022): > At the moment, there are 3 kinds of bindata assets: > > 1. modules/public/bindata.go (the static web assets) > > 2. modules/options/bindata.go (the options, see the options directory) > > 3. modules/templates/bindata.go (the backend templates) > > > When Gitea is running, only 1 can be put in other servers (to be served by other HTTP sites) > > 2 & 3 are necessary for Gitea, either bundle them into Gitea binary, or put them in Gitea's working directory (IIRC some documents about customizing may have mentioned that) So.. About that `#3` above... Is it possible that, the `STATIC_URL_PREFIX` is being _incorrectly_ applied to items in this bucket? That actually would align with the behavior that I was seeing after adding `bindata`.[^1] > Do you mean it could be documented, or be changed in source code? ... It's getting a bit late here in Central US and I have to get up early for work tomorrow. I'll fork and put together a PR to try and improve some of that documentation around PAM - give me a few days. Can't help on the `STATIC_URL_PREFIX` yet because I can't reproduce a working system. [^1]: Along with that PR, I may also try to see if I can find some time to investigate the prefix issue myself to see if I can better see what's going on. Haven't screwed with GO much at all, I'm a C/C++/Python/Assembly guy so will be a bit of a learning curve there. But your comment plus my addiction to regular expression hunting may be just enough to be able to triage this a bit.
GiteaMirror commented 2025-11-02 08:51:17 -06:00
Author
Owner
Copy Link

@wxiaoguang commented on GitHub (Nov 18, 2022):

So.. About that #3 above... Is it possible that, the STATIC_URL_PREFIX is being incorrectly applied to items in this bucket?

  • STATIC_URL_PREFIX is only applied to the generated JS/CSS URLs in HTML pages, and only for the /assets/ directory.
  • Kind3 (backend templates) are loaded by Gitea locally when Gitea runs.

Not sure whehter I understand your question correctly, the logic is like this:

  • server-gitea: Gitea runs
  • server-gitea: Gitea read backend template from bundled bindata or from local file system (https://docs.gitea.io/en-us/customizing-gitea/)
  • browser sends a HTTP request to server-gitea
  • server-gitea renders the HTML page by loaded templates, and fill the STATIC_URL_PREFIX into the template
  • browser gets the rendered page, and get the asset URL in HTML like STATIC_URL_PREFIX/assets/index.js
  • browser loads the STATIC_URL_PREFIX/assets/index.js
  • by default, STATIC_URL_PREFIX/assets/index.js is server-gitea/assets/index.js, so Gitea serves it
  • but you can set STATIC_URL_PREFIX to other URLs.
@wxiaoguang commented on GitHub (Nov 18, 2022): > So.. About that #3 above... Is it possible that, the STATIC_URL_PREFIX is being incorrectly applied to items in this bucket? * `STATIC_URL_PREFIX` is only applied to the generated JS/CSS URLs in HTML pages, and only for the `/assets/` directory. * Kind3 (backend templates) are loaded by Gitea locally when Gitea runs. Not sure whehter I understand your question correctly, the logic is like this: * server-gitea: Gitea runs * server-gitea: Gitea read backend template from bundled bindata or from local file system (https://docs.gitea.io/en-us/customizing-gitea/) * browser sends a HTTP request to server-gitea * server-gitea renders the HTML page by loaded templates, and fill the STATIC_URL_PREFIX into the template * browser gets the rendered page, and get the asset URL in HTML like `STATIC_URL_PREFIX/assets/index.js` * browser loads the `STATIC_URL_PREFIX/assets/index.js` * by default, `STATIC_URL_PREFIX/assets/index.js` is `server-gitea/assets/index.js`, so Gitea serves it * but you can set STATIC_URL_PREFIX to other URLs.
GiteaMirror commented 2025-11-02 08:51:18 -06:00
Author
Owner
Copy Link

@KB3HNS commented on GitHub (Nov 19, 2022):

Very interesting. So, now that I have the server, the behavior is working as expected. It was an issue when I did not have the server configured - a bit of a chicken-and-egg thing in that I couldn't set up the server (pop-up dialog A-la linked issue). Now that the server has been configured and database initialized, the site works as expected. And it was specifically having issues with index.js. Can't really explain that one, but I will also update / clarify that section of the documentation as well in my PR.

@KB3HNS commented on GitHub (Nov 19, 2022): Very interesting. So, now that I have the server, the behavior is working as expected. It was an issue when I did not have the server configured - a bit of a chicken-and-egg thing in that I couldn't set up the server (pop-up dialog A-la linked issue). Now that the server has been configured and database initialized, the site works as expected. And it was specifically having issues with `index.js`. Can't really explain that one, but I will also update / clarify that section of the documentation as well in my PR.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
$20
$250
$50
$500
backport/done
💎 Bounty
docs-update-needed
good first issue
hacktoberfest
issue/bounty
issue/confirmed
issue/critical
issue/duplicate
issue/needs-feedback
issue/not-a-bug
issue/regression
issue/stale
issue/workaround
lgtm/need 2
modifies/api
modifies/translation
outdated/backport/v1.18
outdated/theme/markdown
outdated/theme/timetracker
performance/bigrepo
performance/cpu
performance/memory
performance/speed
pr/breaking
proposal/accepted
proposal/rejected
pr/wip
pull-request
Mirrored from GitHub Pull Request
 reviewed/wontfix
💰 Rewarded
skip-changelog
status/blocked
topic/accessibility
topic/api
topic/authentication
topic/build
topic/code-linting
topic/commit-signing
topic/content-rendering
topic/deployment
topic/distribution
topic/federation
topic/gitea-actions
topic/issues
topic/lfs
topic/mobile
topic/moderation
topic/packages
topic/pr
topic/projects
topic/repo
topic/repo-migration
topic/security
topic/theme
topic/ui
topic/ui-interaction
topic/ux
topic/webhooks
topic/wiki
type/bug
type/deprecation
type/docs
type/enhancement
type/feature
type/miscellaneous
type/proposal
type/question
type/refactoring
type/summary
type/testing
type/upstream
No Label type/question
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/gitea#9849
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?