github-starred/open-webui
2
0
Fork 0
mirror of https://github.com/open-webui/open-webui.git synced 2026-05-06 10:58:17 -05:00
Code Issues 1.3k Packages Projects Releases 126 Wiki Activity

[GH-ISSUE #1076] Docker Mainline does not Start clean from blank volume #12325

New Issue
Closed
opened 2026-04-19 19:13:42 -05:00 by GiteaMirror · 8 comments
GiteaMirror commented 2026-04-19 19:13:42 -05:00
Owner
Copy Link

Originally created by @Eforen on GitHub (Mar 7, 2024).
Original GitHub issue: https://github.com/open-webui/open-webui/issues/1076

Bug Report

Description

Bug Summary:
Spinning up with the same vars as the example .env file in a k3s cluster using blank nfs based drive does not work with main branch

Steps to Reproduce:
Just spin it up

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ollama
  name: deployment-ollama
  namespace: ai
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ollama
  template:
    metadata:
      labels:
        app: ollama
      name: deployment-ollama
    spec:
      nodeSelector:
        "kubernetes.io/os": linux
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: ollama
          image: ollama/ollama:latest
          ports:
          - containerPort: 11434
            protocol: TCP
            name: http
          volumeMounts:
            - name: ollama-data
              mountPath: "/root/.ollama"
        - name: ollama-webui
          image: ghcr.io/open-webui/open-webui:main
          ports:
          - containerPort: 8080
            protocol: TCP
            name: http
          volumeMounts:
            - name: open-webui-data
              mountPath: "/app/backend/data"
          env:
            - name: OLLAMA_BASE_URL
              value: "http://localhost:11434/api"
            - name: WEBUI_SECRET_KEY
              valueFrom: 
                secretKeyRef:
                  name: webui-secret-key
                  key: WEBUI_SECRET_KEY
            - name: SCARF_NO_ANALYTICS
              value: "true"
            - name: DO_NOT_TRACK
              value: "true"
      volumes:
        - name: ollama-data
          persistentVolumeClaim:
            claimName: pvc-ollama
        - name: open-webui-data
          persistentVolumeClaim:
            claimName: pvc-open-webui
  strategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate

Expected Behavior:
to start up cleanly

Actual Behavior:

No sentence-transformers model found with name sentence-transformers/all-MiniLM-L6-v2. Creating a new one with MEAN pooling.
Traceback (most recent call last):
  File "/usr/local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 418, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 587, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 62, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 458, in load
    self.loaded_app = import_from_string(self.app)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string
    module = importlib.import_module(module_str)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/app/backend/main.py", line 25, in <module>
    from apps.rag.main import app as rag_app
  File "/app/backend/apps/rag/main.py", line 85, in <module>
    embedding_functions.SentenceTransformerEmbeddingFunction(
  File "/usr/local/lib/python3.11/site-packages/chromadb/utils/embedding_functions.py", line 72, in __init__
    self.models[model_name] = SentenceTransformer(model_name, device=device)
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 198, in __init__
    modules = self._load_auto_model(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 1063, in _load_auto_model
    transformer_model = Transformer(
                        ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentence_transformers/models/Transformer.py", line 35, in __init__
    config = AutoConfig.from_pretrained(model_name_or_path, **model_args, cache_dir=cache_dir)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1111, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 633, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 688, in _get_config_dict
    resolved_config_file = cached_file(
                           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/utils/hub.py", line 398, in cached_file
    resolved_file = hf_hub_download(
                    ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1504, in hf_hub_download
    _chmod_and_replace(temp_file.name, blob_path)
  File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1717, in _chmod_and_replace
    os.chmod(src, stat.S_IMODE(cache_dir_mode))
PermissionError: [Errno 1] Operation not permitted: '/app/backend/data/cache/embedding/models/tmp9atjzopt'

Environment

  • Operating System: Ubuntu 20.04 with K3s Treafik removed Istio installed in it's place
  • Browser (if applicable): N/A

Reproduction Details

Confirmation:

  • I have read and followed all the instructions provided in the README.md.
  • I am on the latest version of both Open WebUI and Ollama.
  • [NA] I have included the browser console logs.
  • I have included the Docker container logs.

Logs and Screenshots

Browser Console Logs:
[Include relevant browser console logs, if applicable]

Docker Container Logs:
See Above

Screenshots (if applicable):
[Attach any relevant screenshots to help illustrate the issue]

Installation Method

Docker

Additional Information

Note

My ghcr.io/ollama-webui/ollama-webui:dev deployment works fine so does not seem to be a system issue on my side.

Originally created by @Eforen on GitHub (Mar 7, 2024). Original GitHub issue: https://github.com/open-webui/open-webui/issues/1076 # Bug Report ## Description **Bug Summary:** Spinning up with the same vars as the example .env file in a k3s cluster using blank nfs based drive does not work with main branch **Steps to Reproduce:** Just spin it up ```yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: ollama name: deployment-ollama namespace: ai spec: replicas: 1 selector: matchLabels: app: ollama template: metadata: labels: app: ollama name: deployment-ollama spec: nodeSelector: "kubernetes.io/os": linux dnsPolicy: ClusterFirstWithHostNet containers: - name: ollama image: ollama/ollama:latest ports: - containerPort: 11434 protocol: TCP name: http volumeMounts: - name: ollama-data mountPath: "/root/.ollama" - name: ollama-webui image: ghcr.io/open-webui/open-webui:main ports: - containerPort: 8080 protocol: TCP name: http volumeMounts: - name: open-webui-data mountPath: "/app/backend/data" env: - name: OLLAMA_BASE_URL value: "http://localhost:11434/api" - name: WEBUI_SECRET_KEY valueFrom: secretKeyRef: name: webui-secret-key key: WEBUI_SECRET_KEY - name: SCARF_NO_ANALYTICS value: "true" - name: DO_NOT_TRACK value: "true" volumes: - name: ollama-data persistentVolumeClaim: claimName: pvc-ollama - name: open-webui-data persistentVolumeClaim: claimName: pvc-open-webui strategy: rollingUpdate: maxSurge: 0 maxUnavailable: 1 type: RollingUpdate ``` **Expected Behavior:** to start up cleanly **Actual Behavior:** ``` No sentence-transformers model found with name sentence-transformers/all-MiniLM-L6-v2. Creating a new one with MEAN pooling. Traceback (most recent call last): File "/usr/local/bin/uvicorn", line 8, in <module> sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 418, in main run( File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 587, in run server.run() File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 62, in run return asyncio.run(self.serve(sockets=sockets)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve config.load() File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 458, in load self.loaded_app = import_from_string(self.app) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/uvicorn/importer.py", line 21, in import_from_string module = importlib.import_module(module_str) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1204, in _gcd_import File "<frozen importlib._bootstrap>", line 1176, in _find_and_load File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 690, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 940, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/app/backend/main.py", line 25, in <module> from apps.rag.main import app as rag_app File "/app/backend/apps/rag/main.py", line 85, in <module> embedding_functions.SentenceTransformerEmbeddingFunction( File "/usr/local/lib/python3.11/site-packages/chromadb/utils/embedding_functions.py", line 72, in __init__ self.models[model_name] = SentenceTransformer(model_name, device=device) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 198, in __init__ modules = self._load_auto_model( ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sentence_transformers/SentenceTransformer.py", line 1063, in _load_auto_model transformer_model = Transformer( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/sentence_transformers/models/Transformer.py", line 35, in __init__ config = AutoConfig.from_pretrained(model_name_or_path, **model_args, cache_dir=cache_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1111, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 633, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 688, in _get_config_dict resolved_config_file = cached_file( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/transformers/utils/hub.py", line 398, in cached_file resolved_file = hf_hub_download( ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1504, in hf_hub_download _chmod_and_replace(temp_file.name, blob_path) File "/usr/local/lib/python3.11/site-packages/huggingface_hub/file_download.py", line 1717, in _chmod_and_replace os.chmod(src, stat.S_IMODE(cache_dir_mode)) PermissionError: [Errno 1] Operation not permitted: '/app/backend/data/cache/embedding/models/tmp9atjzopt' ``` ## Environment - **Operating System:** Ubuntu 20.04 with K3s Treafik removed Istio installed in it's place - **Browser (if applicable):** N/A ## Reproduction Details **Confirmation:** - [X] I have read and followed all the instructions provided in the README.md. - [X] I am on the latest version of both Open WebUI and Ollama. - [NA] I have included the browser console logs. - [X] I have included the Docker container logs. ## Logs and Screenshots **Browser Console Logs:** [Include relevant browser console logs, if applicable] **Docker Container Logs:** See Above **Screenshots (if applicable):** [Attach any relevant screenshots to help illustrate the issue] ## Installation Method Docker ## Additional Information ## Note My `ghcr.io/ollama-webui/ollama-webui:dev` deployment works fine so does not seem to be a system issue on my side.
GiteaMirror commented 2026-04-19 19:13:43 -05:00
Author
Owner
Copy Link

@justinh-rahb commented on GitHub (Mar 7, 2024):

PermissionError: [Errno 1] Operation not permitted: '/app/backend/data/cache/embedding/models/tmp9atjzopt'

This looks like a problem, but unknown why it would work for :dev and not :main right now, but that could also depend on how old your image is. A hotfix was released ~2 hours ago, 4 hours after the initial v0.1.109 release earlier today.

<!-- gh-comment-id:1982250836 --> @justinh-rahb commented on GitHub (Mar 7, 2024): > ``` > PermissionError: [Errno 1] Operation not permitted: '/app/backend/data/cache/embedding/models/tmp9atjzopt' > ``` This looks like a problem, but unknown why it would work for `:dev` and not `:main` right now, but that could also depend on how old your image is. A hotfix was released ~2 hours ago, 4 hours after the initial v0.1.109 release earlier today.
GiteaMirror commented 2026-04-19 19:13:43 -05:00
Author
Owner
Copy Link

@justinh-rahb commented on GitHub (Mar 7, 2024):

Appears the issue is something mentioned in discussions:

<!-- gh-comment-id:1982254772 --> @justinh-rahb commented on GitHub (Mar 7, 2024): Appears the issue is something mentioned in discussions: - #918
GiteaMirror commented 2026-04-19 19:13:44 -05:00
Author
Owner
Copy Link

@Eforen commented on GitHub (Mar 7, 2024):

I had actually found that myself and already proved that that is not the issue.

      initContainers:
      - name: init-data
        image: ghcr.io/open-webui/open-webui:main
        command: ["/bin/bash", "-c", "if [ ! -f /data/webui.db ]; then echo 'Cloning default data folder'; cp -r /app/backend/data /data; echo 'Done'; else echo 'Data folder already exists'; fi"]
        volumeMounts:
          - name: open-webui-data
            mountPath: "/data"

This would have fixed the problem but no change

<!-- gh-comment-id:1982277004 --> @Eforen commented on GitHub (Mar 7, 2024): I had actually found that myself and already proved that that is not the issue. ```yaml initContainers: - name: init-data image: ghcr.io/open-webui/open-webui:main command: ["/bin/bash", "-c", "if [ ! -f /data/webui.db ]; then echo 'Cloning default data folder'; cp -r /app/backend/data /data; echo 'Done'; else echo 'Data folder already exists'; fi"] volumeMounts: - name: open-webui-data mountPath: "/data" ``` This would have fixed the problem but no change
GiteaMirror commented 2026-04-19 19:13:44 -05:00
Author
Owner
Copy Link

@Eforen commented on GitHub (Mar 7, 2024):

In my init container I got the logs

Cloning default data folder                                                                                                                                                                         Done

showing that it did indeed copy the data folder into the volume prior to mounting the volume in the final container that would run the software.

<!-- gh-comment-id:1982278727 --> @Eforen commented on GitHub (Mar 7, 2024): In my init container I got the logs ``` Cloning default data folder Done ``` showing that it did indeed copy the data folder into the volume prior to mounting the volume in the final container that would run the software.
GiteaMirror commented 2026-04-19 19:13:45 -05:00
Author
Owner
Copy Link

@justinh-rahb commented on GitHub (Mar 7, 2024):

We have about 2 maybe 3 regular contributors that use kubernetes, so I hope one of them or you has got some ideas or further context to add :/

<!-- gh-comment-id:1982280148 --> @justinh-rahb commented on GitHub (Mar 7, 2024): We have about 2 maybe 3 regular contributors that use kubernetes, so I hope one of them or you has got some ideas or further context to add :/
GiteaMirror commented 2026-04-19 19:13:45 -05:00
Author
Owner
Copy Link

@Eforen commented on GitHub (Mar 7, 2024):

I'll see if I can find anything else.

<!-- gh-comment-id:1982283155 --> @Eforen commented on GitHub (Mar 7, 2024): I'll see if I can find anything else.
GiteaMirror commented 2026-04-19 19:13:46 -05:00
Author
Owner
Copy Link

@Eforen commented on GitHub (Mar 7, 2024):

It may actually be related to that other issue... I remove the volume mount all together and it started up find... Obviousely that does not work... long term. When I get this sorted I'll make an example k8s config for people.

<!-- gh-comment-id:1982287557 --> @Eforen commented on GitHub (Mar 7, 2024): It may actually be related to that other issue... I remove the volume mount all together and it started up find... Obviousely that does not work... long term. When I get this sorted I'll make an example k8s config for people.
GiteaMirror commented 2026-04-19 19:13:47 -05:00
Author
Owner
Copy Link

@Eforen commented on GitHub (Mar 7, 2024):

I had a typo in my cp command >.< 😊

      initContainers:
      - name: init-data
        image: ghcr.io/open-webui/open-webui:main
        command: ["/bin/bash", "-c", "if [ ! -f /data/config.json ]; then echo 'Cloning default data folder'; cp -Rv /app/backend/data/* /data/; echo 'Done'; else echo 'Data folder already exists'; fi"]
        volumeMounts:
          - name: open-webui-data
            mountPath: "/data"

This makes everything work fine. Though this should not be needed.

<!-- gh-comment-id:1982297723 --> @Eforen commented on GitHub (Mar 7, 2024): I had a typo in my cp command >.< 😊 ```yaml initContainers: - name: init-data image: ghcr.io/open-webui/open-webui:main command: ["/bin/bash", "-c", "if [ ! -f /data/config.json ]; then echo 'Cloning default data folder'; cp -Rv /app/backend/data/* /data/; echo 'Done'; else echo 'Data folder already exists'; fi"] volumeMounts: - name: open-webui-data mountPath: "/data" ``` This makes everything work fine. Though this should not be needed.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
confirmed
confirmed issue
core
documentation
enhancement
good first issue
help wanted
non-core
pull-request
Mirrored from GitHub Pull Request
 python
question
testing wanted
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
GiteaMirror ninjasurge
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: github-starred/open-webui#12325
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?