You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 2024-06-09 23:01:48
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Framework 13 with AMD running llms
Hi, I've been running Arch Linux on my newish Framework 13/AMD laptop without an issue since it arrived. However, I've found that llm performance on my laptop isn't great. Here's the output of "llm_benchmark" python utility:
-------Linux----------
No GPU detected.
Total memory size : 30.66 GB
cpu_info: AMD Ryzen 7 7840U w/ Radeon 780M Graphics
gpu_info: no_gpu
os_version: "Arch Linux"
ollama_version: 0.1.41
Your machine UUID : a79f8302-d19d-5b65-a214-213535d44a16
-------Linux----------
No GPU detected.
{
"mistral:7b": "6.09",
"llama3:8b": "5.64",
"phi3:3.8b": "9.54",
"gemma:2b": "14.14",
"gemma:7b": "4.85",
"llava:7b": "6.21",
"llava:13b": "3.43",
"uuid": "a79f8302-d19d-5b65-a214-213535d44a16",
"ollama_version": "0.1.41"
}These numbers seem much lower than those posted by others running other distributions, like here: https://community.frame.work/t/llm-benc … 840u/49351
I have the latest firmware:
# dmidecode -s bios-version
03.05And amdgpu seems to be working fine:
$ lsmod | grep amdgpu
amdgpu 13885440 23
amdxcp 12288 1 amdgpu
i2c_algo_bit 20480 1 amdgpu
drm_ttm_helper 12288 1 amdgpu
ttm 102400 2 amdgpu,drm_ttm_helper
drm_exec 12288 1 amdgpu
gpu_sched 65536 1 amdgpu
drm_suballoc_helper 12288 1 amdgpu
drm_buddy 20480 1 amdgpu
drm_display_helper 262144 1 amdgpu
video 77824 1 amdgpuAnd glxinfo doesn't say anything alarming
$ glxinfo | head -n 50
name of display: :0
display: :0 screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
server glx extensions:
GLX_ARB_context_flush_control, GLX_ARB_create_context,
GLX_ARB_create_context_no_error, GLX_ARB_create_context_profile,
GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float,
GLX_ARB_framebuffer_sRGB, GLX_ARB_multisample,
GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile,
GLX_EXT_fbconfig_packed_float, GLX_EXT_framebuffer_sRGB,
GLX_EXT_get_drawable_type, GLX_EXT_libglvnd, GLX_EXT_no_config_context,
GLX_EXT_texture_from_pixmap, GLX_EXT_visual_info, GLX_EXT_visual_rating,
GLX_MESA_copy_sub_buffer, GLX_OML_swap_method, GLX_SGIS_multisample,
GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGIX_visual_select_group,
GLX_SGI_make_current_read
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
client glx extensions:
GLX_ARB_context_flush_control, GLX_ARB_create_context,
GLX_ARB_create_context_no_error, GLX_ARB_create_context_profile,
GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float,
GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample,
GLX_ATI_pixel_format_float, GLX_EXT_buffer_age,
GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile,
GLX_EXT_fbconfig_packed_float, GLX_EXT_framebuffer_sRGB,
GLX_EXT_import_context, GLX_EXT_no_config_context, GLX_EXT_swap_control,
GLX_EXT_swap_control_tear, GLX_EXT_texture_from_pixmap,
GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_INTEL_swap_event,
GLX_MESA_copy_sub_buffer, GLX_MESA_gl_interop, GLX_MESA_query_renderer,
GLX_MESA_swap_control, GLX_NV_float_buffer, GLX_OML_sync_control,
GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer,
GLX_SGIX_visual_select_group, GLX_SGI_make_current_read,
GLX_SGI_swap_control, GLX_SGI_video_sync
GLX version: 1.4
GLX extensions:
GLX_ARB_context_flush_control, GLX_ARB_create_context,
GLX_ARB_create_context_no_error, GLX_ARB_create_context_profile,
GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float,
GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample,
GLX_EXT_buffer_age, GLX_EXT_create_context_es2_profile,
GLX_EXT_create_context_es_profile, GLX_EXT_fbconfig_packed_float,
GLX_EXT_framebuffer_sRGB, GLX_EXT_no_config_context, GLX_EXT_swap_control,
GLX_EXT_swap_control_tear, GLX_EXT_texture_from_pixmap,
GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_MESA_copy_sub_buffer,
GLX_MESA_gl_interop, GLX_MESA_query_renderer, GLX_MESA_swap_control,
GLX_OML_sync_control, GLX_SGIS_multisample, GLX_SGIX_fbconfig,
GLX_SGIX_pbuffer, GLX_SGIX_visual_select_group, GLX_SGI_make_current_read,
GLX_SGI_video_sync
This is the output when starting "ollama serve":
$ ollama serve
2024/06/09 15:49:11 routes.go:1007: INFO server config env="map[OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_HOST: OLLAMA_KEEP_ALIVE: OLLAMA_LLM_LIBRARY: OLLAMA_MAX_LOADED_MODELS:1 OLLAMA_MAX_QUEUE:512 OLLAMA_MAX_VRAM:0 OLLAMA_MODELS: OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:1 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:*] OLLAMA_RUNNERS_DIR: OLLAMA_TMPDIR:]"
time=2024-06-09T15:49:11.655-07:00 level=INFO source=images.go:729 msg="total blobs: 38"
time=2024-06-09T15:49:11.655-07:00 level=INFO source=images.go:736 msg="total unused blobs removed: 0"
[GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
- using env: export GIN_MODE=release
- using code: gin.SetMode(gin.ReleaseMode)
[GIN-debug] POST /api/pull --> github.com/ollama/ollama/server.(*Server).PullModelHandler-fm (5 handlers)
[GIN-debug] POST /api/generate --> github.com/ollama/ollama/server.(*Server).GenerateHandler-fm (5 handlers)
[GIN-debug] POST /api/chat --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (5 handlers)
[GIN-debug] POST /api/embeddings --> github.com/ollama/ollama/server.(*Server).EmbeddingsHandler-fm (5 handlers)
[GIN-debug] POST /api/create --> github.com/ollama/ollama/server.(*Server).CreateModelHandler-fm (5 handlers)
[GIN-debug] POST /api/push --> github.com/ollama/ollama/server.(*Server).PushModelHandler-fm (5 handlers)
[GIN-debug] POST /api/copy --> github.com/ollama/ollama/server.(*Server).CopyModelHandler-fm (5 handlers)
[GIN-debug] DELETE /api/delete --> github.com/ollama/ollama/server.(*Server).DeleteModelHandler-fm (5 handlers)
[GIN-debug] POST /api/show --> github.com/ollama/ollama/server.(*Server).ShowModelHandler-fm (5 handlers)
[GIN-debug] POST /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).CreateBlobHandler-fm (5 handlers)
[GIN-debug] HEAD /api/blobs/:digest --> github.com/ollama/ollama/server.(*Server).HeadBlobHandler-fm (5 handlers)
[GIN-debug] GET /api/ps --> github.com/ollama/ollama/server.(*Server).ProcessHandler-fm (5 handlers)
[GIN-debug] POST /v1/chat/completions --> github.com/ollama/ollama/server.(*Server).ChatHandler-fm (6 handlers)
[GIN-debug] GET / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] GET /api/tags --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] GET /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
[GIN-debug] HEAD / --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func1 (5 handlers)
[GIN-debug] HEAD /api/tags --> github.com/ollama/ollama/server.(*Server).ListModelsHandler-fm (5 handlers)
[GIN-debug] HEAD /api/version --> github.com/ollama/ollama/server.(*Server).GenerateRoutes.func2 (5 handlers)
time=2024-06-09T15:49:11.656-07:00 level=INFO source=routes.go:1053 msg="Listening on 127.0.0.1:11434 (version 0.1.41)"
time=2024-06-09T15:49:11.657-07:00 level=INFO source=payload.go:30 msg="extracting embedded files" dir=/tmp/ollama3663068283/runners
time=2024-06-09T15:49:12.656-07:00 level=INFO source=payload.go:44 msg="Dynamic LLM libraries [cpu rocm]"
time=2024-06-09T15:49:12.689-07:00 level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-09T15:49:12.689-07:00 level=INFO source=amd_linux.go:233 msg="unsupported Radeon iGPU detected skipping" id=0 total="512.0 MiB"
time=2024-06-09T15:49:12.689-07:00 level=INFO source=amd_linux.go:311 msg="no compatible amdgpu devices detected"
time=2024-06-09T15:49:12.690-07:00 level=INFO source=types.go:71 msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="30.7 GiB" available="26.8 GiB"Any help or pointers would be greatly appreciated!
Offline
#2 2024-06-10 11:05:21
- Lone_Wolf
- Administrator
- From: Netherlands, Europe
- Registered: 2005-10-04
- Posts: 12,772
Re: Framework 13 with AMD running llms
Are you using ollama-rocm ?
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
#3 2024-06-10 12:51:19
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Re: Framework 13 with AMD running llms
It's at least installed:
$ sudo pacman -S ollama-rocm
warning: ollama-rocm-0.1.41-1 is up to date -- reinstallingI had previously installed ollama outside of pacman, but followed these steps to remove it in favor of the pacman version: https://collabnix.com/how-to-uninstall-ollama/
Offline
#4 2024-06-10 13:10:19
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Re: Framework 13 with AMD running llms
Actually it turns out that I had not run anything while using "ollama-rocm" as I just tried it and it seg faults when trying to generate:
time=2024-06-10T06:05:47.808-07:00 level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-10T06:05:47.808-07:00 level=INFO source=amd_linux.go:233 msg="unsupported Radeon iGPU detected skipping" id=0 total="512.0 MiB"
time=2024-06-10T06:05:47.808-07:00 level=INFO source=amd_linux.go:311 msg="no compatible amdgpu devices detected"
time=2024-06-10T06:05:47.899-07:00 level=INFO source=memory.go:133 msg="offload to gpu" layers.requested=-1 layers.real=0 memory.available="267.0 MiB" memory.required.full="4.3 GiB" memory.required.partial="302.0 MiB" memory.required.kv="256.0 MiB" memory.weights.total="3.8 GiB" memory.weights.repeating="3.7 GiB" memory.weights.nonrepeating="105.0 MiB" memory.graph.full="164.0 MiB" memory.graph.partial="185.0 MiB"
time=2024-06-10T06:05:47.900-07:00 level=INFO source=server.go:341 msg="starting llama server" cmd="/tmp/ollama3536283051/runners/cpu/ollama_llama_server --model /home/pierre/.ollama/models/blobs/sha256-ff82381e2bea77d91c1b824c7afb83f6fb73e9f7de9dda631bcdbca564aa5435 --ctx-size 2048 --batch-size 512 --embedding --log-disable --parallel 1 --port 43071"
time=2024-06-10T06:05:47.900-07:00 level=INFO source=sched.go:338 msg="loaded runners" count=1
time=2024-06-10T06:05:47.900-07:00 level=INFO source=server.go:529 msg="waiting for llama runner to start responding"
time=2024-06-10T06:05:47.900-07:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server error"
INFO [main] build info | build=3051 commit="5921b8f08" tid="125214278028352" timestamp=1718024747
INFO [main] system info | n_threads=8 n_threads_batch=-1 system_info="AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 | " tid="125214278028352" timestamp=1718024747 total_threads=16
INFO [main] HTTP server listening | hostname="127.0.0.1" n_threads_http="15" port="43071" tid="125214278028352" timestamp=1718024747
llama_model_loader: loaded meta data with 25 key-value pairs and 291 tensors from /home/pierre/.ollama/models/blobs/sha256-ff82381e2bea77d91c1b824c7afb83f6fb73e9f7de9dda631bcdbca564aa5435 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv 0: general.architecture str = llama
llama_model_loader: - kv 1: general.name str = Mistral-7B-Instruct-v0.3
llama_model_loader: - kv 2: llama.block_count u32 = 32
llama_model_loader: - kv 3: llama.context_length u32 = 32768
llama_model_loader: - kv 4: llama.embedding_length u32 = 4096
llama_model_loader: - kv 5: llama.feed_forward_length u32 = 14336
llama_model_loader: - kv 6: llama.attention.head_count u32 = 32
llama_model_loader: - kv 7: llama.attention.head_count_kv u32 = 8
llama_model_loader: - kv 8: llama.rope.freq_base f32 = 1000000.000000
llama_model_loader: - kv 9: llama.attention.layer_norm_rms_epsilon f32 = 0.000010
llama_model_loader: - kv 10: general.file_type u32 = 2
llama_model_loader: - kv 11: llama.vocab_size u32 = 32768
llama_model_loader: - kv 12: llama.rope.dimension_count u32 = 128
llama_model_loader: - kv 13: tokenizer.ggml.model str = llama
llama_model_loader: - kv 14: tokenizer.ggml.pre str = default
llama_model_loader: - kv 15: tokenizer.ggml.tokens arr[str,32768] = ["<unk>", "<s>", "</s>", "[INST]", "[...
llama_model_loader: - kv 16: tokenizer.ggml.scores arr[f32,32768] = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv 17: tokenizer.ggml.token_type arr[i32,32768] = [2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
llama_model_loader: - kv 18: tokenizer.ggml.bos_token_id u32 = 1
llama_model_loader: - kv 19: tokenizer.ggml.eos_token_id u32 = 2
llama_model_loader: - kv 20: tokenizer.ggml.unknown_token_id u32 = 0
llama_model_loader: - kv 21: tokenizer.ggml.add_bos_token bool = true
llama_model_loader: - kv 22: tokenizer.ggml.add_eos_token bool = false
llama_model_loader: - kv 23: tokenizer.chat_template str = {{ bos_token }}{% for message in mess...
llama_model_loader: - kv 24: general.quantization_version u32 = 2
llama_model_loader: - type f32: 65 tensors
llama_model_loader: - type q4_0: 225 tensors
llama_model_loader: - type q6_K: 1 tensors
llm_load_vocab: special tokens cache size = 1027
llm_load_vocab: token to piece cache size = 0.3368 MB
llm_load_print_meta: format = GGUF V3 (latest)
llm_load_print_meta: arch = llama
llm_load_print_meta: vocab type = SPM
llm_load_print_meta: n_vocab = 32768
llm_load_print_meta: n_merges = 0
llm_load_print_meta: n_ctx_train = 32768
llm_load_print_meta: n_embd = 4096
llm_load_print_meta: n_head = 32
llm_load_print_meta: n_head_kv = 8
llm_load_print_meta: n_layer = 32
llm_load_print_meta: n_rot = 128
llm_load_print_meta: n_embd_head_k = 128
llm_load_print_meta: n_embd_head_v = 128
llm_load_print_meta: n_gqa = 4
llm_load_print_meta: n_embd_k_gqa = 1024
llm_load_print_meta: n_embd_v_gqa = 1024
llm_load_print_meta: f_norm_eps = 0.0e+00
llm_load_print_meta: f_norm_rms_eps = 1.0e-05
llm_load_print_meta: f_clamp_kqv = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale = 0.0e+00
llm_load_print_meta: n_ff = 14336
llm_load_print_meta: n_expert = 0
llm_load_print_meta: n_expert_used = 0
llm_load_print_meta: causal attn = 1
llm_load_print_meta: pooling type = 0
llm_load_print_meta: rope type = 0
llm_load_print_meta: rope scaling = linear
llm_load_print_meta: freq_base_train = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_yarn_orig_ctx = 32768
llm_load_print_meta: rope_finetuned = unknown
llm_load_print_meta: ssm_d_conv = 0
llm_load_print_meta: ssm_d_inner = 0
llm_load_print_meta: ssm_d_state = 0
llm_load_print_meta: ssm_dt_rank = 0
llm_load_print_meta: model type = 7B
llm_load_print_meta: model ftype = Q4_0
llm_load_print_meta: model params = 7.25 B
llm_load_print_meta: model size = 3.83 GiB (4.54 BPW)
llm_load_print_meta: general.name = Mistral-7B-Instruct-v0.3
llm_load_print_meta: BOS token = 1 '<s>'
llm_load_print_meta: EOS token = 2 '</s>'
llm_load_print_meta: UNK token = 0 '<unk>'
llm_load_print_meta: LF token = 781 '<0x0A>'
rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx1103
time=2024-06-10T06:05:48.352-07:00 level=INFO source=server.go:567 msg="waiting for server to become available" status="llm server not responding"
time=2024-06-10T06:05:48.603-07:00 level=ERROR source=sched.go:344 msg="error loading llama server" error="llama runner process has terminated: signal: segmentation fault (core dumped) error:Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx1103"
[GIN] 2024/06/10 - 06:05:48 | 500 | 812.059238ms | 127.0.0.1 | POST "/api/generate"Doing a "pacman -S ollama" to replace the rocm version installs fine and has the bad performance shown in the first post.
Offline
#5 2024-06-11 09:21:58
- Lone_Wolf
- Administrator
- From: Netherlands, Europe
- Registered: 2005-10-04
- Posts: 12,772
Re: Framework 13 with AMD running llms
time=2024-06-10T06:05:47.808-07:00 level=INFO source=amd_linux.go:233 msg="unsupported Radeon iGPU detected skipping" id=0 total="512.0 MiB"
time=2024-06-10T06:05:47.808-07:00 level=INFO source=amd_linux.go:311 msg="no compatible amdgpu devices detected"According to https://rocm.docs.amd.com/projects/rade … sites.html rocm doesn't support any integrated GPU .
ollama without rocm is CPU bound and cpus are not great at Machine Learning.
Maybe a dedicated gpu can be added to the laptop ?
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
#6 2024-06-12 03:00:53
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Re: Framework 13 with AMD running llms
I'm sure it would help. However, the benchmarks provided by others using the same laptops on other linux distros report 2x+ speeds. Their output also notes "gpu_info: no_gpu" in the output.
I did find this issue which seems to be similar to what I'm running into:
time=2024-06-05T10:56:17.756+02:00 level=WARN source=sched.go:512 msg="gpu VRAM usage didn't recover within timeout" seconds=5.279895057
time=2024-06-05T10:56:17.783+02:00 level=WARN source=amd_linux.go:48 msg="ollama recommends running the https://www.amd.com/en/support/linux-drivers" error="amdgpu version file missing: /sys/module/amdgpu/version stat /sys/module/amdgpu/version: no such file or directory"
time=2024-06-05T10:56:17.785+02:00 level=INFO sourc
Error: llama runner process has terminated: signal: segmentation faultThe recommendation from the Catchy folks seems to be to install the proprietary amdgpu drivers, but I'm not sure how. I've installed the "amdgpu-pro-installer-debug-23.40_1741713-1-x86_64.pkg.tar.zst" package from the amdgpu-pro-installer AUR and restarted but it doesn't seem to have worked:
$ glxinfo | grep "OpenGL vendor string"
(standard input):OpenGL vendor string: AMDThere are quite a few other packages that were built as part of this makepgk:
$ ls *.zst
amdgpu-pro-installer-debug-23.40_1741713-1-x86_64.pkg.tar.zst lib32-amdgpu-pro-oglp-23.40_1741713-1-x86_64.pkg.tar.zst
amdgpu-pro-oglp-23.40_1741713-1-x86_64.pkg.tar.zst lib32-vulkan-amdgpu-pro-23.40_1741713-1-x86_64.pkg.tar.zst
amf-amdgpu-pro-23.40_1741713-1-x86_64.pkg.tar.zst vulkan-amdgpu-pro-23.40_1741713-1-x86_64.pkg.tar.zstShould I install all of them? Or another one in particular? Or uninstall my current, open source drivers first? The wiki doesn't have many details.
By the way, @Lone_Wolf - thank you for keeping this thread going!
Last edited by pierrel (2024-06-12 03:01:40)
Offline
#7 2024-06-12 08:53:55
- Lone_Wolf
- Administrator
- From: Netherlands, Europe
- Registered: 2005-10-04
- Posts: 12,772
Re: Framework 13 with AMD running llms
It seems ollama tries to use HIP and for amd that's only present in rocm.
The proprietary drivers packaage you can download from amd do come with rocm and some other stuff that as far as I know is not included in the aur amd proprietary packages.
There may be another alternative though .
https://www.reddit.com/r/LocalLLaMA/com … work_with/ and https://www.reddit.com/r/LocalLLaMA/com … xtx_using/ indicate ollama can use OpenCL .
Install rocm-opencl-runtime & clinfo .
Post the output of clinfo .
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
#8 2024-06-14 19:27:58
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Re: Framework 13 with AMD running llms
Here's the out put of clinfo:
$ clinfo
Number of platforms 2
Platform Name AMD Accelerated Parallel Processing
Platform Vendor Advanced Micro Devices, Inc.
Platform Version OpenCL 2.1 AMD-APP.dbg (3602.0)
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_amd_event_callback
Platform Extensions function suffix AMD
Platform Host timer resolution 1ns
Platform Name Clover
Platform Vendor Mesa
Platform Version OpenCL 1.1 Mesa 24.1.1-arch1.1
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd
Platform Extensions function suffix MESA
Platform Name AMD Accelerated Parallel Processing
Number of devices 1
Device Name gfx1103
Device Vendor Advanced Micro Devices, Inc.
Device Vendor ID 0x1002
Device Version OpenCL 2.0
Driver Version 3602.0 (HSA1.1,LC)
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Board Name (AMD) AMD Radeon 780M
Device PCI-e ID (AMD) 0x15bf
Device Topology (AMD) PCI-E, 0000:ffffffc1:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 6
SIMD per compute unit (AMD) 4
SIMD width (AMD) 32
SIMD instruction width (AMD) 1
Max clock frequency 2700MHz
Graphics IP (AMD) 11.0
Device Partition (core)
Max number of sub-devices 6
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 1024x1024x1024
Max work group size 256
Preferred work group size (AMD) 256
Max work group size (AMD) 1024
Preferred work group size multiple (kernel) 32
Wavefront width (AMD) 32
Preferred / native vector sizes
char 4 / 4
short 2 / 2
int 1 / 1
long 1 / 1
half 1 / 1 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (cl_khr_fp16)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 536870912 (512MiB)
Global free memory (AMD) 428032 (418MiB) 428032 (418MiB)
Global memory channels (AMD) 2
Global memory banks per channel (AMD) 4
Global memory bank width (AMD) 256 bytes
Error Correction support No
Max memory allocation 402653184 (384MiB)
Unified memory for Host and Device No
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing Yes
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Preferred alignment for atomics
SVM 0 bytes
Global 0 bytes
Local 0 bytes
Max size for global variable 402653184 (384MiB)
Preferred total size of global vars 536870912 (512MiB)
Global Memory cache type Read/Write
Global Memory cache size 32768 (32KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 8192 images
Base address alignment for 2D image buffers 256 bytes
Pitch alignment for 2D image buffers 256 pixels
Max 2D image size 16384x16384 pixels
Max 3D image size 16384x16384x8192 pixels
Max number of read image args 128
Max number of write image args 8
Max number of read/write image args 64
Max number of pipe args 16
Max active pipe reservations 16
Max pipe packet size 402653184 (384MiB)
Local memory type Local
Local memory size 65536 (64KiB)
Local memory size per CU (AMD) 65536 (64KiB)
Local memory banks (AMD) 32
Max number of constant args 8
Max constant buffer size 402653184 (384MiB)
Preferred constant buffer size (AMD) 16384 (16KiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution No
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 262144 (256KiB)
Max size 8388608 (8MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Number of P2P devices (AMD) 0
Profiling timer resolution 1ns
Profiling timer offset since Epoch (AMD) 0ns (Wed Dec 31 16:00:00 1969)
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Thread trace supported (AMD) No
Number of async queues (AMD) 8
Max real-time compute queues (AMD) 8
Max real-time compute units (AMD) 6
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program
Platform Name Clover
Number of devices 1
Device Name AMD Radeon 780M (radeonsi, gfx1103_r1, LLVM 17.0.6, DRM 3.57, 6.9.3-arch1-1)
Device Vendor AMD
Device Vendor ID 0x1002
Device Version OpenCL 1.1 Mesa 24.1.1-arch1.1
Device Numeric Version 0x401000 (1.1.0)
Driver Version 24.1.1-arch1.1
Device OpenCL C Version OpenCL C 1.1
Device OpenCL C Numeric Version 0x401000 (1.1.0)
Device Type GPU
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Max compute units 12
Max clock frequency 2700MHz
Max work item dimensions 3
Max work item sizes 256x256x256
Max work group size 256
=== CL_PROGRAM_BUILD_LOG ===
fatal error: cannot open file '/usr/share/clc/gfx1103-amdgcn-mesa-mesa3d.bc': No such file or directory
Preferred work group size multiple (kernel) <getWGsizes:1980: create kernel : error -46>
Preferred / native vector sizes
char 16 / 16
short 8 / 8
int 4 / 4
long 2 / 2
half 0 / 0 (n/a)
float 4 / 4
double 2 / 2 (cl_khr_fp64)
Half-precision Floating-point support (n/a)
Single-precision Floating-point support (core)
Denormals No
Infinity and NANs Yes
Round to nearest Yes
Round to zero No
Round to infinity No
IEEE754-2008 fused multiply-add No
Support is emulated in software No
Correctly-rounded divide and sqrt operations No
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 16461004800 (15.33GiB)
Error Correction support No
Max memory allocation 4115251200 (3.833GiB)
Unified memory for Host and Device No
Minimum alignment for any data type 128 bytes
Alignment of base address 32768 bits (4096 bytes)
Global Memory cache type None
Image support No
Local memory type Local
Local memory size 65536 (64KiB)
Max number of constant args 16
Max constant buffer size 67108864 (64MiB)
Max size of kernel argument 1024
Queue properties
Out-of-order execution No
Profiling Yes
Profiling timer resolution 0ns
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
ILs with version SPIR-V 0x400000 (1.0.0)
Built-in kernels with version (n/a)
Device Extensions cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64 cl_khr_extended_versioning
Device Extensions with Version cl_khr_byte_addressable_store 0x400000 (1.0.0)
cl_khr_global_int32_base_atomics 0x400000 (1.0.0)
cl_khr_global_int32_extended_atomics 0x400000 (1.0.0)
cl_khr_local_int32_base_atomics 0x400000 (1.0.0)
cl_khr_local_int32_extended_atomics 0x400000 (1.0.0)
cl_khr_int64_base_atomics 0x400000 (1.0.0)
cl_khr_int64_extended_atomics 0x400000 (1.0.0)
cl_khr_fp64 0x400000 (1.0.0)
cl_khr_extended_versioning 0x400000 (1.0.0)
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [AMD]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103Another thing to note, when trying to copy to xclip, I receive the following error (that doesn't show when not trying to pipe):
$ clinfo | xclip
=== CL_PROGRAM_BUILD_LOG ===
fatal error: cannot open file '/usr/share/clc/gfx1103-amdgcn-mesa-mesa3d.bc': No such file or directoryAfter installing these (no restart) and running llm_benchmark again I still get poor performance.
Offline
#9 2024-06-14 21:08:13
- Lone_Wolf
- Administrator
- From: Netherlands, Europe
- Registered: 2005-10-04
- Posts: 12,772
Re: Framework 13 with AMD running llms
Clover relies on libclc and support for cards above gfx9 was only added in libclc 18 and later .
Even if you ran those versions, clover is unlikely to provide what ollama needs .
remove opencl-clover-mesa.
run clinfo --list to verify you have 1 platform (AMD Accelerated Parallel Processing) with 1 device .
Then try ollama again.
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
#10 2024-07-12 15:31:33
- pierrel
- Member
- Registered: 2024-06-08
- Posts: 6
Re: Framework 13 with AMD running llms
Thanks again for your help here. I'm still not getting better performance after following your steps. Here's clinfo again:
$ clinfo
Number of platforms 1
Platform Name AMD Accelerated Parallel Processing
Platform Vendor Advanced Micro Devices, Inc.
Platform Version OpenCL 2.1 AMD-APP.dbg (3602.0)
Platform Profile FULL_PROFILE
Platform Extensions cl_khr_icd cl_amd_event_callback
Platform Extensions function suffix AMD
Platform Host timer resolution 1ns
Platform Name AMD Accelerated Parallel Processing
Number of devices 1
Device Name gfx1103
Device Vendor Advanced Micro Devices, Inc.
Device Vendor ID 0x1002
Device Version OpenCL 2.0
Driver Version 3602.0 (HSA1.1,LC)
Device OpenCL C Version OpenCL C 2.0
Device Type GPU
Device Board Name (AMD) AMD Radeon 780M
Device PCI-e ID (AMD) 0x15bf
Device Topology (AMD) PCI-E, 0000:ffffffc1:00.0
Device Profile FULL_PROFILE
Device Available Yes
Compiler Available Yes
Linker Available Yes
Max compute units 6
SIMD per compute unit (AMD) 4
SIMD width (AMD) 32
SIMD instruction width (AMD) 1
Max clock frequency 2700MHz
Graphics IP (AMD) 11.0
Device Partition (core)
Max number of sub-devices 6
Supported partition types None
Supported affinity domains (n/a)
Max work item dimensions 3
Max work item sizes 1024x1024x1024
Max work group size 256
Preferred work group size (AMD) 256
Max work group size (AMD) 1024
Preferred work group size multiple (kernel) 32
Wavefront width (AMD) 32
Preferred / native vector sizes
char 4 / 4
short 2 / 2
int 1 / 1
long 1 / 1
half 1 / 1 (cl_khr_fp16)
float 1 / 1
double 1 / 1 (cl_khr_fp64)
Half-precision Floating-point support (cl_khr_fp16)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Single-precision Floating-point support (core)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Correctly-rounded divide and sqrt operations Yes
Double-precision Floating-point support (cl_khr_fp64)
Denormals Yes
Infinity and NANs Yes
Round to nearest Yes
Round to zero Yes
Round to infinity Yes
IEEE754-2008 fused multiply-add Yes
Support is emulated in software No
Address bits 64, Little-Endian
Global memory size 16460988416 (15.33GiB)
Global free memory (AMD) 15839196 (15.11GiB) 15839196 (15.11GiB)
Global memory channels (AMD) 2
Global memory banks per channel (AMD) 4
Global memory bank width (AMD) 256 bytes
Error Correction support No
Max memory allocation 13991840152 (13.03GiB)
Unified memory for Host and Device No
Shared Virtual Memory (SVM) capabilities (core)
Coarse-grained buffer sharing Yes
Fine-grained buffer sharing Yes
Fine-grained system sharing No
Atomics No
Minimum alignment for any data type 128 bytes
Alignment of base address 1024 bits (128 bytes)
Preferred alignment for atomics
SVM 0 bytes
Global 0 bytes
Local 0 bytes
Max size for global variable 13991840152 (13.03GiB)
Preferred total size of global vars 16460988416 (15.33GiB)
Global Memory cache type Read/Write
Global Memory cache size 32768 (32KiB)
Global Memory cache line size 64 bytes
Image support Yes
Max number of samplers per kernel 16
Max size for 1D images from buffer 134217728 pixels
Max 1D or 2D image array size 8192 images
Base address alignment for 2D image buffers 256 bytes
Pitch alignment for 2D image buffers 256 pixels
Max 2D image size 16384x16384 pixels
Max 3D image size 16384x16384x8192 pixels
Max number of read image args 128
Max number of write image args 8
Max number of read/write image args 64
Max number of pipe args 16
Max active pipe reservations 16
Max pipe packet size 1106938264 (1.031GiB)
Local memory type Local
Local memory size 65536 (64KiB)
Local memory size per CU (AMD) 65536 (64KiB)
Local memory banks (AMD) 32
Max number of constant args 8
Max constant buffer size 13991840152 (13.03GiB)
Preferred constant buffer size (AMD) 16384 (16KiB)
Max size of kernel argument 1024
Queue properties (on host)
Out-of-order execution No
Profiling Yes
Queue properties (on device)
Out-of-order execution Yes
Profiling Yes
Preferred size 262144 (256KiB)
Max size 8388608 (8MiB)
Max queues on device 1
Max events on device 1024
Prefer user sync for interop Yes
Number of P2P devices (AMD) 0
Profiling timer resolution 1ns
Profiling timer offset since Epoch (AMD) 0ns (Wed Dec 31 16:00:00 1969)
Execution capabilities
Run OpenCL kernels Yes
Run native kernels No
Thread trace supported (AMD) No
Number of async queues (AMD) 8
Max real-time compute queues (AMD) 8
Max real-time compute units (AMD) 6
printf() buffer size 4194304 (4MiB)
Built-in kernels (n/a)
Device Extensions cl_khr_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_media_ops cl_amd_media_ops2 cl_khr_image2d_from_buffer cl_khr_subgroups cl_khr_depth_images cl_amd_copy_buffer_p2p cl_amd_assembly_program
NULL platform behavior
clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform
clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform
clCreateContext(NULL, ...) [default] No platform
clCreateContext(NULL, ...) [other] Success [AMD]
clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No devices found in platform
clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) Success (1)
Platform Name AMD Accelerated Parallel Processing
Device Name gfx1103And here's the output of another llm_benchmark run I did this morning:
{
"mistral:7b": "6.06",
"llama3:8b": "5.65",
"phi3:3.8b": "10.02",
"gemma:2b": "13.65",
"gemma:7b": "4.88",
"llava:7b": "6.22",
"llava:13b": "3.45",
"uuid": "a79f8302-d19d-5b65-a214-213535d44a16",
"ollama_version": "0.1.48"
}
----------
====================
-------Linux----------
No GPU detected.
-------Linux----------
No GPU detected.
{
"system": "Linux",
"memory": 30.66098403930664,
"cpu": "AMD Ryzen 7 7840U w/ Radeon 780M Graphics",
"gpu": "no_gpu",
"os_version": "\"Arch Linux\"",
"system_name": "Linux",
"uuid": "a79f8302-d19d-5b65-a214-213535d44a16"
}
----------Offline
#11 2024-08-04 06:41:32
- lsr
- Member
- Registered: 2016-12-31
- Posts: 3
Re: Framework 13 with AMD running llms
Have you set HSA_OVERRIDE_GFX_VERSION ?
Support for the APUs isn't compiled in everywhere, though you can override the architecture and use the general support for the family (RDNA 3 in this case).
For ollama you can use the systemctl override trick (# systemctl edit ollama)
cat /etc/systemd/system/ollama.service.d/override.conf
[Service]
Environment="HSA_OVERRIDE_GFX_VERSION=11.0.0"The same env var is useful for many parts of the stack. It seems AMD is working towards some kind of generic architecture support, but until then we have this.
Note that kernel 6.10 finally made memory allocation on AMD APUs use GTT if necessary. That is, it actually flexibly uses main memory at runtime. Ollama still tries to use available VRAM to decide how much to offload to GPU, so it'll only use a tiny subset.
On the commandline in ollama you can override the number of layers by doing
/set parameter num_gpu NUM_LAYERSThe ollama logs will show the total layers the model has, suggest using it if you have enough main memory for a given model.
Offline
Pages: 1