rc/dela
Watch
1
0
Fork
You've already forked dela
 0
Code Issues 1 Pull requests Projects Releases 2 Packages Wiki Activity Actions

integrate GLM-OCR to pre-populate /receipts/new #3

New issue
Open
opened 2026-07-20 20:15:50 +00:00 by rc · 1 comment
rc commented 2026-07-20 20:15:50 +00:00
Owner
Copy link

maybe use telegram bot as interface?

maybe use telegram bot as interface?
rc commented 2026-07-29 15:12:13 +00:00
Author
Owner
Copy link

start llama-server with GLM-OCR-GGUF

llama-server -hf ggml-org/GLM-OCR-GGUF`

run the following:

export {};

const prompt = `
read this restaurant receipt. extract all ordered items. return ONLY valid JSON using exactly this format: {"items": ["quantity": 1, "description": "Item name", "price": 250}]}
rules:
- return JSON only, no markdown
- quantity is the number of units ordered
- description is the item name exactly as printed
- price is the unit price in cents(for example 250 means 2.50)
- do not include totals, taxes, table numbers, dates, or explanations
`;

const payload = {
	model: "glm-ocr",
	temperature: 0,
	response_format: { type: "json_object" },
	messages: [
		{
			role: "user",
			content: [
				{ type: "text", text: prompt },
				{ type: "image_url", image_url: { url: "https://f.rcastellotti.dev/tmp/dela/toast.jpg" } },
			],
		},
	],
};

const response = await fetch("http://localhost:8080/v1/chat/completions", {
	method: "POST",
	headers: { "Content-Type": "application/json" },
	body: JSON.stringify(payload),
});

if (!response.ok) {
	console.error(await response.text());
	process.exit(1);
}

const result = await response.json();
console.log(result.choices[0].message.content);

outputs:

```json
{
    "items": [
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "BIRRA CL.60 BECK'S BOTT.",
            "price": 250.0
        },
        {
            "quantity": 1,
            "description": "TORTILLA CHIPS PAQUITA G.4",
            "price": 229.0
        },
        {
            "quantity": 1,
            "description": "SALAME MILANO AFFETTATO G.",
            "price": 229.0
        },
        {
            "quantity": 1,
            "description": "SALAME MILANO AFFETTATO G.",
            "price": 229.0
        },
        {
            "quantity": 1,
            "description": "PROSCIUTTO CRUDO STELLA G.",
            "price": 399.0
        },
        {
            "quantity": 1,
            "description": "PROSCIUTTO CRUDO STELLA G.",
            "price": 399.0
        },
        {
            "quantity": 1,
            "description": "PROSCIUTTO COTTO A.Q.G.180",
            "price": 339.0
        },
        {
            "quantity": 1,
            "description": "PROSCIUTTO COTTO A.Q.G.180",
            "price": 339.0
        },
        {
            "quantity": 1,
            "description": "PROSCIUTTO COTTO A.Q.G.180",
            "price": 339.0
        },
        {
            "quantity": 1,
            "description": "AMERICAN SANDWICH G.550 PR",
            "price": 118.0
        },
        {
            "quantity": 1,
            "description": "AMERICAN SANDWICH G.550 PR",
            "price": 118.0
        },
        {
            "quantity": 1,
            "description": "ICEBERG G.350",
            "price": 189.0
        },
        {
            "quantity": 1,
            "description": "POMODORO CUORE DI BUE QUAL",
            "price": 112.0
        },
        {
            "quantity": 1,
            "description": "FORM.FUSO FETTE G.400 PRIM",
            "price": 279.0
        },
        {
            "quantity": 1,
            "description": "FORM.FUSO FETTE G.400 PRIM",
            "price": 279.0
        },
        {
            "quantity": 1,
            "description": "MAIONESE ORIG.ML.400/G.395",
            "price": 349.0
        },
        {
            "quantity": 1,
            "description": "FANTA STRAWBERRY+KIWI CL.3",
            "price": 159.0
        },
        {
            "quantity": 1,
            "description": "NUTELLA G.400",
            "price": 399.0
        }
    ]
}
```

rc@den ~/g/dela (main) [1]> llama-server -hf ggml-org/GLM-OCR-GGUF
0.00.351.674 I cmn  common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg)
0.00.353.309 I srv    load_model: loading model 'ggml-org/GLM-OCR-GGUF'
0.00.529.470 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect
0.00.541.789 W model has unused tensor blk.16.attn_norm.weight (size = 6144 bytes) -- ignoring
0.00.541.794 W model has unused tensor blk.16.attn_q.weight (size = 3342336 bytes) -- ignoring
0.00.541.795 W model has unused tensor blk.16.attn_k.weight (size = 1671168 bytes) -- ignoring
0.00.541.797 W model has unused tensor blk.16.attn_v.weight (size = 1671168 bytes) -- ignoring
0.00.541.802 W model has unused tensor blk.16.attn_output.weight (size = 3342336 bytes) -- ignoring
0.00.541.804 W model has unused tensor blk.16.post_attention_norm.weight (size = 6144 bytes) -- ignoring
0.00.541.805 W model has unused tensor blk.16.ffn_norm.weight (size = 6144 bytes) -- ignoring
0.00.541.807 W model has unused tensor blk.16.ffn_down.weight (size = 7520256 bytes) -- ignoring
0.00.541.808 W model has unused tensor blk.16.ffn_up.weight (size = 15040512 bytes) -- ignoring
0.00.541.810 W model has unused tensor blk.16.post_ffw_norm.weight (size = 6144 bytes) -- ignoring
0.00.541.813 W model has unused tensor blk.16.nextn.eh_proj.weight (size = 5013504 bytes) -- ignoring
0.00.541.815 W model has unused tensor blk.16.nextn.enorm.weight (size = 6144 bytes) -- ignoring
0.00.541.817 W model has unused tensor blk.16.nextn.hnorm.weight (size = 6144 bytes) -- ignoring
0.00.541.819 W model has unused tensor blk.16.nextn.embed_tokens.weight (size = 96927744 bytes) -- ignoring
0.00.541.820 W model has unused tensor blk.16.nextn.shared_head_head.weight (size = 96927744 bytes) -- ignoring
0.00.541.822 W model has unused tensor blk.16.nextn.shared_head_norm.weight (size = 6144 bytes) -- ignoring
0.04.135.603 I srv    load_model: loaded multimodal model, '/home/rc/.cache/huggingface/hub/models--ggml-org--GLM-OCR-GGUF/snapshots/65a42de1148dbed2297e922b5dbc7d9b70c36578/mmproj-GLM-OCR-Q8_0.gguf'
0.04.785.770 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 131072, kv_unified = 'true'
0.04.789.895 I srv  llama_server: model loaded
0.04.789.899 I srv  llama_server: listening on http://127.0.0.1:8080
0.07.712.402 I srv  handle_media: downloading image from 'https://f.rcastellotti.dev/tmp/dela/toast.jpg'
0.08.258.809 I srv  handle_media: downloaded 218662 bytes
0.08.310.348 I slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
0.08.310.385 I slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
0.40.160.396 I slot print_timing: id  3 | task 0 | n_decoded =    161, tg =  53.35 t/s, tg_3s =  53.35 t/s
0.43.167.505 I slot print_timing: id  3 | task 0 | n_decoded =    314, tg =  52.12 t/s, tg_3s =  50.88 t/s
0.46.169.087 I slot print_timing: id  3 | task 0 | n_decoded =    465, tg =  51.51 t/s, tg_3s =  50.31 t/s
0.49.171.372 I slot print_timing: id  3 | task 0 | n_decoded =    611, tg =  50.79 t/s, tg_3s =  48.63 t/s
0.52.171.898 I slot print_timing: id  3 | task 0 | n_decoded =    753, tg =  50.10 t/s, tg_3s =  47.33 t/s
0.55.184.305 I slot print_timing: id  3 | task 0 | n_decoded =    893, tg =  49.50 t/s, tg_3s =  46.47 t/s
0.58.204.352 I slot print_timing: id  3 | task 0 | n_decoded =   1029, tg =  48.86 t/s, tg_3s =  45.03 t/s
1.00.595.289 I slot print_timing: id  3 | task 0 | prompt eval time =   28831.99 ms /  1824 tokens (   15.81 ms per token,    63.26 tokens per second)
1.00.595.294 I slot print_timing: id  3 | task 0 |        eval time =   23452.76 ms /  1135 tokens (   20.66 ms per token,    48.40 tokens per second)
1.00.595.295 I slot print_timing: id  3 | task 0 |       total time =   52284.74 ms /  2959 tokens
1.00.595.296 I slot print_timing: id  3 | task 0 |    graphs reused =       1130
1.00.595.319 I slot      release: id  3 | task 0 | stop processing: n_tokens = 2958, truncated = 0
start [`llama-server`](https://github.com/ggml-org/llama.cpp) with [GLM-OCR-GGUF](https://huggingface.co/ggml-org/GLM-OCR-GGUF) ```sh llama-server -hf ggml-org/GLM-OCR-GGUF` ``` run the following: ```ts export {}; const prompt = ` read this restaurant receipt. extract all ordered items. return ONLY valid JSON using exactly this format: {"items": ["quantity": 1, "description": "Item name", "price": 250}]} rules: - return JSON only, no markdown - quantity is the number of units ordered - description is the item name exactly as printed - price is the unit price in cents(for example 250 means 2.50) - do not include totals, taxes, table numbers, dates, or explanations `; const payload = { model: "glm-ocr", temperature: 0, response_format: { type: "json_object" }, messages: [ { role: "user", content: [ { type: "text", text: prompt }, { type: "image_url", image_url: { url: "https://f.rcastellotti.dev/tmp/dela/toast.jpg" } }, ], }, ], }; const response = await fetch("http://localhost:8080/v1/chat/completions", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(payload), }); if (!response.ok) { console.error(await response.text()); process.exit(1); } const result = await response.json(); console.log(result.choices[0].message.content); ``` outputs: ```` ```json { "items": [ { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "BIRRA CL.60 BECK'S BOTT.", "price": 250.0 }, { "quantity": 1, "description": "TORTILLA CHIPS PAQUITA G.4", "price": 229.0 }, { "quantity": 1, "description": "SALAME MILANO AFFETTATO G.", "price": 229.0 }, { "quantity": 1, "description": "SALAME MILANO AFFETTATO G.", "price": 229.0 }, { "quantity": 1, "description": "PROSCIUTTO CRUDO STELLA G.", "price": 399.0 }, { "quantity": 1, "description": "PROSCIUTTO CRUDO STELLA G.", "price": 399.0 }, { "quantity": 1, "description": "PROSCIUTTO COTTO A.Q.G.180", "price": 339.0 }, { "quantity": 1, "description": "PROSCIUTTO COTTO A.Q.G.180", "price": 339.0 }, { "quantity": 1, "description": "PROSCIUTTO COTTO A.Q.G.180", "price": 339.0 }, { "quantity": 1, "description": "AMERICAN SANDWICH G.550 PR", "price": 118.0 }, { "quantity": 1, "description": "AMERICAN SANDWICH G.550 PR", "price": 118.0 }, { "quantity": 1, "description": "ICEBERG G.350", "price": 189.0 }, { "quantity": 1, "description": "POMODORO CUORE DI BUE QUAL", "price": 112.0 }, { "quantity": 1, "description": "FORM.FUSO FETTE G.400 PRIM", "price": 279.0 }, { "quantity": 1, "description": "FORM.FUSO FETTE G.400 PRIM", "price": 279.0 }, { "quantity": 1, "description": "MAIONESE ORIG.ML.400/G.395", "price": 349.0 }, { "quantity": 1, "description": "FANTA STRAWBERRY+KIWI CL.3", "price": 159.0 }, { "quantity": 1, "description": "NUTELLA G.400", "price": 399.0 } ] } ``` ```` ``` rc@den ~/g/dela (main) [1]> llama-server -hf ggml-org/GLM-OCR-GGUF 0.00.351.674 I cmn common_param: common_params_print_info: verbosity = 3 (adjust with the `-lv N` CLI arg) 0.00.353.309 I srv load_model: loading model 'ggml-org/GLM-OCR-GGUF' 0.00.529.470 W load: special_eot_id is not in special_eog_ids - the tokenizer config may be incorrect 0.00.541.789 W model has unused tensor blk.16.attn_norm.weight (size = 6144 bytes) -- ignoring 0.00.541.794 W model has unused tensor blk.16.attn_q.weight (size = 3342336 bytes) -- ignoring 0.00.541.795 W model has unused tensor blk.16.attn_k.weight (size = 1671168 bytes) -- ignoring 0.00.541.797 W model has unused tensor blk.16.attn_v.weight (size = 1671168 bytes) -- ignoring 0.00.541.802 W model has unused tensor blk.16.attn_output.weight (size = 3342336 bytes) -- ignoring 0.00.541.804 W model has unused tensor blk.16.post_attention_norm.weight (size = 6144 bytes) -- ignoring 0.00.541.805 W model has unused tensor blk.16.ffn_norm.weight (size = 6144 bytes) -- ignoring 0.00.541.807 W model has unused tensor blk.16.ffn_down.weight (size = 7520256 bytes) -- ignoring 0.00.541.808 W model has unused tensor blk.16.ffn_up.weight (size = 15040512 bytes) -- ignoring 0.00.541.810 W model has unused tensor blk.16.post_ffw_norm.weight (size = 6144 bytes) -- ignoring 0.00.541.813 W model has unused tensor blk.16.nextn.eh_proj.weight (size = 5013504 bytes) -- ignoring 0.00.541.815 W model has unused tensor blk.16.nextn.enorm.weight (size = 6144 bytes) -- ignoring 0.00.541.817 W model has unused tensor blk.16.nextn.hnorm.weight (size = 6144 bytes) -- ignoring 0.00.541.819 W model has unused tensor blk.16.nextn.embed_tokens.weight (size = 96927744 bytes) -- ignoring 0.00.541.820 W model has unused tensor blk.16.nextn.shared_head_head.weight (size = 96927744 bytes) -- ignoring 0.00.541.822 W model has unused tensor blk.16.nextn.shared_head_norm.weight (size = 6144 bytes) -- ignoring 0.04.135.603 I srv load_model: loaded multimodal model, '/home/rc/.cache/huggingface/hub/models--ggml-org--GLM-OCR-GGUF/snapshots/65a42de1148dbed2297e922b5dbc7d9b70c36578/mmproj-GLM-OCR-Q8_0.gguf' 0.04.785.770 I srv load_model: initializing, n_slots = 4, n_ctx_slot = 131072, kv_unified = 'true' 0.04.789.895 I srv llama_server: model loaded 0.04.789.899 I srv llama_server: listening on http://127.0.0.1:8080 0.07.712.402 I srv handle_media: downloading image from 'https://f.rcastellotti.dev/tmp/dela/toast.jpg' 0.08.258.809 I srv handle_media: downloaded 218662 bytes 0.08.310.348 I slot get_availabl: id 3 | task -1 | selected slot by LRU, t_last = -1 0.08.310.385 I slot launch_slot_: id 3 | task 0 | processing task, is_child = 0 0.40.160.396 I slot print_timing: id 3 | task 0 | n_decoded = 161, tg = 53.35 t/s, tg_3s = 53.35 t/s 0.43.167.505 I slot print_timing: id 3 | task 0 | n_decoded = 314, tg = 52.12 t/s, tg_3s = 50.88 t/s 0.46.169.087 I slot print_timing: id 3 | task 0 | n_decoded = 465, tg = 51.51 t/s, tg_3s = 50.31 t/s 0.49.171.372 I slot print_timing: id 3 | task 0 | n_decoded = 611, tg = 50.79 t/s, tg_3s = 48.63 t/s 0.52.171.898 I slot print_timing: id 3 | task 0 | n_decoded = 753, tg = 50.10 t/s, tg_3s = 47.33 t/s 0.55.184.305 I slot print_timing: id 3 | task 0 | n_decoded = 893, tg = 49.50 t/s, tg_3s = 46.47 t/s 0.58.204.352 I slot print_timing: id 3 | task 0 | n_decoded = 1029, tg = 48.86 t/s, tg_3s = 45.03 t/s 1.00.595.289 I slot print_timing: id 3 | task 0 | prompt eval time = 28831.99 ms / 1824 tokens ( 15.81 ms per token, 63.26 tokens per second) 1.00.595.294 I slot print_timing: id 3 | task 0 | eval time = 23452.76 ms / 1135 tokens ( 20.66 ms per token, 48.40 tokens per second) 1.00.595.295 I slot print_timing: id 3 | task 0 | total time = 52284.74 ms / 2959 tokens 1.00.595.296 I slot print_timing: id 3 | task 0 | graphs reused = 1130 1.00.595.319 I slot release: id 3 | task 0 | stop processing: n_tokens = 2958, truncated = 0 ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
elisa
2.0
1.0
Labels
Clear labels   
bug
Something is not working

  
duplicate
This issue or pull request already exists

  
enhancement
New feature

  
help wanted
Need some help

  
invalid
Something is wrong

  
question
More information is needed

  
wontfix
This won't be fixed

No labels
bug
duplicate
enhancement
help wanted
invalid
question
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
rc/dela#3
No description provided.