2026/9/17 2:52:46
FEATURED · 精选文章 
omiGlass 多视觉模型图像描述流水线基于 Ollama 的看图评测样本解析与复现【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend本文以 omiGlass 项目中 img_44.md 这一份真实的多模型图像描述输出为样本结合omiGlass/sources下的源码系统讲解 omiGlass 如何通过 Ollama 本地部署多个视觉语言模型VLM对摄像头拍摄的图片进行看图说话、批量生成评测数据并最终把图像描述接入问答 Agent 的完整链路。读完本文你将掌握 omiGlass 图像描述链路的调用实现、模型选型、环境配置与本地复现方法并能独立用同一套流水线评测任意图片在多模型下的理解表现。一、样本是什么img_44.md 的内容结构omiGlass/prompts/series_1/img_44.md是series_1评测集中第 44 张测试图片的多模型图像描述输出文件。它本身不是配置文档而是批量评测脚本自动生成的模型答卷文件内按####标题####分节保存了 4 个视觉模型对同一张图片的英文描述。原文档的完整内容如下####Description#### The image features a group of people standing in an open space, possibly inside a building or near a bridge. There are at least five individuals visible in the scene, with one person holding a drink and another person looking up towards the sky. The atmosphere appears to be casual and relaxed as they engage in conversation or observe their surroundings. ####Description (llava-llama3)#### The image captures a lively scene at an indoor event. The focal point is a white archway, adorned with four windows, each fitted with metal grilles. The archway stands against the backdrop of a white ceiling, punctuated by a skylight that allows natural light to filter in. A group of people are gathered around the archway, their attention drawn to something or someone outside the frame. Their curiosity is palpable as they stand close together, forming a semi-circle that suggests an intimate gathering or discussion. Adding to the ambiance of the scene is a white air conditioning unit nestled near the ceiling, its presence subtly reminding us of the indoor setting. Despite being an inanimate object, it seems to be contributing to the overall atmosphere of the event. The image exudes a sense of anticipation and camaraderie, as if everyone present is waiting for something significant to happen or someone to arrive. The precise positioning of the objects and people, along with their actions, paints a vivid picture of an indoor gathering filled with anticipation and conversation. ####Description (llava:34b-v1.6)#### In the image, you can see an interior scene with architectural features that suggest it could be a building with historical significance. The ceiling is adorned with exposed beams and large windows allowing natural light to enter the space. There are people present in the scene; one person appears to be seated at what looks like a desk or workstation on the right side of the image, while another person stands slightly behind them. Both individuals seem engaged in an activity that could involve paperwork or some form of documentation, given the presence of papers and a pen. The quality of the photo is somewhat grainy, possibly due to lighting conditions, camera settings, or motion blur. Overall, the scene appears casual yet focused, with people interacting within a space designed for work or study. ####Description (moondream:1.8b-v2-fp16)#### In the image, there is a group of people gathered together in what appears to be an indoor space. The individuals are standing next to each other and engaged in conversation or interaction. Among them, one person seems to be drinking from a cup while others observe the scene. Theres also a bottle visible on a surface nearby. The setting features an arched ceiling with large windows that allow natural light to enter the room. A refrigerator is located at the edge of the space, indicating it might be in a cafe or similar establishment where people often gather and socialize.可以看到四个模型对同一张室内人群场景的感知各有侧重模型场景判定人物描述细节捕获Description默认链路室内或桥边开放空间至少 5 人有人持饮料、有人仰望天空氛围轻松随意偏宏观llava-llama3室内活动白色拱门 4 扇金属格栅窗半圆形聚集、充满好奇天窗、空调内机等建筑细节llava:34b-v1.6有历史感的建筑室内一坐一立、疑似处理文件横梁天花板、纸张与笔且指出画面颗粒感moondream:1.8b-v2-fp16咖啡馆类社交场所站立交谈一人持杯桌面上的瓶子、角落的冰箱这份样本的价值在于同一 prompt、同一张图、不同模型的输出差异本身就是模型能力评估的原始证据——有的模型擅长建筑结构感知llava 系列有的模型擅长氛围与人物关系推断moondream而转写画面文字、识别物体类别等能力需要逐项对照验证。二、样本从哪来generate.ts 批量生成脚本omiGlass/prompts/generate.ts是产生这类.md文件的批处理脚本其工作流程是遍历系列目录读取prompts/下每个子目录如series_1收集所有.jpeg图片文件并把输出路径定为同名.mdreplace(.jpeg, .md)见 generate.ts逐模型跑测试对每一张图片依次执行 4 轮runTest分别调用imageDescription模型依次为默认模型、llava-llama3、llava:34b-v1.6、moondream:1.8b-v2-fp16并把结果以####标题####分节追加到输出字符串见 generate.ts写回文件全部模型跑完后把拼接结果整体写入对应.md形成 img_44.md 这样的多模型对比报告见 generate.ts。脚本中还保留了被注释的imageBlurry模糊度检测扩展点说明该评测体系的设计意图是支持更多图像质量维度的自动化检测。三、核心实现imageDescription 的 prompt 设计与默认模型描述能力由 imageDescription.ts 提供核心函数签名如下export async function imageDescription(src: Uint8Array, model: KnownModel moondream:1.8b-v2-fp16): Promisestring函数向 Ollama 发送两条消息system 指令You are a very advanced model and your task is to describe the image as precisely as possible. Transcribe any text you see.——明确要求尽可能精确地描述图像并转写画面中出现的任何文字。这条指令解释了为什么描述里会出现窗户带金属格栅exposed beams这类精确用词系统希望模型做客观转写而非泛泛而谈。user 消息Describe the scene并将图片以Uint8Array二进制形式挂到消息的images字段。默认模型是moondream:1.8b-v2-fp16这与 README 中安装模型的命令ollama pull moondream:1.8b-v2-fp16一致见 README.md。moondream 是一个轻量视觉语言模型fp16 权重版本更适合在本地设备上以较低显存运行符合 omiGlass 强调本地化、低功耗的定位。四、模型矩阵与底层调用链ollamaInference 做了什么可选的模型集合定义在 ollama.ts 的KnownModel联合类型中export type KnownModel | llama3 | llama3-gradient | llama3:8b-instruct-fp16 | llava-llama3 | llava:34b-v1.6 | moondream:1.8b-v2-fp16 | moondream:1.8b-v2-moondream2-text-model-f16ollamaInference是该流水线的传输层见 ollama.ts关键实现细节图片编码将每条消息中的Uint8Array图片通过toBase64转为 base64 字符串后放入images字段这是 Ollama/api/chat接口的标准图片输入格式请求构造axios.post(keys.ollama, { stream: false, model, messages })服务地址来自keys.ollama非流式一次拿到完整回复重试机制整个请求被backoff包装网络抖动或模型加载时自动退避重试提升批处理稳定性文本规整trimIdent用于清理消息内容与回复中的多余缩进空白保证写入.md的描述文本干净可读。API 地址与密钥集中在 keys.tsexport const keys { groq: process.env.EXPO_PUBLIC_GROQ_API_KEY ?? , ollama: process.env.EXPO_PUBLIC_OLLAMA_API_URL ?? , openai: process.env.EXPO_PUBLIC_OPENAI_API_KEY ?? , };其中ollama对应 Ollama 本地服务的http://localhost:11434/api/chat默认地址其余两个分别为 Groq 与 OpenAI 的密钥说明该流水线同时保留了云端 LLM 的接入通道。五、从看图说话到问答 Agent描述如何被消费图像描述并不是终点而是 omiGlass Agent 的输入原料。Agent.ts 的addPhoto方法在加锁状态下逐张调用imageDescription(p)把{ photo, description }存入内存照片列表用户提问时answer方法把所有历史描述拼接成Image #0 / #1 ...的文本上下文交给llamaFind(question, combined)做检索式问答。llamaFind走的是Groq 通道见 imageDescription.ts其 system prompt 明确约束只基于提供的图像描述回答、不得泛化推测、不得在答案中提及图像/场景/描述本身、必须简洁具体。这与 img_44.md 中描述尽可能精确客观的风格一脉相承——整条链路的设计哲学是视觉理解交给本地 VLM 输出事实性描述推理问答交给云端 LLM 基于描述作答各司其职、相互解耦。六、本地复现从零生成你自己的多模型描述报告按 omiGlass README 的指引可在本地完整复现 img_44.md 的生成过程# 1. 克隆仓库并安装依赖Expo 项目 git clone 仓库地址 cd omiGlass npm install # 或 yarn install # 2. 配置 API 密钥与环境变量 cp .env.template .env # 在 .env 中填入 # EXPO_PUBLIC_GROQ_API_KEY Groq 密钥问答链路使用 # EXPO_PUBLIC_OPENAI_API_KEY OpenAI 密钥语音链路使用 # EXPO_PUBLIC_OLLAMA_API_URL Ollama 地址默认 http://localhost:11434/api/chat # 3. 拉取本地视觉模型默认模型 ollama pull moondream:1.8b-v2-fp16 # 4. 启动应用 npm start # 或 yarn start若要把你自己的图片加入评测集将.jpeg文件放入omiGlass/prompts/series_name/目录然后运行generate.tsts-node或编译后执行脚本会为每张新图生成包含 4 个模型输出的同名.md报告。注意运行前请确保 Ollama 服务已启动且已ollama pull所需的全部模型至少moondream:1.8b-v2-fp16若保留llava-llama3、llava:34b-v1.6的评测轮次还需额外拉取对应模型。七、方法论价值为什么要把同一张图交给多个模型img_44.md 这类文件本质上是一份可控变量的模型对比实验记录——prompt 固定、图片固定唯一变量是模型。它的价值体现在三个方面能力画像从样本可以看出 llava 系列对建筑结构拱门、格栅窗、横梁更敏感moondream 对人物关系与场景氛围咖啡馆、社交更擅长为后续按场景选择默认模型提供了依据错误交叉验证多个模型描述存在分歧的细节例如场景到底是桥边还是室内咖啡馆、人物是在交谈还是处理文件正是需要人工复核的疑点避免单一模型一本正经地胡说提示工程基线Describe the scene 强制文字转写的 system 指令构成了一套可复用的最小 prompt 模板新的评测任务可以在此基线上迭代。结语omiGlass/prompts/series_1/img_44.md表面上是 4 段英文图片描述背后则是一条完整的本地多模型视觉理解流水线generate.ts批量驱动 →imageDescription构造 prompt →ollamaInference完成 base64 编码、请求、退避重试与文本规整 → 描述文本进入Agent供llamaFind问答消费。理解这条链路你就掌握了 omiGlass看世界的第一步也获得了一套可复用的多模型图像理解评测方法。【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
omiGlass 多视觉模型图像描述流水线：基于 Ollama 的看图评测样本解析与复现
omiGlass 多视觉模型图像描述流水线基于 Ollama 的看图评测样本解析与复现【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend本文以 omiGlass 项目中 img_44.md 这一份真实的多模型图像描述输出为样本结合omiGlass/sources下的源码系统讲解 omiGlass 如何通过 Ollama 本地部署多个视觉语言模型VLM对摄像头拍摄的图片进行看图说话、批量生成评测数据并最终把图像描述接入问答 Agent 的完整链路。读完本文你将掌握 omiGlass 图像描述链路的调用实现、模型选型、环境配置与本地复现方法并能独立用同一套流水线评测任意图片在多模型下的理解表现。一、样本是什么img_44.md 的内容结构omiGlass/prompts/series_1/img_44.md是series_1评测集中第 44 张测试图片的多模型图像描述输出文件。它本身不是配置文档而是批量评测脚本自动生成的模型答卷文件内按####标题####分节保存了 4 个视觉模型对同一张图片的英文描述。原文档的完整内容如下####Description#### The image features a group of people standing in an open space, possibly inside a building or near a bridge. There are at least five individuals visible in the scene, with one person holding a drink and another person looking up towards the sky. The atmosphere appears to be casual and relaxed as they engage in conversation or observe their surroundings. ####Description (llava-llama3)#### The image captures a lively scene at an indoor event. The focal point is a white archway, adorned with four windows, each fitted with metal grilles. The archway stands against the backdrop of a white ceiling, punctuated by a skylight that allows natural light to filter in. A group of people are gathered around the archway, their attention drawn to something or someone outside the frame. Their curiosity is palpable as they stand close together, forming a semi-circle that suggests an intimate gathering or discussion. Adding to the ambiance of the scene is a white air conditioning unit nestled near the ceiling, its presence subtly reminding us of the indoor setting. Despite being an inanimate object, it seems to be contributing to the overall atmosphere of the event. The image exudes a sense of anticipation and camaraderie, as if everyone present is waiting for something significant to happen or someone to arrive. The precise positioning of the objects and people, along with their actions, paints a vivid picture of an indoor gathering filled with anticipation and conversation. ####Description (llava:34b-v1.6)#### In the image, you can see an interior scene with architectural features that suggest it could be a building with historical significance. The ceiling is adorned with exposed beams and large windows allowing natural light to enter the space. There are people present in the scene; one person appears to be seated at what looks like a desk or workstation on the right side of the image, while another person stands slightly behind them. Both individuals seem engaged in an activity that could involve paperwork or some form of documentation, given the presence of papers and a pen. The quality of the photo is somewhat grainy, possibly due to lighting conditions, camera settings, or motion blur. Overall, the scene appears casual yet focused, with people interacting within a space designed for work or study. ####Description (moondream:1.8b-v2-fp16)#### In the image, there is a group of people gathered together in what appears to be an indoor space. The individuals are standing next to each other and engaged in conversation or interaction. Among them, one person seems to be drinking from a cup while others observe the scene. Theres also a bottle visible on a surface nearby. The setting features an arched ceiling with large windows that allow natural light to enter the room. A refrigerator is located at the edge of the space, indicating it might be in a cafe or similar establishment where people often gather and socialize.可以看到四个模型对同一张室内人群场景的感知各有侧重模型场景判定人物描述细节捕获Description默认链路室内或桥边开放空间至少 5 人有人持饮料、有人仰望天空氛围轻松随意偏宏观llava-llama3室内活动白色拱门 4 扇金属格栅窗半圆形聚集、充满好奇天窗、空调内机等建筑细节llava:34b-v1.6有历史感的建筑室内一坐一立、疑似处理文件横梁天花板、纸张与笔且指出画面颗粒感moondream:1.8b-v2-fp16咖啡馆类社交场所站立交谈一人持杯桌面上的瓶子、角落的冰箱这份样本的价值在于同一 prompt、同一张图、不同模型的输出差异本身就是模型能力评估的原始证据——有的模型擅长建筑结构感知llava 系列有的模型擅长氛围与人物关系推断moondream而转写画面文字、识别物体类别等能力需要逐项对照验证。二、样本从哪来generate.ts 批量生成脚本omiGlass/prompts/generate.ts是产生这类.md文件的批处理脚本其工作流程是遍历系列目录读取prompts/下每个子目录如series_1收集所有.jpeg图片文件并把输出路径定为同名.mdreplace(.jpeg, .md)见 generate.ts逐模型跑测试对每一张图片依次执行 4 轮runTest分别调用imageDescription模型依次为默认模型、llava-llama3、llava:34b-v1.6、moondream:1.8b-v2-fp16并把结果以####标题####分节追加到输出字符串见 generate.ts写回文件全部模型跑完后把拼接结果整体写入对应.md形成 img_44.md 这样的多模型对比报告见 generate.ts。脚本中还保留了被注释的imageBlurry模糊度检测扩展点说明该评测体系的设计意图是支持更多图像质量维度的自动化检测。三、核心实现imageDescription 的 prompt 设计与默认模型描述能力由 imageDescription.ts 提供核心函数签名如下export async function imageDescription(src: Uint8Array, model: KnownModel moondream:1.8b-v2-fp16): Promisestring函数向 Ollama 发送两条消息system 指令You are a very advanced model and your task is to describe the image as precisely as possible. Transcribe any text you see.——明确要求尽可能精确地描述图像并转写画面中出现的任何文字。这条指令解释了为什么描述里会出现窗户带金属格栅exposed beams这类精确用词系统希望模型做客观转写而非泛泛而谈。user 消息Describe the scene并将图片以Uint8Array二进制形式挂到消息的images字段。默认模型是moondream:1.8b-v2-fp16这与 README 中安装模型的命令ollama pull moondream:1.8b-v2-fp16一致见 README.md。moondream 是一个轻量视觉语言模型fp16 权重版本更适合在本地设备上以较低显存运行符合 omiGlass 强调本地化、低功耗的定位。四、模型矩阵与底层调用链ollamaInference 做了什么可选的模型集合定义在 ollama.ts 的KnownModel联合类型中export type KnownModel | llama3 | llama3-gradient | llama3:8b-instruct-fp16 | llava-llama3 | llava:34b-v1.6 | moondream:1.8b-v2-fp16 | moondream:1.8b-v2-moondream2-text-model-f16ollamaInference是该流水线的传输层见 ollama.ts关键实现细节图片编码将每条消息中的Uint8Array图片通过toBase64转为 base64 字符串后放入images字段这是 Ollama/api/chat接口的标准图片输入格式请求构造axios.post(keys.ollama, { stream: false, model, messages })服务地址来自keys.ollama非流式一次拿到完整回复重试机制整个请求被backoff包装网络抖动或模型加载时自动退避重试提升批处理稳定性文本规整trimIdent用于清理消息内容与回复中的多余缩进空白保证写入.md的描述文本干净可读。API 地址与密钥集中在 keys.tsexport const keys { groq: process.env.EXPO_PUBLIC_GROQ_API_KEY ?? , ollama: process.env.EXPO_PUBLIC_OLLAMA_API_URL ?? , openai: process.env.EXPO_PUBLIC_OPENAI_API_KEY ?? , };其中ollama对应 Ollama 本地服务的http://localhost:11434/api/chat默认地址其余两个分别为 Groq 与 OpenAI 的密钥说明该流水线同时保留了云端 LLM 的接入通道。五、从看图说话到问答 Agent描述如何被消费图像描述并不是终点而是 omiGlass Agent 的输入原料。Agent.ts 的addPhoto方法在加锁状态下逐张调用imageDescription(p)把{ photo, description }存入内存照片列表用户提问时answer方法把所有历史描述拼接成Image #0 / #1 ...的文本上下文交给llamaFind(question, combined)做检索式问答。llamaFind走的是Groq 通道见 imageDescription.ts其 system prompt 明确约束只基于提供的图像描述回答、不得泛化推测、不得在答案中提及图像/场景/描述本身、必须简洁具体。这与 img_44.md 中描述尽可能精确客观的风格一脉相承——整条链路的设计哲学是视觉理解交给本地 VLM 输出事实性描述推理问答交给云端 LLM 基于描述作答各司其职、相互解耦。六、本地复现从零生成你自己的多模型描述报告按 omiGlass README 的指引可在本地完整复现 img_44.md 的生成过程# 1. 克隆仓库并安装依赖Expo 项目 git clone 仓库地址 cd omiGlass npm install # 或 yarn install # 2. 配置 API 密钥与环境变量 cp .env.template .env # 在 .env 中填入 # EXPO_PUBLIC_GROQ_API_KEY Groq 密钥问答链路使用 # EXPO_PUBLIC_OPENAI_API_KEY OpenAI 密钥语音链路使用 # EXPO_PUBLIC_OLLAMA_API_URL Ollama 地址默认 http://localhost:11434/api/chat # 3. 拉取本地视觉模型默认模型 ollama pull moondream:1.8b-v2-fp16 # 4. 启动应用 npm start # 或 yarn start若要把你自己的图片加入评测集将.jpeg文件放入omiGlass/prompts/series_name/目录然后运行generate.tsts-node或编译后执行脚本会为每张新图生成包含 4 个模型输出的同名.md报告。注意运行前请确保 Ollama 服务已启动且已ollama pull所需的全部模型至少moondream:1.8b-v2-fp16若保留llava-llama3、llava:34b-v1.6的评测轮次还需额外拉取对应模型。七、方法论价值为什么要把同一张图交给多个模型img_44.md 这类文件本质上是一份可控变量的模型对比实验记录——prompt 固定、图片固定唯一变量是模型。它的价值体现在三个方面能力画像从样本可以看出 llava 系列对建筑结构拱门、格栅窗、横梁更敏感moondream 对人物关系与场景氛围咖啡馆、社交更擅长为后续按场景选择默认模型提供了依据错误交叉验证多个模型描述存在分歧的细节例如场景到底是桥边还是室内咖啡馆、人物是在交谈还是处理文件正是需要人工复核的疑点避免单一模型一本正经地胡说提示工程基线Describe the scene 强制文字转写的 system 指令构成了一套可复用的最小 prompt 模板新的评测任务可以在此基线上迭代。结语omiGlass/prompts/series_1/img_44.md表面上是 4 段英文图片描述背后则是一条完整的本地多模型视觉理解流水线generate.ts批量驱动 →imageDescription构造 prompt →ollamaInference完成 base64 编码、请求、退避重试与文本规整 → 描述文本进入Agent供llamaFind问答消费。理解这条链路你就掌握了 omiGlass看世界的第一步也获得了一套可复用的多模型图像理解评测方法。【免费下载链接】FriendAI that sees your screen, listens to your conversations and tells you what to do项目地址: https://gitcode.com/GitHub_Trending/fr/Friend创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 
