首次提交
This commit is contained in:
16
.cos.yaml
Normal file
16
.cos.yaml
Normal file
@@ -0,0 +1,16 @@
|
||||
cos:
|
||||
base:
|
||||
secretid:
|
||||
secretkey:
|
||||
sessiontoken:
|
||||
protocol: https
|
||||
mode: ""
|
||||
cvmrolename: ""
|
||||
closeautoswitchhost: ""
|
||||
disableencryption: ""
|
||||
buckets:
|
||||
- name:
|
||||
alias:
|
||||
region: ""
|
||||
endpoint:
|
||||
ofs: false
|
||||
215
README.md
Normal file
215
README.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# OCR Data - 视频字幕提取与识别系统
|
||||
|
||||
## 项目简介
|
||||
|
||||
本项目是一个自动化视频字幕提取系统，通过视频抽帧、OCR 文字识别、音频语音识别等技术，从视频中提取硬字幕并生成 SRT 字幕文件。支持多线程批量处理，并与腾讯云 COS 对象存储集成（上传/下载）。
|
||||
|
||||
OCR 主力来自内部 OCR 服务（Tencent Youtu，封装为 `tencentcloud_ocr_api`）；Whisper 仅做辅助校验（识别首尾音频过滤片头片尾台标、与 OCR 文本做拼音匹配），不是字幕文字的主要来源，因此 Whisper 模型大小对最终字幕质量影响有限。
|
||||
|
||||
## 核心功能
|
||||
|
||||
- **视频抽帧**：用 FFmpeg `fps` 滤镜单次顺序解码，按固定频率抽帧（默认每秒 3 帧）
|
||||
- **字幕区域检测**：通过 OCR 自动定位字幕区域，识别/过滤水印
|
||||
- **图片拼接优化**：将多帧字幕区域裁剪后拼接为大图，大幅减少 OCR 调用次数
|
||||
- **并发 OCR**：getsrt 阶段对拼接图/单独帧并发调用 OCR（`subtitle.ocr_workers` 控制并发数）
|
||||
- **音频辅助校验**：用 Whisper 提取首尾音频文字，与 OCR 结果做拼音匹配
|
||||
- **SRT 字幕生成**：合并、去重（处理多种 OCR 差异变体），生成标准 SRT
|
||||
- **批量处理**：`ThreadPoolExecutor` 多视频并发；日志/计时按线程隔离
|
||||
- **运行统计**：每个视频的起止时间、各阶段耗时、图片处理数量写入 `数据.jsonl`
|
||||
|
||||
## 项目结构
|
||||
|
||||
```
|
||||
ocr_data/
|
||||
├── main.py # 主入口，批量任务调度
|
||||
├── rerun_srt.py # 复用已有OCR数据重生成SRT（调试getsrt用，不重抽帧）
|
||||
├── requirements.txt # Python 依赖
|
||||
├── coscli.exe / coscli-linux-amd64 # 腾讯云 COS CLI 工具
|
||||
├── config/
|
||||
│ ├── config.json # 配置文件（OCR密钥、抽帧参数、模型、并发等）
|
||||
│ └── config_utils.py # 配置管理器（单例）
|
||||
├── tool/
|
||||
│ ├── process.py # 单视频处理流程
|
||||
│ ├── merge.py # 图片拼接、字幕区域检测、音频文字提取
|
||||
│ ├── getsrt.py # OCR解析、字幕定位、去重、SRT生成（含并发OCR）
|
||||
│ ├── whisper_manager.py # Whisper 管理器（线程安全单例）
|
||||
│ ├── Ocr/tencentcloud_ocr.py # OCR API 封装（内部Youtu服务）
|
||||
│ ├── VideoProcessing/multithread_algo.py # FFmpeg 抽帧
|
||||
│ └── log/log_manager.py # 统一日志管理器（每视频状态线程本地）
|
||||
└── video/ # 视频文件存放目录
|
||||
```
|
||||
|
||||
## 处理流程
|
||||
|
||||
```
|
||||
1. 从 COS 下载视频（可选）
|
||||
↓
|
||||
2. FFmpeg 抽帧（每秒 3 帧，fps=3.0 精确）
|
||||
↓
|
||||
3. OCR 检测字幕区域 & 水印过滤
|
||||
↓
|
||||
4. 字幕区域裁剪 & 多帧拼接为大图
|
||||
↓
|
||||
5. 对拼接图/单独帧并发调用 OCR
|
||||
↓
|
||||
6. Whisper 提取首尾音频文字（辅助校验）
|
||||
↓
|
||||
7. 字幕定位 & 文本质量评估 & 相似帧合并去重
|
||||
↓
|
||||
8. 生成 SRT 字幕文件
|
||||
↓
|
||||
9. 上传结果到 COS & 记录元数据/运行统计
|
||||
```
|
||||
|
||||
## 环境要求
|
||||
|
||||
- Python 3.10+
|
||||
- FFmpeg（`ffmpeg` + `ffprobe`，**抽帧和音频都依赖**，需加入 PATH）
|
||||
- coscli（COS 命令行工具，配置 `coscli_path`）
|
||||
- CUDA（可选，Whisper GPU 加速）
|
||||
|
||||
## 安装依赖
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Linux 上若 `numpy`/`scikit-learn`/`av` 卡在源码编译（PyAV 编译需系统 ffmpeg 开发库），强制用预编译 wheel：
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple --only-binary=av,numpy,scikit-learn
|
||||
```
|
||||
|
||||
首次运行会从 HuggingFace 下载 Whisper 模型；国内建议先设镜像：
|
||||
|
||||
```bash
|
||||
export HF_ENDPOINT=https://hf-mirror.com
|
||||
```
|
||||
|
||||
## 配置说明
|
||||
|
||||
编辑 `config/config.json`：
|
||||
|
||||
```json
|
||||
{
|
||||
"tencentcloud": {
|
||||
"ocr": { "secret_id": "...", "secret_key": "...", "endpoints": ["http://IP1:PORT", "http://IP2:PORT"] },
|
||||
"cos": { "secret_id": "...", "secret_key": "...", "region": "ap-chongqing", "bucket": "..." }
|
||||
},
|
||||
"audio": { "model_size": "small" },
|
||||
"splitframe": {
|
||||
"interval_seconds": 0.33,
|
||||
"download": true,
|
||||
"upload_path": "cos://your-bucket/path",
|
||||
"delete_video": false,
|
||||
"coscli_path": "./coscli-linux-amd64",
|
||||
"cosconfig_path": ".cos.yaml"
|
||||
},
|
||||
"subtitle": {
|
||||
"min_confidence": 50,
|
||||
"similarity_threshold": 0.7,
|
||||
"ocr_workers": 10
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 配置项说明
|
||||
|
||||
| 配置项 | 说明 |
|
||||
|--------|------|
|
||||
| `tencentcloud.ocr.{secret_id,secret_key}` | OCR 服务凭证 |
|
||||
| `tencentcloud.ocr.endpoints` | OCR 服务 IP 列表（多个时按 QPS 加权分发，失败自动摘除重试）；不填回退到内置单端点 |
|
||||
| `tencentcloud.cos.{secret_id,secret_key,region,bucket}` | COS 凭证与桶信息 |
|
||||
| `audio.model_size` | Whisper 模型：`tiny`/`base`/`small`/`medium`/`large-v3`，默认 `small` |
|
||||
| `splitframe.interval_seconds` | 抽帧间隔，`0.33` 表示每秒 3 帧 |
|
||||
| `splitframe.download` | 是否从 COS 下载视频 |
|
||||
| `splitframe.delete_video` | 处理完是否删除本地视频 |
|
||||
| `splitframe.coscli_path` / `cosconfig_path` | coscli 路径 / COS 配置文件 |
|
||||
| `subtitle.min_confidence` | OCR 最低置信度阈值 |
|
||||
| `subtitle.similarity_threshold` | 文本相似度阈值 |
|
||||
| `subtitle.ocr_workers` | getsrt 阶段 OCR 并发线程数（默认 10） |
|
||||
|
||||
## 使用方法
|
||||
|
||||
### 批量处理
|
||||
|
||||
```bash
|
||||
python main.py --source-list ./text2 --max-workers 10
|
||||
```
|
||||
|
||||
### 命令行参数
|
||||
|
||||
| 参数 | 默认值 | 说明 |
|
||||
|------|--------|------|
|
||||
| `--source-list` | `./text2` | 任务 txt 文件所在目录 |
|
||||
| `--max-workers` | 自动(按本机CPU核数) | 最大并发数；不填/传0则自动选（若本机IP在 `tencentcloud.ocr.endpoints` 里，判定为OCR共置机，自动打对折让资源；再按可用磁盘÷每视频预估占用(`dispatch.est_disk_gb_per_video`，默认2G)收紧一次，两者取小；最后还有个硬上限30，超高核数机器也不会自动选出离谱的并发数） |
|
||||
| `--mem-guard-gb` | `2.0` | 可用内存低于此值(GB)时暂停启动新视频，防 OOM；设 `0` 关闭 |
|
||||
| `--disk-guard-gb` | `10.0` | 本地磁盘可用空间低于此值(GB)时暂停启动新视频，给下载/抽帧/合成图预留空间，防止磁盘写满导致级联失败；设 `0` 关闭 |
|
||||
|
||||
启动时会打印一行机器资源(CPU 核数 / 可用内存 / 可用磁盘 / 负载)与当前并发设置，并对不合理的参数(内存/磁盘阈值过高、并发数过大/过小)给出告警，便于跑前判断。内存保护仅在 Linux 生效(读 `/proc/meminfo` 的 `MemAvailable`)；磁盘保护跨平台(`shutil.disk_usage`)。机器无 swap 时内存保护尤其重要——可用内存触底会被暂停而非被 OOM 杀进程。
|
||||
|
||||
### 任务列表格式
|
||||
|
||||
在 `--source-list` 目录下创建 `.txt` 文件，**每行一个完整的 COS 路径**（上传目标目录由该路径自动推导，无需 `--cos-target`）：
|
||||
|
||||
```
|
||||
cos://bucket/0004/data/XXX/XXX_p1/XXX_p1.mp4
|
||||
cos://bucket/0004/data/XXX/XXX_p2/XXX_p2.mp4
|
||||
```
|
||||
|
||||
每个 txt 处理完会被删除；主循环持续监控目录直到没有 txt。**跨运行续传/去重**通过读取 `数据.jsonl` 的 `video_cos_path` 实现（不依赖 `./success/`）。
|
||||
|
||||
### 压测 OCR 端点 / 多 IP 负载均衡
|
||||
|
||||
OCR 服务有多个 IP 时，先压测每个 IP 的实际 QPS，运行时按 QPS 加权把请求分发到不同 IP：
|
||||
|
||||
```bash
|
||||
# 1) 在 config.json 的 tencentcloud.ocr.endpoints 填上所有 IP
|
||||
# 2) 压测各端点 QPS，结果写入 config/ocr_weights.json
|
||||
python bench_ocr_endpoints.py [样例图片] --requests 60 --concurrency 10
|
||||
# 3) 之后正常跑 main.py，OCR 请求会按权重自动分发；某端点连续失败会被冷却摘除并换端点重试
|
||||
```
|
||||
|
||||
不跑压测时，所有端点等权轮询；压测产出的 `config/ocr_weights.json` 存在则按 QPS 加权（错误率>50% 的端点权重置 0，不参与分发）。
|
||||
|
||||
### 调试 SRT 逻辑
|
||||
|
||||
```bash
|
||||
python tools/rerun_srt.py ./split/<video_name>/composite
|
||||
```
|
||||
|
||||
复用已有 `ocr_result.jsonl` 重生成 SRT，不重新调用 OCR，用于快速迭代 `getsrt.py` 的去重/合并/时间轴逻辑。注意它**不重新抽帧**，无法验证抽帧相关改动。composite 目录现在处理完就随 `split_dir` 一起删了（不再上传/保留到 `./composite/`），所以这条命令只在处理还没走完清理那一步之前有效——比如直接 `import tool.process` 调用 `process_single_video()` 单独调试时（这条路径不经过 `main.py`/`run_worker.py` 的清理逻辑）。
|
||||
|
||||
## 输出说明
|
||||
|
||||
`<video>` 现在是 `tool/process.py::derive_video_name(cos_path)` 算出来的（`<原文件名>_<完整COS路径MD5前8位>`），不是纯文件名——见下面"关键实现要点"里的撞名说明。
|
||||
|
||||
| 目录/文件 | 说明 |
|
||||
|-----------|------|
|
||||
| `./video/<video>.mp4` | 下载的源视频（除非 `delete_video`） |
|
||||
| `./split/<video>/` | 处理中的临时抽帧与 composite。composite（拼接大图、`ocr_result.jsonl` 等）不上传 COS，处理完这个视频后随 `split_dir` 一起整体删除本地 |
|
||||
| `./srt/<video>.srt` | 最终 SRT（上传后移到此处；上传到COS的文件名是 `<video>.ocr.srt`） |
|
||||
| `./info/<video>.json` | 处理成功的元数据（这个视频的 `数据.jsonl` 记录写完后立即删，不等整批跑完） |
|
||||
| `./info_error/<video>.json` | 失败/过滤的元数据（同上，立即删） |
|
||||
| `./log/<video>.json` | 每视频日志：`program_durations`（各阶段耗时）+ `custom_stats`（图片数量）（同上，立即删） |
|
||||
| `数据.jsonl` | 每视频记录：状态、srt 路径、时长、起止时间、`elapsed_seconds`、`image_counts`、`stage_durations` |
|
||||
| `batch_processor.log` | 批量运行日志(Python logging输出)。自动轮转：100M/份，最多留5份 |
|
||||
| `app.log` | 进程整体stdout/stderr(`启动.sh`/`守护重启.sh`用`>>`重定向到这里，比`batch_processor.log`多包含`print()`输出)。不走Python logging，轮转在shell层做：`启动.sh`每次启动前若超过`LOG_MAX_MB`(默认200)就先轮转；`守护重启.sh`的`tick`(每分钟)用copytruncate方式(`cp`+原地截断)独立轮转，因为进程可能还在用append方式往这个文件写，直接`mv`会导致新文件一直是空的 |
|
||||
|
||||
## 过滤规则
|
||||
|
||||
以下情况会过滤视频（不生成字幕）：
|
||||
|
||||
- 未检测到字幕区域
|
||||
- 空白拼接图超过一半
|
||||
- 冗余文字过多 / 水印占比过大
|
||||
- 有效字幕数量过少
|
||||
- 文本质量评分不达标（数字/特殊字符占比过高）
|
||||
|
||||
## 关键实现要点（勿回退）
|
||||
|
||||
- **视频本地/COS标识不能只用文件名**：源数据里大量不同父目录复用同样的数字文件名（如 `0021/17688/00046.mp4` 和 `0021/17690/00046.mp4`），如果本地工作目录只按文件名（不含父目录）命名，两个不同视频并发处理时会互相踩踏同一份本地文件（`./split/00046` 同时被两边写/删），引发目录未清空崩溃和连锁失败——实测占某台机器全部错误记录的 100%。现在 `main.py`/`tool/process.py` 都调用 `derive_video_name(cos_path)`（`<文件名>_<完整路径MD5前8位>`）作为唯一标识，两处必须算出同一个值。副作用：最终 SRT 文件名、composite 的 COS 目录名都会带上这个哈希后缀。
|
||||
- **抽帧频率**：每秒 3 帧时 FFmpeg `fps` 必须是 `3.0`，不能用 `1/0.33=3.03`（会比真正 3fps 快约 1%，导致帧内容与 SRT 标注随帧号累积漂移，18 分钟处可差约 10 秒）。
|
||||
- **SRT 合并的空串前缀**：`generate_srt_from_results` 的"渐进蹦字"判断要求 `current_text` 非空，否则空白间隔后的新字幕会继承空白段的起始时间。
|
||||
- **OCR 变体去重**：`_similar` 合并同一句的不同 OCR 损失版本——支持前缀、子序列、标点归一化（`······` vs `......`）、去全部标点后的长公共前缀判定（处理开头稳定、中/尾各自丢失的闪烁）。
|
||||
- **日志线程隔离**：`UnifiedLogger` 的每视频状态是线程本地的；嵌套子线程（如 OCR 并发池）不要直接调 `logger.log`，应把结果/错误回传给视频主线程统一记录。
|
||||
BIN
config/2026-08.lnk
Normal file
BIN
config/2026-08.lnk
Normal file
Binary file not shown.
0
config/__init__.py
Normal file
0
config/__init__.py
Normal file
BIN
config/__pycache__/__init__.cpython-311.pyc
Normal file
BIN
config/__pycache__/__init__.cpython-311.pyc
Normal file
Binary file not shown.
BIN
config/__pycache__/config_utils.cpython-311.pyc
Normal file
BIN
config/__pycache__/config_utils.cpython-311.pyc
Normal file
Binary file not shown.
113
config/config.json
Normal file
113
config/config.json
Normal file
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"tencentcloud": {
|
||||
"ocr": {
|
||||
"secret_id": "",
|
||||
"secret_key": "",
|
||||
"max_attempts": 4,
|
||||
"timeout": 30,
|
||||
"dispatch_log_every": 1000,
|
||||
"endpoints": [
|
||||
]
|
||||
},
|
||||
"cos": {
|
||||
"secret_id": "",
|
||||
"secret_key": "",
|
||||
"region": "",
|
||||
"bucket": ""
|
||||
}
|
||||
},
|
||||
"audio": {
|
||||
"model_size": "small",
|
||||
"device": "cpu",
|
||||
"compute_type": "float16",
|
||||
"device_index": "auto",
|
||||
"cpu_threads": 2
|
||||
},
|
||||
"splitframe": {
|
||||
"interval_seconds": 0.33,
|
||||
"ffmpeg_threads": 3,
|
||||
"use_gpu": true,
|
||||
"gpu_skip_codecs": [
|
||||
"av1"
|
||||
],
|
||||
"gpu_reserve_last_for_whisper": false,
|
||||
"gpu_skip_indices": [],
|
||||
"gpu_auto_select": true,
|
||||
"gpu_busy_mem_mb": 4096,
|
||||
"crop_bottom_ratio": null,
|
||||
"download": true,
|
||||
"root_path": "./video",
|
||||
"upload_path": "",
|
||||
"delete_video": true,
|
||||
"coscli_path": "./coscli-linux-amd64",
|
||||
"cosconfig_path": ".cos.yaml",
|
||||
"ffmpeg_timeout": 3600
|
||||
},
|
||||
"subtitle": {
|
||||
"min_confidence": 50,
|
||||
"similarity_threshold": 0.7,
|
||||
"ocr_workers": 20,
|
||||
"require_chinese": true,
|
||||
"redundant_bands": 12,
|
||||
"redundant_merge_ratio": 0.97,
|
||||
"no_composite": false,
|
||||
"min_duration_seconds": 0,
|
||||
"merge_workers": 4,
|
||||
"cv_detect": true,
|
||||
"cv_use_region": false,
|
||||
"max_tiles_per_composite": 24,
|
||||
"ocr_max_image_side": 4000,
|
||||
"filter_static": true,
|
||||
"static_frac": 0.25,
|
||||
"filter_lowconf": true,
|
||||
"lowconf_thr": 80,
|
||||
"lowconf_filter_frac": 0.6,
|
||||
"lowconf_min_det": 200,
|
||||
"strip_static_prefix": false,
|
||||
"filter_watermark_pos": true,
|
||||
"wm_dominant_frac": 0.25,
|
||||
"wm_min_count": 20,
|
||||
"split_far_boxes": true,
|
||||
"max_duration_seconds": 0,
|
||||
"long_video_list": "./超长待重跑.txt",
|
||||
"ocr_no_drop": true,
|
||||
"ocr_max_passes": 5,
|
||||
"ocr_retry_sleep": 2,
|
||||
"ocr_fail_list": "./OCR失败待重跑.txt",
|
||||
"max_global_ocr": 0,
|
||||
"no_subtitle_reasons": [
|
||||
"字幕太少",
|
||||
"冗余文字过多",
|
||||
"冗余文字太多",
|
||||
"无字幕区域",
|
||||
"非字幕内容",
|
||||
"置信度太低",
|
||||
"未检测",
|
||||
"特殊字体",
|
||||
"视频过短",
|
||||
"视频过长"
|
||||
]
|
||||
},
|
||||
"dispatch": {
|
||||
"enabled": true,
|
||||
"source": "cos",
|
||||
"run_long_video": false,
|
||||
"auto_fallback_long_video": false,
|
||||
"global_dedup_on_startup": false,
|
||||
"machine_id": "",
|
||||
"jobs_root": "",
|
||||
"shard_size": 2000,
|
||||
"keep_alive": true,
|
||||
"idle_poll_seconds": 60,
|
||||
"claim_shuffle": true,
|
||||
"claim_match": "",
|
||||
"heartbeat_seconds": 300,
|
||||
"daily_backup_path": "",
|
||||
"monitor_root": "",
|
||||
"long_list_backup_path": "",
|
||||
"long_list_clear_after_upload": true,
|
||||
"long_list_min_batch": 1000,
|
||||
"daily_backup_hour": 10,
|
||||
"stop_flag_key": ""
|
||||
}
|
||||
}
|
||||
131
config/config_utils.py
Normal file
131
config/config_utils.py
Normal file
@@ -0,0 +1,131 @@
|
||||
# config_utils.py
|
||||
import json
|
||||
import os
|
||||
|
||||
class ConfigManager:
|
||||
_instance = None
|
||||
_config = None
|
||||
|
||||
def __new__(cls):
|
||||
if cls._instance is None:
|
||||
cls._instance = super(ConfigManager, cls).__new__(cls)
|
||||
cls._instance.load_config()
|
||||
return cls._instance
|
||||
|
||||
def load_config(self, config_path="./config/config.json"):
|
||||
"""加载配置文件"""
|
||||
try:
|
||||
if not os.path.exists(config_path):
|
||||
# 如果配置文件不存在，创建一个默认配置
|
||||
default_config = {
|
||||
"tencentcloud": {
|
||||
"ocr": {
|
||||
"secret_id": "please_set_your_secret_id",
|
||||
"secret_key": "please_set_your_secret_key",
|
||||
"region": "ap-beijing",
|
||||
"endpoint": "ocr.tencentcloudapi.com"
|
||||
}
|
||||
}
|
||||
}
|
||||
with open(config_path, 'w', encoding='utf-8') as f:
|
||||
json.dump(default_config, f, indent=4, ensure_ascii=False)
|
||||
print(f"已创建默认配置文件: {config_path}")
|
||||
print("请编辑此文件并填写您的腾讯云OCR密钥")
|
||||
|
||||
with open(config_path, 'r', encoding='utf-8') as f:
|
||||
self._config = json.load(f)
|
||||
|
||||
except Exception as e:
|
||||
print(f"加载配置文件失败: {e}")
|
||||
self._config = {}
|
||||
|
||||
def get(self, key, default=None):
|
||||
"""获取配置值"""
|
||||
try:
|
||||
keys = key.split('.')
|
||||
value = self._config
|
||||
for k in keys:
|
||||
value = value[k]
|
||||
return value
|
||||
except (KeyError, TypeError):
|
||||
return default
|
||||
|
||||
def get_tencent_ocr_config(self):
|
||||
"""获取腾讯云OCR配置"""
|
||||
return self.get('tencentcloud.ocr', {})
|
||||
|
||||
def get_splitframe_config(self):
|
||||
"""获取Whisper配置"""
|
||||
return self.get('splitframe', {})
|
||||
|
||||
def get_audio_config(self):
|
||||
"""获取音频配置"""
|
||||
return self.get('audio', {})
|
||||
|
||||
def get_subtitle_config(self):
|
||||
"""获取字幕配置"""
|
||||
return self.get('subtitle', {})
|
||||
|
||||
def get_cos_config(self):
|
||||
"""获取COS配置"""
|
||||
return self.get('tencentcloud.cos', {})
|
||||
|
||||
def get_dispatch_config(self):
|
||||
"""获取 COS 任务分发配置（常驻 worker / 认领锁 / 每日备份）。"""
|
||||
return self.get('dispatch', {})
|
||||
|
||||
def _prepend_local_ffmpeg_to_path():
|
||||
"""若项目根的 bin/ 下有自带 ffmpeg，则把 bin/ 提到 PATH 最前，
|
||||
让所有 ffmpeg/ffprobe 调用(抽帧、音频、pydub 等)都优先用项目自带的，
|
||||
不再依赖系统 PATH。"""
|
||||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
bin_dir = os.path.join(root, "bin")
|
||||
ff = os.path.join(bin_dir, "ffmpeg")
|
||||
if os.path.isfile(ff):
|
||||
try:
|
||||
os.chmod(ff, 0o755)
|
||||
ffprobe = os.path.join(bin_dir, "ffprobe")
|
||||
if os.path.isfile(ffprobe):
|
||||
os.chmod(ffprobe, 0o755)
|
||||
except OSError:
|
||||
pass
|
||||
os.environ["PATH"] = bin_dir + os.pathsep + os.environ.get("PATH", "")
|
||||
|
||||
|
||||
def _ensure_coscli_executable():
|
||||
"""给项目根的 coscli-linux-amd64 自动加执行权限（解压后常丢失 +x，导致无法调用）。"""
|
||||
if os.name == "nt":
|
||||
return
|
||||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
for name in ("coscli-linux-amd64", "coscli"):
|
||||
p = os.path.join(root, name)
|
||||
if os.path.isfile(p) and not os.access(p, os.X_OK):
|
||||
try:
|
||||
os.chmod(p, 0o755)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
_prepend_local_ffmpeg_to_path()
|
||||
_ensure_coscli_executable()
|
||||
|
||||
|
||||
def _resolve_bin(name):
|
||||
"""返回项目 bin/ 下的可执行文件绝对路径(若存在)，否则返回名字本身(回退到 PATH)。"""
|
||||
root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
p = os.path.join(root, "bin", name)
|
||||
return p if os.path.isfile(p) else name
|
||||
|
||||
|
||||
def get_ffmpeg():
|
||||
"""ffmpeg 可执行路径：优先项目自带 bin/ffmpeg，否则 PATH 里的 ffmpeg。"""
|
||||
return _resolve_bin("ffmpeg")
|
||||
|
||||
|
||||
def get_ffprobe():
|
||||
"""ffprobe 可执行路径：优先项目自带 bin/ffprobe，否则 PATH 里的 ffprobe。"""
|
||||
return _resolve_bin("ffprobe")
|
||||
|
||||
|
||||
# 全局配置实例
|
||||
config = ConfigManager()
|
||||
BIN
coscli-linux-amd64
Normal file
BIN
coscli-linux-amd64
Normal file
Binary file not shown.
628
main.py
Normal file
628
main.py
Normal file
@@ -0,0 +1,628 @@
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
import stat
|
||||
import threading
|
||||
import time
|
||||
import cv2
|
||||
from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed
|
||||
from datetime import datetime
|
||||
from config.config_utils import config
|
||||
from tool.process import process_single_video, derive_video_name
|
||||
from tool.log.log_manager import logger as steplog
|
||||
from tool import job_cos
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s',
|
||||
handlers=[
|
||||
# 常驻worker一跑就是数天/数周, 普通FileHandler会无限增长; 100M/份, 最多留5份(共500M封顶)
|
||||
RotatingFileHandler('batch_processor.log', maxBytes=100 * 1024 * 1024, backupCount=5, encoding='utf-8'),
|
||||
logging.StreamHandler()
|
||||
]
|
||||
)
|
||||
|
||||
JSONL_PATH = "数据.jsonl"
|
||||
|
||||
|
||||
# ==================== 工具函数 ====================
|
||||
|
||||
def parse_cos_path(cos_path: str):
|
||||
"""cos://bucket/path/video.mp4 → ('cos://bucket/path/', 'video.mp4')"""
|
||||
cos_path = cos_path.strip()
|
||||
idx = cos_path.rfind('/')
|
||||
return cos_path[:idx + 1], cos_path[idx + 1:]
|
||||
|
||||
|
||||
def get_video_duration(local_path: str) -> float:
|
||||
if not os.path.exists(local_path):
|
||||
return 0.0
|
||||
cap = cv2.VideoCapture(local_path)
|
||||
if not cap.isOpened():
|
||||
return 0.0
|
||||
frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)
|
||||
fps = cap.get(cv2.CAP_PROP_FPS)
|
||||
cap.release()
|
||||
return round(frames / fps, 2) if fps > 0 else 0.0
|
||||
|
||||
|
||||
def _read_log_stats(video_name: str):
|
||||
"""从 log/<video>.json 读取图片数量统计与各阶段耗时。
|
||||
返回 (image_counts: dict|None, stage_durations: dict|None, video_duration: float|None)，读取失败返回全 None。"""
|
||||
log_path = os.path.join("log", f"{video_name}.json")
|
||||
if not os.path.exists(log_path):
|
||||
return None, None, None
|
||||
try:
|
||||
with open(log_path, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
meta = data.get("metadata", {})
|
||||
custom = meta.get("custom_stats", {}) or {}
|
||||
durations = meta.get("program_durations", {}) or {}
|
||||
image_counts = {
|
||||
"frame_count": custom.get("frame_count"),
|
||||
"composite_ocr_count": custom.get("composite_ocr_count"),
|
||||
"single_frame_ocr_count": custom.get("single_frame_ocr_count"),
|
||||
"total_ocr_count": custom.get("total_ocr_count"),
|
||||
}
|
||||
# 各阶段耗时（秒），四舍五入到2位
|
||||
stage_durations = {k: round(v, 2) for k, v in durations.items()}
|
||||
video_duration = custom.get("video_duration") # process.py 在删视频前用 ffprobe 存的(AV1也准)
|
||||
return image_counts, stage_durations, video_duration
|
||||
except (OSError, IOError, json.JSONDecodeError) as e:
|
||||
logging.warning(f"读取日志统计失败 {log_path}: {e}")
|
||||
return None, None, None
|
||||
|
||||
|
||||
def upload_srt(srt_local: str, cos_dir: str, coscli_path: str, cos_config_path: str):
|
||||
"""上传 SRT 文件到视频对应的 COS 目录，返回 COS 路径；失败返回 None"""
|
||||
if not os.path.exists(srt_local):
|
||||
return None
|
||||
cos_dest = cos_dir + os.path.basename(srt_local)
|
||||
cmd = [coscli_path, "-c", cos_config_path, "cp", srt_local, cos_dest]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
if result.returncode == 0:
|
||||
return cos_dest
|
||||
logging.error(f"SRT 上传失败: {result.stderr}")
|
||||
return None
|
||||
|
||||
|
||||
def get_available_mem_gb():
|
||||
"""读取系统可用内存(GB)。Linux 读 /proc/meminfo 的 MemAvailable；
|
||||
非 Linux 或读取失败返回 None（此时内存保护自动失效）。"""
|
||||
try:
|
||||
with open('/proc/meminfo', 'r') as f:
|
||||
for line in f:
|
||||
if line.startswith('MemAvailable:'):
|
||||
return float(line.split()[1]) / 1024 / 1024 # kB → GB
|
||||
except (OSError, ValueError, IndexError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def get_available_disk_gb(path: str = "."):
|
||||
"""读取 path 所在磁盘的可用空间(GB)。用 shutil.disk_usage，跨平台(Windows/Linux都行，
|
||||
不像内存检查那样只能读 /proc/meminfo)。读取失败(路径不存在等)返回 None，此时磁盘保护自动失效。"""
|
||||
try:
|
||||
return shutil.disk_usage(path).free / 1024 / 1024 / 1024
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def get_load_avg():
|
||||
"""返回 (1, 5, 15) 分钟负载；非 Unix 返回 None。"""
|
||||
try:
|
||||
return os.getloadavg()
|
||||
except (OSError, AttributeError):
|
||||
return None
|
||||
|
||||
|
||||
def load_processed(jsonl_path: str) -> set:
|
||||
"""从数据.jsonl 读取已处理的 COS 路径集合，用于跨批次断点续传"""
|
||||
processed = set()
|
||||
if not os.path.exists(jsonl_path):
|
||||
return processed
|
||||
with open(jsonl_path, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
vid = rec.get('video_cos_path')
|
||||
if vid:
|
||||
processed.add(vid)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return processed
|
||||
|
||||
|
||||
def force_rmtree(path):
|
||||
"""强制删除目录（兼容 Windows 和 Linux）"""
|
||||
if not os.path.exists(path):
|
||||
return
|
||||
for attempt in range(3):
|
||||
try:
|
||||
shutil.rmtree(path)
|
||||
return
|
||||
except PermissionError:
|
||||
if attempt < 2:
|
||||
time.sleep(1)
|
||||
for root, dirs, files in os.walk(path):
|
||||
for name in files:
|
||||
try:
|
||||
os.chmod(os.path.join(root, name), stat.S_IWUSR | stat.S_IRUSR)
|
||||
except Exception:
|
||||
pass
|
||||
else:
|
||||
if os.name == 'nt':
|
||||
os.system(f'rmdir /s /q "{path}"')
|
||||
else:
|
||||
os.system(f'rm -rf "{path}"')
|
||||
|
||||
|
||||
# ==================== 统计 ====================
|
||||
|
||||
class Stats:
|
||||
def __init__(self, total: int):
|
||||
self._lock = threading.Lock()
|
||||
self.success = 0
|
||||
self.no_subtitle = 0
|
||||
self.error = 0
|
||||
self.done = 0
|
||||
self.total = total
|
||||
|
||||
def update(self, status: str):
|
||||
with self._lock:
|
||||
self.done += 1
|
||||
if status == 'success':
|
||||
self.success += 1
|
||||
elif status == 'no_subtitle':
|
||||
self.no_subtitle += 1
|
||||
else:
|
||||
self.error += 1
|
||||
print(
|
||||
f"[进度 {self.done}/{self.total}] "
|
||||
f"成功: {self.success} 无字幕: {self.no_subtitle} 错误: {self.error} "
|
||||
f"剩余: {self.total - self.done}",
|
||||
flush=True
|
||||
)
|
||||
|
||||
|
||||
# ==================== 处理器 ====================
|
||||
|
||||
def _wait_for_memory(video_name: str, mem_guard_gb: float):
|
||||
"""内存可用量低于阈值时阻塞，等已在处理的视频释放内存后再启动新视频，避免 OOM。
|
||||
模块级函数：进程池下每个 worker 进程独立调用。读不到内存时直接放行。"""
|
||||
if mem_guard_gb <= 0:
|
||||
return
|
||||
waited = 0
|
||||
while True:
|
||||
avail = get_available_mem_gb()
|
||||
if avail is None or avail >= mem_guard_gb:
|
||||
return
|
||||
if waited % 15 == 0:
|
||||
logging.warning(
|
||||
f"内存不足(可用 {avail:.1f}G < 阈值 {mem_guard_gb}G)，"
|
||||
f"暂停启动 {video_name}，等待回收... 已等 {waited}s"
|
||||
)
|
||||
time.sleep(3)
|
||||
waited += 3
|
||||
|
||||
|
||||
def _wait_for_disk(video_name: str, disk_guard_gb: float):
|
||||
"""可用磁盘空间低于阈值时阻塞，等已在处理的视频清理释放空间后再启动新视频。
|
||||
防的是磁盘写满级联失败(下载/抽帧写一半失败、GPU硬解报错误判成"GPU失败"其实是写不进磁盘、
|
||||
info_error都写不进去导致视频异常处理链路整个乱掉——实际发生过的连锁故障)。
|
||||
模块级函数：进程池下每个 worker 进程独立调用。读不到磁盘空间时直接放行。"""
|
||||
if disk_guard_gb <= 0:
|
||||
return
|
||||
waited = 0
|
||||
while True:
|
||||
avail = get_available_disk_gb(".")
|
||||
if avail is None or avail >= disk_guard_gb:
|
||||
return
|
||||
if waited % 15 == 0:
|
||||
logging.warning(
|
||||
f"磁盘空间不足(可用 {avail:.1f}G < 阈值 {disk_guard_gb}G)，"
|
||||
f"暂停启动 {video_name}，等待清理释放空间... 已等 {waited}s"
|
||||
)
|
||||
time.sleep(3)
|
||||
waited += 3
|
||||
|
||||
|
||||
def _short_reason(text: str) -> str:
|
||||
"""把 info_error 里的长错误串归一成一个短分类标签，便于 数据.jsonl 里按原因统计/筛选。"""
|
||||
if not text:
|
||||
return None
|
||||
for kw in ("视频过长", "视频过短", "字幕太少", "冗余文字过多",
|
||||
"下载失败", "抽帧", "拼接失败", "字幕生成失败", "OCR"):
|
||||
if kw in text:
|
||||
return kw
|
||||
if "置信度" in text or "confidence" in text.lower():
|
||||
return "置信度太低"
|
||||
if "未找到合适的字幕区域" in text or "未检测" in text:
|
||||
return "无字幕区域"
|
||||
# 兜底截断: 之前只截40字符——异常串常见格式是"RuntimeError: 视频<video_name>真正原因",
|
||||
# video_name(文件名+MD5后缀)本身就有二三十字符，40字符的窗口经常在真正原因(冒号后面
|
||||
# 那段)出现之前就被切没了，数据.jsonl里全变成"RuntimeError: 视频<name>"这种没有信息量
|
||||
# 的记录（实测发生过：一小时63条error全部是这个模式）。加大到150，给真实原因留出空间；
|
||||
# 完整未截断的原始异常串仍然保留在 record["error_reason"] 里，这里只是短标签。
|
||||
return text.strip()[:150]
|
||||
|
||||
|
||||
def _read_error_reason(video_name: str) -> str:
|
||||
"""从 ./info_error/<video>.json 读出过滤/错误原因(归类后的短标签)。"""
|
||||
err_json = f"./info_error/{video_name}.json"
|
||||
if not os.path.exists(err_json):
|
||||
return None
|
||||
try:
|
||||
with open(err_json, 'r', encoding='utf-8') as f:
|
||||
data = json.load(f)
|
||||
return _short_reason((data.get("main_process") or {}).get("error"))
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
# 这些原因是"正常过滤(视频本就没有可用字幕)", 不是程序硬故障 → 应记 no_subtitle 而非 error。
|
||||
# 旧逻辑用 '"被过滤" in json文本' 猜测, 但过滤原因只存在 main_process.error 里(不含"被过滤"),
|
||||
# 于是字幕太少/冗余/置信度/非字幕等全被错记成 error。改为按原因关键词匹配, 清单可在
|
||||
# config.subtitle.no_subtitle_reasons 覆盖。
|
||||
_NO_SUB_REASONS_DEFAULT = [
|
||||
"字幕太少", "冗余文字过多", "冗余文字太多", "无字幕区域",
|
||||
"非字幕内容", "置信度太低", "未检测", "特殊字体",
|
||||
"视频过短", "视频过长",
|
||||
]
|
||||
|
||||
|
||||
def _is_no_subtitle_reason(reason: str) -> bool:
|
||||
"""原因是否属于'无可用字幕的正常过滤'(→ no_subtitle), 而非硬故障(→ error)。"""
|
||||
if not reason:
|
||||
return False
|
||||
try:
|
||||
keys = config.get_subtitle_config().get('no_subtitle_reasons') or _NO_SUB_REASONS_DEFAULT
|
||||
except Exception:
|
||||
keys = _NO_SUB_REASONS_DEFAULT
|
||||
return any(k in reason for k in keys)
|
||||
|
||||
|
||||
def process_video_task(cos_path: str, coscli_path: str, cos_config_path: str,
|
||||
split_cfg: dict, mem_guard_gb: float, disk_guard_gb: float = 0) -> dict:
|
||||
"""处理单个视频，返回完整 record（不触碰共享状态，可被进程池 pickle 调用）。
|
||||
进程池模式下：本函数整体在独立子进程里跑 → 各自的 Python 解释器/GIL/whisper 模型 → 真多核。"""
|
||||
cos_dir, filename = parse_cos_path(cos_path)
|
||||
video_name = derive_video_name(cos_path) # 含路径MD5后缀, 避免不同父目录同名视频本地/COS撞车
|
||||
local_video = os.path.join("./video", video_name + os.path.splitext(filename)[1])
|
||||
|
||||
# 内存/磁盘保护（放在计时前，等待时间不计入 elapsed_seconds）
|
||||
_wait_for_memory(video_name, mem_guard_gb)
|
||||
_wait_for_disk(video_name, disk_guard_gb)
|
||||
|
||||
status = "error"
|
||||
start_dt = datetime.now()
|
||||
start_perf = time.time()
|
||||
record = {
|
||||
"timestamp": start_dt.isoformat(),
|
||||
"video_cos_path": cos_path,
|
||||
"status": None,
|
||||
"reason": None,
|
||||
"srt_cos_path": None,
|
||||
"duration": None,
|
||||
"process_start": start_dt.isoformat(),
|
||||
"process_end": None,
|
||||
"elapsed_seconds": None,
|
||||
"image_counts": None,
|
||||
"stage_durations": None,
|
||||
}
|
||||
|
||||
try:
|
||||
success = process_single_video(cos_path, cos_dir)
|
||||
record["duration"] = get_video_duration(local_video)
|
||||
|
||||
if success:
|
||||
srt_local = f"./split/{video_name}/composite/{video_name}.ocr.srt"
|
||||
srt_cos = upload_srt(srt_local, cos_dir, coscli_path, cos_config_path)
|
||||
status = "success"
|
||||
record["srt_cos_path"] = srt_cos
|
||||
if os.path.exists(srt_local):
|
||||
os.makedirs("./srt", exist_ok=True)
|
||||
shutil.move(srt_local, os.path.join("./srt", f"{video_name}.srt"))
|
||||
steplog.step(video_name, "上传完成", "srt已传" if srt_cos else "srt上传失败")
|
||||
logging.info(f"成功 {filename} → {srt_cos}")
|
||||
else:
|
||||
# 按原因关键词归类(config 可调): 无可用字幕的正常过滤 → no_subtitle, 其余 → error。
|
||||
reason = _read_error_reason(video_name)
|
||||
record["reason"] = reason
|
||||
is_filtered = _is_no_subtitle_reason(reason)
|
||||
status = "no_subtitle" if is_filtered else "error"
|
||||
logging.info(f"{'无字幕' if is_filtered else '失败'} {filename}")
|
||||
|
||||
except Exception as e:
|
||||
reason = _short_reason(f"{type(e).__name__}: {e}")
|
||||
record["error_reason"] = f"{type(e).__name__}: {e}"
|
||||
record["reason"] = reason
|
||||
# 过滤型异常(如 raise '未检测到字幕区域'/'视频过短') 也应归 no_subtitle, 不是硬故障。
|
||||
status = "no_subtitle" if _is_no_subtitle_reason(reason) else "error"
|
||||
logging.error(f"异常 {filename}: {e}")
|
||||
|
||||
finally:
|
||||
# composite（含拼接大图/OCR中间结果）不上传 COS，处理完直接随 split_dir 一起删除本地。
|
||||
split_dir = f"./split/{video_name}"
|
||||
if os.path.exists(split_dir):
|
||||
force_rmtree(split_dir)
|
||||
if split_cfg.get('delete_video') and os.path.exists(local_video):
|
||||
try:
|
||||
os.remove(local_video)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
end_dt = datetime.now()
|
||||
record["status"] = status
|
||||
record["process_end"] = end_dt.isoformat()
|
||||
record["elapsed_seconds"] = round(time.time() - start_perf, 2)
|
||||
image_counts, stage_durations, video_duration = _read_log_stats(video_name)
|
||||
record["image_counts"] = image_counts
|
||||
record["stage_durations"] = stage_durations
|
||||
# 时长优先用 process.py 删视频前 ffprobe 存的(cv2读不了AV1、且delete_video后源视频已删)
|
||||
if (not record.get("duration")) and video_duration:
|
||||
record["duration"] = video_duration
|
||||
|
||||
# 该读的(日志统计/错误原因)都读完了，本视频的临时小文件立即删，不等整批/整片跑完再统一清
|
||||
# (之前是攒到 main.py 主循环/run_worker._process_shard 末尾才批量删，一片上千个视频攒一路)。
|
||||
for f in (f"./log/{video_name}.json", f"./info/{video_name}.json", f"./info_error/{video_name}.json"):
|
||||
try:
|
||||
if os.path.exists(f):
|
||||
os.remove(f)
|
||||
except OSError:
|
||||
pass
|
||||
return record
|
||||
|
||||
|
||||
def _ocr_worker_init(sem):
|
||||
"""进程池 worker 启动时调用：注入【全局 OCR 并发信号量】,使所有进程共享一个硬性并发上限。"""
|
||||
try:
|
||||
from tool.Ocr import tencentcloud_ocr
|
||||
tencentcloud_ocr.set_ocr_semaphore(sem)
|
||||
except Exception as e:
|
||||
logging.warning(f"注入全局OCR信号量失败: {e}")
|
||||
|
||||
|
||||
class VideoProcessor:
|
||||
def __init__(self, max_workers: int = 10, mem_guard_gb: float = 2.0, executor: str = "process",
|
||||
max_tasks_per_child: int = 30, disk_guard_gb: float = 0):
|
||||
self.max_workers = max_workers
|
||||
self.mem_guard_gb = mem_guard_gb # 可用内存低于此值(GB)时暂停启动新视频，<=0 关闭
|
||||
self.disk_guard_gb = disk_guard_gb # 本地磁盘可用空间低于此值(GB)时暂停启动新视频，<=0 关闭
|
||||
self.executor = executor # "process"=进程池(绕开GIL,真多核) / "thread"=线程池
|
||||
self.max_tasks_per_child = max_tasks_per_child # 进程池: worker处理这么多视频后重启释放内存
|
||||
self._jsonl_lock = threading.Lock()
|
||||
|
||||
def _append_jsonl(self, record: dict):
|
||||
with self._jsonl_lock:
|
||||
with open(JSONL_PATH, 'a', encoding='utf-8') as f:
|
||||
f.write(json.dumps(record, ensure_ascii=False) + "\n")
|
||||
|
||||
def process_batch(self, tasks: list, coscli_path: str,
|
||||
cos_config_path: str, split_cfg: dict) -> Stats:
|
||||
stats = Stats(total=len(tasks))
|
||||
mode = "进程池(多核,绕GIL)" if self.executor == "process" else "线程池"
|
||||
logging.info(f"开始处理 {len(tasks)} 个视频 ({mode} 并发={self.max_workers})")
|
||||
|
||||
# 后台定时打印进度（每30秒一次，处理完后停止）
|
||||
stop_event = threading.Event()
|
||||
def _heartbeat():
|
||||
while not stop_event.wait(30):
|
||||
with stats._lock:
|
||||
d, t = stats.done, stats.total
|
||||
s, n, e = stats.success, stats.no_subtitle, stats.error
|
||||
print(
|
||||
f"[进行中 {d}/{t}] 成功: {s} 无字幕: {n} 错误: {e} 剩余: {t - d}",
|
||||
flush=True
|
||||
)
|
||||
threading.Thread(target=_heartbeat, daemon=True).start()
|
||||
|
||||
# 进程池: 每个视频在独立子进程跑 → 各自 GIL/whisper → 真多核(getsrt 的 jieba/拼音不再串行)
|
||||
# 线程池: 退化兜底(--executor thread)
|
||||
# 用 forkserver: 从干净的服务进程 fork worker,避免"主进程多线程+fork"继承到被持有的锁而死锁;
|
||||
# 重模块(cv2/whisper)在 forkserver 里只导入一次, worker 共享(COW),启动快。
|
||||
if self.executor == "process":
|
||||
import multiprocessing
|
||||
try:
|
||||
_ctx = multiprocessing.get_context("forkserver")
|
||||
except ValueError:
|
||||
_ctx = multiprocessing.get_context() # 非 POSIX 等回退默认
|
||||
# 全局OCR并发上限: 父进程建跨进程信号量, 经 initializer 注入每个 worker →
|
||||
# 不管 max-workers × ocr_workers 多大, 全局同时在飞的OCR请求被硬性卡在此值, 防打爆端点。
|
||||
_max_global_ocr = int(config.get_subtitle_config().get('max_global_ocr', 0) or 0)
|
||||
_init, _initargs = None, ()
|
||||
if _max_global_ocr > 0:
|
||||
_init = _ocr_worker_init
|
||||
_initargs = (_ctx.Semaphore(_max_global_ocr),)
|
||||
logging.info(f"全局OCR并发上限: {_max_global_ocr}")
|
||||
# max_tasks_per_child: worker 处理这么多视频后【自动重启】释放内存(需 Python≥3.11)。
|
||||
# ⚠️ Python 3.11 该机制有"补不满"的坑: 频繁回收会让 worker 数随时间从 N 缩到个位数甚至0
|
||||
# → as_completed 永久挂起 → 进度冻住。内存充裕时建议【关闭】(设 0/None), 靠整体重启释放内存。
|
||||
_mtpc = self.max_tasks_per_child if (self.max_tasks_per_child and self.max_tasks_per_child > 0) else None
|
||||
if _mtpc is None:
|
||||
logging.info("max_tasks_per_child=关闭(worker不自动回收, 避免3.11回收缩水; 内存靠整体重启/mem-guard控制)")
|
||||
try:
|
||||
pool = ProcessPoolExecutor(max_workers=self.max_workers, mp_context=_ctx,
|
||||
max_tasks_per_child=_mtpc,
|
||||
initializer=_init, initargs=_initargs)
|
||||
except TypeError:
|
||||
logging.warning("当前Python不支持 max_tasks_per_child(需≥3.11), 进程不自动回收, 注意内存")
|
||||
pool = ProcessPoolExecutor(max_workers=self.max_workers, mp_context=_ctx,
|
||||
initializer=_init, initargs=_initargs)
|
||||
else:
|
||||
# 线程池模式: 同进程内, 用 threading 信号量直接限流(可选)
|
||||
_max_global_ocr = int(config.get_subtitle_config().get('max_global_ocr', 0) or 0)
|
||||
if _max_global_ocr > 0:
|
||||
from tool.Ocr import tencentcloud_ocr
|
||||
tencentcloud_ocr.set_ocr_semaphore(threading.Semaphore(_max_global_ocr))
|
||||
logging.info(f"全局OCR并发上限: {_max_global_ocr}")
|
||||
pool = ThreadPoolExecutor(max_workers=self.max_workers)
|
||||
with pool as executor:
|
||||
futures = {
|
||||
executor.submit(
|
||||
process_video_task, cos_path,
|
||||
coscli_path, cos_config_path, split_cfg, self.mem_guard_gb, self.disk_guard_gb
|
||||
): cos_path
|
||||
for cos_path in tasks
|
||||
}
|
||||
# 父进程单线程收集结果 → 统一写 jsonl(单写者,无需跨进程锁) + 更新统计
|
||||
for future in as_completed(futures):
|
||||
cos_path = futures[future]
|
||||
try:
|
||||
record = future.result()
|
||||
except Exception as e:
|
||||
# 子进程崩溃(段错误/被OOM杀)等 → 仍记一条 error
|
||||
logging.error(f"任务异常 {cos_path}: {e}")
|
||||
record = {"video_cos_path": cos_path, "status": "error",
|
||||
"error_reason": f"worker崩溃: {type(e).__name__}: {e}",
|
||||
"process_end": datetime.now().isoformat()}
|
||||
self._append_jsonl(record)
|
||||
stats.update(record.get("status", "error"))
|
||||
|
||||
stop_event.set() # 停止心跳线程
|
||||
logging.info(
|
||||
f"批次完成 — 成功: {stats.success} | "
|
||||
f"无字幕: {stats.no_subtitle} | 错误: {stats.error}"
|
||||
)
|
||||
return stats
|
||||
|
||||
|
||||
# ==================== 入口 ====================
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description='视频字幕批量提取（txt 内填 COS 完整路径）')
|
||||
parser.add_argument('--source-list', default='./text2',
|
||||
help='包含 COS 路径列表的 txt 文件目录')
|
||||
parser.add_argument('--max-workers', type=int, default=None,
|
||||
help='最大并发线程数 (不填/0 = 自动按本机CPU核数选)')
|
||||
parser.add_argument('--mem-guard-gb', type=float, default=2.0,
|
||||
help='可用内存低于此值(GB)时暂停启动新视频，防 OOM；设 0 关闭 (默认: 2.0)')
|
||||
parser.add_argument('--disk-guard-gb', type=float, default=10.0,
|
||||
help='本地磁盘可用空间低于此值(GB)时暂停启动新视频，给下载/抽帧/合成图预留空间，'
|
||||
'防止磁盘写满导致级联失败；设 0 关闭 (默认: 10.0)')
|
||||
parser.add_argument('--max-tasks-per-child', type=int, default=30,
|
||||
help='进程池模式: 每个worker处理这么多视频后重启释放内存(默认30, 需Python≥3.11)')
|
||||
parser.add_argument('--executor', choices=['process', 'thread'], default='process',
|
||||
help='并发方式: process=进程池(默认,绕开GIL真多核) / thread=线程池(旧行为,CPU重计算会被GIL串行)')
|
||||
args = parser.parse_args()
|
||||
|
||||
# ---- 启动时打印机器资源与并发设置，便于判断参数是否合理 ----
|
||||
cpu = os.cpu_count() or 0
|
||||
if not args.max_workers:
|
||||
args.max_workers = job_cos.auto_max_workers(cpu, disk_guard_gb=args.disk_guard_gb)
|
||||
logging.info(f"--max-workers 未指定, 自动选为 {args.max_workers}(本机CPU核数={cpu or '未知'}; "
|
||||
f"若本机还跑了OCR服务但不在 tencentcloud.ocr.endpoints 里, 自动探测不到, 请手动传小一点的值; "
|
||||
f"同时按可用磁盘 ÷ 每视频预估占用(dispatch.est_disk_gb_per_video)做了收紧)")
|
||||
# 供子进程(forkserver fork出来的worker)读取: splitframe.ffmpeg_threads=auto 时按此摊分CPU
|
||||
os.environ['OCR_MAX_WORKERS'] = str(args.max_workers)
|
||||
mem = get_available_mem_gb()
|
||||
disk = get_available_disk_gb(".")
|
||||
load = get_load_avg()
|
||||
mem_str = f"{mem:.1f}G" if mem is not None else "未知(非Linux?)"
|
||||
disk_str = f"{disk:.1f}G" if disk is not None else "未知"
|
||||
load_str = (f"{load[0]:.2f}/{load[1]:.2f}/{load[2]:.2f}" if load else "未知")
|
||||
logging.info("=" * 56)
|
||||
logging.info(f"机器资源: CPU {cpu} 核 | 可用内存 {mem_str} | 可用磁盘 {disk_str} | 负载(1/5/15) {load_str}")
|
||||
_exec_str = "进程池(多核,绕GIL)" if args.executor == "process" else "线程池(GIL会串行CPU计算)"
|
||||
logging.info(f"并发设置: {_exec_str} | max-workers={args.max_workers} | "
|
||||
f"内存保护阈值={args.mem_guard_gb}G | 磁盘保护阈值={args.disk_guard_gb}G")
|
||||
if args.executor == "process" and cpu:
|
||||
logging.info(f"提示: 进程池模式建议 max-workers ≈ CPU核数({cpu})上下，每个视频独立进程真多核；"
|
||||
f"内存够的话(每进程约+0.5~1G含whisper模型)可略超")
|
||||
if mem is not None and args.mem_guard_gb >= mem:
|
||||
logging.warning(f"⚠️ 内存保护阈值({args.mem_guard_gb}G) ≥ 当前可用内存({mem:.1f}G)，"
|
||||
f"几乎所有视频都会被卡住等待，请调小 --mem-guard-gb 或释放内存")
|
||||
if disk is not None and args.disk_guard_gb >= disk:
|
||||
logging.warning(f"⚠️ 磁盘保护阈值({args.disk_guard_gb}G) ≥ 当前可用磁盘({disk:.1f}G)，"
|
||||
f"几乎所有视频都会被卡住等待，请调小 --disk-guard-gb 或清理磁盘")
|
||||
if cpu and args.max_workers > cpu * 3:
|
||||
logging.warning(f"⚠️ max-workers({args.max_workers}) > CPU核数({cpu})×3，"
|
||||
f"可能超额订阅反而变慢")
|
||||
if cpu and args.max_workers < cpu:
|
||||
logging.info(f"提示: max-workers({args.max_workers}) < CPU核数({cpu})，"
|
||||
f"CPU 可能吃不满，可适当调大")
|
||||
logging.info("=" * 56)
|
||||
|
||||
split_cfg = config.get_splitframe_config()
|
||||
coscli_path = split_cfg.get('coscli_path', './coscli-linux-amd64')
|
||||
cos_config_path = split_cfg.get('cosconfig_path', '.cos.yaml')
|
||||
|
||||
processor = VideoProcessor(max_workers=args.max_workers, mem_guard_gb=args.mem_guard_gb,
|
||||
executor=args.executor, max_tasks_per_child=args.max_tasks_per_child,
|
||||
disk_guard_gb=args.disk_guard_gb)
|
||||
|
||||
# 从 数据.jsonl 读取历史，支持跨批次断点续传
|
||||
processed = load_processed(JSONL_PATH)
|
||||
if processed:
|
||||
logging.info(f"已读取 {len(processed)} 条历史记录，将跳过已处理视频")
|
||||
|
||||
os.makedirs("./video", exist_ok=True)
|
||||
|
||||
# 每日备份后台线程: 本地模式也自动上传 数据.jsonl + 超长列表到 COS(与 COS 模式一致)。
|
||||
# 仅在 dispatch 已配置备份地址时启用; 上传函数内部对缺失/失败都做了容错。
|
||||
_backup_stop = threading.Event()
|
||||
_dcfg = config.get_dispatch_config() or {}
|
||||
if _dcfg.get("enabled", True) and (_dcfg.get("daily_backup_path") or _dcfg.get("jobs_root")):
|
||||
_mid = job_cos.get_machine_id()
|
||||
_hour = int(_dcfg.get("daily_backup_hour", 10))
|
||||
threading.Thread(target=job_cos.run_daily_backup_loop,
|
||||
args=(_mid, _hour, _backup_stop, JSONL_PATH),
|
||||
daemon=True).start()
|
||||
logging.info(f"[每日备份] 已启用: 每天{_hour}点上传 数据.jsonl + 超长列表 (机器号={_mid})")
|
||||
|
||||
while True:
|
||||
txt_files = glob.glob(os.path.join(args.source_list, "*.txt"))
|
||||
if not txt_files:
|
||||
break
|
||||
|
||||
file_path = txt_files[0]
|
||||
logging.info(f"开始处理任务文件: {file_path}")
|
||||
|
||||
with open(file_path, 'r', encoding='utf-8') as f:
|
||||
all_tasks = [line.strip() for line in f if line.strip()]
|
||||
|
||||
# 跳过已处理
|
||||
tasks = [t for t in all_tasks if t not in processed]
|
||||
skipped = len(all_tasks) - len(tasks)
|
||||
if skipped:
|
||||
logging.info(f"跳过 {skipped} 条已处理记录")
|
||||
|
||||
if tasks:
|
||||
processor.process_batch(tasks, coscli_path, cos_config_path, split_cfg)
|
||||
processed.update(tasks)
|
||||
|
||||
# 清理共享临时目录
|
||||
for d in ["./info", "./info_error", "./log", "./video"]:
|
||||
if os.path.exists(d):
|
||||
force_rmtree(d)
|
||||
|
||||
os.makedirs("./video", exist_ok=True)
|
||||
os.remove(file_path)
|
||||
logging.info(f"已完成并删除任务文件: {file_path}")
|
||||
|
||||
_backup_stop.set()
|
||||
# 收尾: 本地批次跑完即上传一次(不必等到点), 保证哪怕在 backup_hour 之前结束也传到 COS。
|
||||
if _dcfg.get("enabled", True) and (_dcfg.get("daily_backup_path") or _dcfg.get("jobs_root")):
|
||||
try:
|
||||
_mid = job_cos.get_machine_id()
|
||||
_d = job_cos.upload_daily_backup(JSONL_PATH, _mid)
|
||||
if _d:
|
||||
logging.info(f"[收尾备份] 数据.jsonl 已上传 -> {_d}")
|
||||
_l = job_cos.upload_longlist_backup(_mid)
|
||||
if _l:
|
||||
logging.info(f"[收尾备份] 超长待重跑列表已上传 -> {_l}")
|
||||
except Exception as e:
|
||||
logging.error(f"[收尾备份] 上传失败: {e}")
|
||||
logging.info("所有任务处理完成")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
31
pyproject.toml
Normal file
31
pyproject.toml
Normal file
@@ -0,0 +1,31 @@
|
||||
[project]
|
||||
name = "ocr-subtitle-extractor"
|
||||
version = "0.1.0"
|
||||
description = "视频字幕提取系统：抽帧 + OCR + 语音识别，输出 SRT"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10,<3.13"
|
||||
|
||||
# 运行时依赖（与 requirements.txt 保持一致）
|
||||
dependencies = [
|
||||
# 图像与数值处理
|
||||
"opencv-python-headless>=4.8.0", # headless 版无需系统 libGL，适合无界面/离线服务器
|
||||
"numpy>=1.24,<3.0",
|
||||
"Pillow>=10.0.0",
|
||||
"scikit-learn>=1.3.0", # DBSCAN 水印聚类
|
||||
# 中文文本处理
|
||||
"jieba>=0.42.1", # 分词
|
||||
"pypinyin>=0.49.0", # 拼音匹配（音频与 OCR 文本对齐）
|
||||
"zhconv>=1.4.3", # 繁简转换
|
||||
# 音频与语音识别
|
||||
"pydub>=0.25.1", # 音频切片（依赖外部 ffmpeg）
|
||||
"av>=11.0,<14.0", # faster-whisper 依赖；钉带 wheel 的版本，免本地编译
|
||||
"faster-whisper>=1.0.0", # Whisper 语音识别（int8 量化，CPU/GPU）
|
||||
# 腾讯云
|
||||
"cos-python-sdk-v5>=1.9.0", # COS 对象存储（import qcloud_cos）
|
||||
"requests>=2.28.0", # 内部 OCR API 调用
|
||||
]
|
||||
|
||||
# 国内镜像（华为云），让 uv sync / uv pip 默认走这里
|
||||
[[tool.uv.index]]
|
||||
url = "https://repo.huaweicloud.com/repository/pypi/simple"
|
||||
default = true
|
||||
23
requirements.txt
Normal file
23
requirements.txt
Normal file
@@ -0,0 +1,23 @@
|
||||
# 视频字幕提取系统依赖
|
||||
# 安装: pip install -r requirements.txt
|
||||
# 另需 PATH 中的外部工具: ffmpeg / ffprobe、coscli（详见 CLAUDE.md）
|
||||
|
||||
# 图像与数值处理
|
||||
opencv-python-headless>=4.8.0 # headless 版无需系统 libGL，适合无界面/离线服务器
|
||||
numpy>=1.24,<3.0
|
||||
Pillow>=10.0.0
|
||||
scikit-learn>=1.3.0 # DBSCAN 水印聚类
|
||||
|
||||
# 中文文本处理
|
||||
jieba>=0.42.1 # 分词
|
||||
pypinyin>=0.49.0 # 拼音匹配（音频与OCR文本对齐）
|
||||
zhconv>=1.4.3 # 繁简转换
|
||||
|
||||
# 音频与语音识别
|
||||
pydub>=0.25.1 # 音频切片（依赖 ffmpeg）
|
||||
av>=11.0,<14.0 # faster-whisper 依赖；钉带预编译wheel的版本，避免本地编译ffmpeg
|
||||
faster-whisper>=1.0.0 # Whisper 语音识别（int8 量化，CPU/GPU）
|
||||
|
||||
# 腾讯云
|
||||
cos-python-sdk-v5>=1.9.0 # COS 对象存储（import qcloud_cos）
|
||||
requests>=2.28.0 # 腾讯云 OCR API 调用
|
||||
394
run_worker.py
Normal file
394
run_worker.py
Normal file
@@ -0,0 +1,394 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""常驻 worker: 从 COS 自动认领任务分片、处理、回写完成标记, 没任务就空闲轮询;
|
||||
每天指定时刻把本地 数据.jsonl 备份到 COS。
|
||||
|
||||
用法:
|
||||
python run_worker.py --max-workers 32 --executor process
|
||||
|
||||
机器ID、取任务地址(jobs_root)、每日备份位置(daily_backup_path)等全部读 config.dispatch。
|
||||
"""
|
||||
import argparse
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
import os
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
from datetime import datetime
|
||||
|
||||
from config.config_utils import config
|
||||
from tool import job_cos
|
||||
# 复用 main.py 既有的处理流水线与清理逻辑
|
||||
from main import (VideoProcessor, load_processed,
|
||||
force_rmtree, JSONL_PATH, get_available_disk_gb)
|
||||
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s [%(levelname)s] %(message)s',
|
||||
handlers=[
|
||||
# 常驻worker一跑就是数天/数周, 普通FileHandler会无限增长; 100M/份, 最多留5份(共500M封顶)
|
||||
RotatingFileHandler('batch_processor.log', maxBytes=100 * 1024 * 1024, backupCount=5, encoding='utf-8'),
|
||||
logging.StreamHandler(),
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
# ------------------------------------------------------------ 取下一片 ----
|
||||
|
||||
def _next_shard(machine_id: str, shuffle: bool, claim_match=None):
|
||||
"""返回下一片要做的 shard 名; 没有可领返回 None。
|
||||
优先级: ①续做我名下未完成的(重启续跑, 不受 claim_match 限制) -> ②原子认领一片新的(受 claim_match 过滤)。"""
|
||||
# ① 粘性续做: 我上次没跑完的分片(别的机器不会碰; 已经是我的, 不再过滤)
|
||||
for shard in job_cos.my_open_claims(machine_id):
|
||||
logging.info(f"[续做] 接回我名下未完成分片: {shard}")
|
||||
return shard
|
||||
# ② 认领新分片
|
||||
todo = job_cos.list_todo()
|
||||
# 排除已完成 / 已被锁的
|
||||
todo = [s for s in todo if not job_cos.is_done(s)]
|
||||
# 只认领名称匹配的(claim_match 留空=全部)
|
||||
if claim_match:
|
||||
todo = [s for s in todo if job_cos.shard_matches(s, claim_match)]
|
||||
if shuffle:
|
||||
random.shuffle(todo)
|
||||
for shard in todo:
|
||||
if job_cos.read_lock(shard) is not None:
|
||||
continue # 已被人锁
|
||||
if job_cos.claim(shard, machine_id):
|
||||
logging.info(f"[认领] 抢到分片: {shard}")
|
||||
return shard
|
||||
# 409: 刚好被别人抢走, 试下一片
|
||||
return None
|
||||
|
||||
|
||||
def _claim_long_batch(machine_id: str, shuffle: bool, min_batch: int):
|
||||
"""长视频队列专用: 一次性多领几片, 累计任务数够 min_batch(或没片可领了)才停手 ——
|
||||
长视频队列每天上传的还是各机器各自攒的小分片(可能就几十条), 这里只是在【领取】这一侧
|
||||
把多个小分片攒成一大批一起处理, 不改上传那边的行为。
|
||||
先照 _next_shard() 同样的优先级续做粘性未完成分片(不受 min_batch 限制, 必须先了结),
|
||||
没有才开始新领; min_batch<=0 视为1(至少领一片，不做批量)。
|
||||
返回 [(shard, tasks), ...]；某一片没片可领时提前结束(不会因为凑不够 min_batch 空等)。"""
|
||||
min_batch = max(1, int(min_batch or 1))
|
||||
batch = []
|
||||
total = 0
|
||||
|
||||
# ① 粘性续做: 我名下所有未完成分片先原样接回来, 不受 min_batch 限制(必须先了结)
|
||||
open_claims = job_cos.my_open_claims(machine_id)
|
||||
for shard in open_claims:
|
||||
tasks = job_cos.download_shard(shard)
|
||||
batch.append((shard, tasks))
|
||||
total += len(tasks)
|
||||
if batch:
|
||||
logging.info(f"[长视频批量] 接回 {len(batch)} 片未完成分片, 共{total}条, 直接处理(不再新领)")
|
||||
return batch
|
||||
|
||||
# ② 新领: 不断认领新分片直到攒够 min_batch 条(或没片可领了)
|
||||
todo = job_cos.list_todo()
|
||||
todo = [s for s in todo if not job_cos.is_done(s)]
|
||||
if shuffle:
|
||||
random.shuffle(todo)
|
||||
for shard in todo:
|
||||
if total >= min_batch:
|
||||
break
|
||||
if job_cos.read_lock(shard) is not None:
|
||||
continue # 已被人锁
|
||||
if not job_cos.claim(shard, machine_id):
|
||||
continue # 409: 刚好被别人抢走
|
||||
tasks = job_cos.download_shard(shard)
|
||||
batch.append((shard, tasks))
|
||||
total += len(tasks)
|
||||
logging.info(f"[长视频批量] 认领分片: {shard} ({len(tasks)}条, 累计{total}/{min_batch})")
|
||||
return batch
|
||||
|
||||
|
||||
def _process_shard_batch(batch, processor, machine_id, split_cfg,
|
||||
coscli_path, cos_config_path, hb_interval, global_processed=None):
|
||||
"""处理一批长视频分片(多片任务清单先合并成一个大列表再统一跑一次 process_batch,
|
||||
而不是一片一片分别跑) —— 逻辑和 _process_shard 一致, 只是输入是 [(shard, tasks), ...]。
|
||||
完成后把批次里每一片都各自 mark_done + release(全部标同一份聚合统计, 只是省事,
|
||||
不是精确到每片的独立统计, 不影响正确性)。"""
|
||||
shards = [s for s, _ in batch]
|
||||
tasks = [t for _, ts in batch for t in ts]
|
||||
processed = load_processed(JSONL_PATH)
|
||||
if global_processed:
|
||||
processed = processed | global_processed
|
||||
todo_tasks = [t for t in tasks if t not in processed]
|
||||
skipped = len(tasks) - len(todo_tasks)
|
||||
logging.info(f"长视频批次({len(shards)}片): 共 {len(tasks)} 条, 跳过已处理 {skipped}, "
|
||||
f"待处理 {len(todo_tasks)}")
|
||||
|
||||
hb_stop = threading.Event()
|
||||
def _hb():
|
||||
while not hb_stop.wait(hb_interval):
|
||||
for shard in shards:
|
||||
try:
|
||||
job_cos.heartbeat(shard, machine_id)
|
||||
except Exception as e:
|
||||
logging.warning(f"心跳刷新失败 {shard}: {e}")
|
||||
threading.Thread(target=_hb, daemon=True).start()
|
||||
|
||||
stats_dict = {"total": len(todo_tasks), "success": 0, "no_subtitle": 0, "error": 0}
|
||||
try:
|
||||
if todo_tasks:
|
||||
stats = processor.process_batch(todo_tasks, coscli_path, cos_config_path, split_cfg)
|
||||
stats_dict.update({"success": stats.success,
|
||||
"no_subtitle": stats.no_subtitle, "error": stats.error})
|
||||
finally:
|
||||
hb_stop.set()
|
||||
|
||||
for shard in shards:
|
||||
job_cos.mark_done(shard, machine_id, stats_dict)
|
||||
job_cos.release(shard)
|
||||
logging.info(f"长视频批次({len(shards)}片)完成并释放: {stats_dict}")
|
||||
|
||||
for d in ["./info", "./info_error", "./log", "./video"]:
|
||||
if os.path.exists(d):
|
||||
force_rmtree(d)
|
||||
os.makedirs("./video", exist_ok=True)
|
||||
|
||||
|
||||
def _fetch_global_processed(attempts: int = 3, backoff: float = 10.0) -> set:
|
||||
"""启动时拉全局已处理集合, 失败重试(COS 抖动/网络问题很常见, 直接放弃太可惜)。
|
||||
每次失败等 backoff 秒(线性,不是指数,重试次数少没必要搞复杂)再试; 全部失败返回空集合,
|
||||
调用方会据此退化为只按本机 数据.jsonl 去重, 不阻塞 worker 启动。"""
|
||||
last_err = None
|
||||
for i in range(1, attempts + 1):
|
||||
try:
|
||||
return job_cos.global_processed_set()
|
||||
except Exception as e:
|
||||
last_err = e
|
||||
logging.warning(f"[全局去重] 第{i}/{attempts}次拉取失败: {e}")
|
||||
if i < attempts:
|
||||
time.sleep(backoff)
|
||||
logging.warning(f"[全局去重] 重试{attempts}次仍失败, 本次运行退化为只按本机 数据.jsonl 去重: {last_err}")
|
||||
return set()
|
||||
|
||||
|
||||
def _process_shard(shard, processor, machine_id, split_cfg,
|
||||
coscli_path, cos_config_path, hb_interval, global_processed=None):
|
||||
"""处理一个分片: 下载清单 -> 跳过(本机已处理 ∪ 全局已处理) -> 跑流水线 -> 写 done -> 释放。
|
||||
global_processed: 启动时拉取的全局已处理集合(见 main() 里的 job_cos.global_processed_set()),
|
||||
用于跨机器去重 —— 本机 数据.jsonl 只挡得住"这台机器"重复跑同一个视频, 挡不住这个视频被
|
||||
错误地灌进了另一台机器认领的分片; 传 None/空集合时退化为只按本机去重(旧行为)。"""
|
||||
tasks = job_cos.download_shard(shard)
|
||||
processed = load_processed(JSONL_PATH) # 本机已处理(续跑去重)
|
||||
if global_processed:
|
||||
processed = processed | global_processed # 并上全局已处理(跨机器去重)
|
||||
todo_tasks = [t for t in tasks if t not in processed]
|
||||
skipped = len(tasks) - len(todo_tasks)
|
||||
logging.info(f"分片 {shard}: 共 {len(tasks)} 条, 跳过已处理 {skipped}, 待处理 {len(todo_tasks)}")
|
||||
|
||||
# 心跳线程: 处理期间周期刷新锁的 hb_ts(留作监控/未来回收用)
|
||||
hb_stop = threading.Event()
|
||||
def _hb():
|
||||
while not hb_stop.wait(hb_interval):
|
||||
try:
|
||||
job_cos.heartbeat(shard, machine_id)
|
||||
except Exception as e:
|
||||
logging.warning(f"心跳刷新失败 {shard}: {e}")
|
||||
threading.Thread(target=_hb, daemon=True).start()
|
||||
|
||||
stats_dict = {"total": len(todo_tasks), "success": 0, "no_subtitle": 0, "error": 0}
|
||||
try:
|
||||
if todo_tasks:
|
||||
stats = processor.process_batch(todo_tasks, coscli_path, cos_config_path, split_cfg)
|
||||
stats_dict.update({"success": stats.success,
|
||||
"no_subtitle": stats.no_subtitle, "error": stats.error})
|
||||
finally:
|
||||
hb_stop.set()
|
||||
|
||||
# 完成: 写 done -> 删 todo + lock
|
||||
job_cos.mark_done(shard, machine_id, stats_dict)
|
||||
job_cos.release(shard)
|
||||
logging.info(f"分片 {shard} 完成并释放: {stats_dict}")
|
||||
|
||||
# 清理共享临时目录(沿用 main 的批次后清理)
|
||||
for d in ["./info", "./info_error", "./log", "./video"]:
|
||||
if os.path.exists(d):
|
||||
force_rmtree(d)
|
||||
os.makedirs("./video", exist_ok=True)
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="常驻 worker: 从 COS 认领任务并处理")
|
||||
ap.add_argument('--max-workers', type=int, default=None,
|
||||
help='不填/0 = 自动按本机CPU核数选')
|
||||
ap.add_argument('--mem-guard-gb', type=float, default=2.0)
|
||||
ap.add_argument('--disk-guard-gb', type=float, default=10.0,
|
||||
help='本地磁盘可用空间低于此值(GB)时暂停启动新视频，给数据处理预留空间；设 0 关闭')
|
||||
ap.add_argument('--max-tasks-per-child', type=int, default=30)
|
||||
ap.add_argument('--executor', choices=['process', 'thread'], default='process')
|
||||
args = ap.parse_args()
|
||||
|
||||
if not args.max_workers:
|
||||
cpu = os.cpu_count() or 0
|
||||
args.max_workers = job_cos.auto_max_workers(cpu, disk_guard_gb=args.disk_guard_gb)
|
||||
logging.info(f"--max-workers 未指定, 自动选为 {args.max_workers}(本机CPU核数={cpu or '未知'}; "
|
||||
f"若本机还跑了OCR服务但不在 tencentcloud.ocr.endpoints 里, 自动探测不到, 请手动传小一点的值; "
|
||||
f"同时按可用磁盘 ÷ 每视频预估占用(dispatch.est_disk_gb_per_video)做了收紧)")
|
||||
# 供子进程(forkserver fork出来的worker)读取: splitframe.ffmpeg_threads=auto 时按此摊分CPU
|
||||
os.environ['OCR_MAX_WORKERS'] = str(args.max_workers)
|
||||
|
||||
dcfg = config.get_dispatch_config() or {}
|
||||
if not dcfg.get("enabled", True):
|
||||
logging.error("config.dispatch.enabled=false, 退出。如需启用请改 config。")
|
||||
return
|
||||
|
||||
machine_id = job_cos.get_machine_id()
|
||||
keep_alive = bool(dcfg.get("keep_alive", True))
|
||||
idle_poll = int(dcfg.get("idle_poll_seconds", 60))
|
||||
shuffle = bool(dcfg.get("claim_shuffle", True))
|
||||
hb_interval = int(dcfg.get("heartbeat_seconds", 300))
|
||||
backup_hour = int(dcfg.get("daily_backup_hour", 10))
|
||||
claim_match = dcfg.get("claim_match") or None # 只领匹配的分片(留空=全部)
|
||||
|
||||
run_long = bool(dcfg.get("run_long_video"))
|
||||
long_root = dcfg.get("long_list_backup_path") or None
|
||||
# 只有"本来跑主队列"的机器(非专职长视频机)、且配了长视频队列地址、且没有被显式关掉时,
|
||||
# 才会在主队列空闲时顺手兼跑长视频队列 —— 专职长视频机本来就一直在长视频队列里, 不涉及
|
||||
# 这个开关。auto_fallback_long_video 默认 true(维持原行为); 想让某台机器只专心跑主队列、
|
||||
# 空闲就等着(不去碰长视频)的话, 在这台机器的 config.json 里配 false 即可, 不影响它自己
|
||||
# 超长待重跑清单的每日备份(那个走 job_cos.upload_longlist_backup 各自独立)。
|
||||
fallback_enabled = dcfg.get("auto_fallback_long_video", True)
|
||||
auto_fallback_long = (not run_long) and bool(long_root) and bool(fallback_enabled)
|
||||
active_root = job_cos._active_jobs_root()
|
||||
logging.info("=" * 56)
|
||||
logging.info(f"机器ID: {machine_id} | {'【专跑长视频】队列' if run_long else '主队列'}: {active_root}")
|
||||
if auto_fallback_long:
|
||||
logging.info(f"主队列空闲时自动兼跑长视频队列: {long_root}")
|
||||
if claim_match:
|
||||
logging.info(f"只认领匹配分片: {claim_match}")
|
||||
logging.info(f"常驻: {keep_alive} 空闲轮询: {idle_poll}s | 每日备份: 每天{backup_hour}点")
|
||||
disk = get_available_disk_gb(".")
|
||||
disk_str = f"{disk:.1f}G" if disk is not None else "未知"
|
||||
logging.info(f"磁盘保护阈值: {args.disk_guard_gb}G | 当前可用磁盘: {disk_str}")
|
||||
if disk is not None and args.disk_guard_gb >= disk:
|
||||
logging.warning(f"⚠️ 磁盘保护阈值({args.disk_guard_gb}G) ≥ 当前可用磁盘({disk:.1f}G)，"
|
||||
f"几乎所有视频都会被卡住等待，请调小 --disk-guard-gb 或清理磁盘")
|
||||
logging.info("=" * 56)
|
||||
|
||||
# 启动时拉一次全局已处理集合(各机器最新一天的已处理清单, 见 job_cos.global_processed_set),
|
||||
# 跟本机 数据.jsonl 一起用于分片去重 —— 挡住"这个视频被错误地灌进了另一台机器分片"的情况。
|
||||
# 只在启动时拉一次、缓存到本次运行结束, 不是每领一片分片都重新拉(清单本身也就每天更新一次,
|
||||
# 没必要频繁重拉); 失败会重试(见 _fetch_global_processed), 重试仍失败才退化为只按本机去重。
|
||||
# global_dedup_on_startup=false 时直接跳过这一步、只按本机 数据.jsonl 去重 —— 每台机器的
|
||||
# 已处理清单可能有几MB, 20多台机器挨个拉下来这一步可能耗时较久(且下载期间没有中间日志输出,
|
||||
# 看起来像卡住了), 网络不好/机器多的场合可以关掉。
|
||||
global_processed = set()
|
||||
if dcfg.get("daily_backup_path") and dcfg.get("global_dedup_on_startup", True):
|
||||
global_processed = _fetch_global_processed()
|
||||
if global_processed:
|
||||
logging.info(f"[全局去重] 启动时拉取全局已处理集合: {len(global_processed)} 条")
|
||||
else:
|
||||
logging.info("[全局去重] 已关闭(global_dedup_on_startup=false), 只按本机 数据.jsonl 去重")
|
||||
|
||||
split_cfg = config.get_splitframe_config()
|
||||
coscli_path = split_cfg.get('coscli_path', './coscli-linux-amd64')
|
||||
cos_config_path = split_cfg.get('cosconfig_path', '.cos.yaml')
|
||||
processor = VideoProcessor(max_workers=args.max_workers, mem_guard_gb=args.mem_guard_gb,
|
||||
executor=args.executor, max_tasks_per_child=args.max_tasks_per_child,
|
||||
disk_guard_gb=args.disk_guard_gb)
|
||||
|
||||
os.makedirs("./video", exist_ok=True)
|
||||
|
||||
# 每日备份后台线程(数据.jsonl + 超长列表)
|
||||
stop = threading.Event()
|
||||
threading.Thread(target=job_cos.run_daily_backup_loop,
|
||||
args=(machine_id, backup_hour, stop, JSONL_PATH),
|
||||
daemon=True).start()
|
||||
|
||||
# 每小时监控摘要上传线程(常态化监控用, 见 docs/superpowers/specs/2026-07-08-fleet-monitoring-design.md)
|
||||
threading.Thread(target=job_cos.run_hourly_digest_loop,
|
||||
args=(machine_id, stop, JSONL_PATH),
|
||||
daemon=True).start()
|
||||
|
||||
# 长视频队列每次【领取】攒够这么多条任务才开始处理(领的时候多片一起领, 上传那边还是各机器
|
||||
# 各自每天传小分片不受影响) —— 见 _claim_long_batch。<=0 视为1(不做批量, 领一片处理一片)。
|
||||
min_long_batch = int(dcfg.get("long_list_min_batch", 1000) or 1)
|
||||
|
||||
try:
|
||||
while True:
|
||||
if job_cos.stop_requested():
|
||||
logging.info("检测到 COS STOP 标记, 优雅停机。")
|
||||
break
|
||||
|
||||
# 专职长视频机: 全程都在长视频队列, 直接批量领+批量处理, 不走主队列那套单片逻辑
|
||||
if run_long:
|
||||
try:
|
||||
batch = _claim_long_batch(machine_id, shuffle, min_long_batch)
|
||||
except Exception as e:
|
||||
logging.error(f"✗ 访问长视频队列失败(稍后重试): {e}")
|
||||
time.sleep(min(idle_poll, 30))
|
||||
continue
|
||||
if not batch:
|
||||
if keep_alive:
|
||||
logging.info(f"长视频队列无可领任务, 空闲等待 {idle_poll}s...")
|
||||
time.sleep(idle_poll)
|
||||
continue
|
||||
else:
|
||||
logging.info("长视频队列无可领任务, keep_alive=false, 退出。")
|
||||
break
|
||||
try:
|
||||
_process_shard_batch(batch, processor, machine_id, split_cfg,
|
||||
coscli_path, cos_config_path, hb_interval, global_processed)
|
||||
except Exception as e:
|
||||
logging.error(f"✗ 长视频批次处理出错(锁保留, 重启可续做): {e}")
|
||||
time.sleep(5)
|
||||
continue
|
||||
|
||||
# ① 先看主队列(单片领取, 行为不变)
|
||||
try:
|
||||
shard = _next_shard(machine_id, shuffle, claim_match)
|
||||
except Exception as e:
|
||||
logging.error(f"✗ 访问 COS 队列失败(稍后重试): {e}")
|
||||
time.sleep(min(idle_poll, 30))
|
||||
continue
|
||||
|
||||
# ② 主队列没活 -> 顺手看看长视频队列有没有能领的(批量领, 不是单片)
|
||||
used_long = False
|
||||
long_batch = None
|
||||
if shard is None and auto_fallback_long:
|
||||
job_cos.set_active_root_override(long_root)
|
||||
try:
|
||||
long_batch = _claim_long_batch(machine_id, shuffle, min_long_batch)
|
||||
except Exception as e:
|
||||
logging.error(f"✗ 访问长视频队列失败(稍后重试): {e}")
|
||||
job_cos.set_active_root_override(None)
|
||||
time.sleep(min(idle_poll, 30))
|
||||
continue
|
||||
if not long_batch:
|
||||
job_cos.set_active_root_override(None) # 长视频队列也没活, 恢复主队列上下文
|
||||
else:
|
||||
used_long = True
|
||||
logging.info(f"主队列空闲, 转去处理长视频队列批次: {len(long_batch)}片")
|
||||
|
||||
if shard is None and not long_batch:
|
||||
# 两个队列都真·没活了
|
||||
if keep_alive:
|
||||
logging.info(f"无可领任务, 空闲等待 {idle_poll}s(加新任务到 todo/ 会自动接上)...")
|
||||
time.sleep(idle_poll)
|
||||
continue
|
||||
else:
|
||||
logging.info("无可领任务, keep_alive=false, 退出。")
|
||||
break
|
||||
|
||||
try:
|
||||
if long_batch:
|
||||
_process_shard_batch(long_batch, processor, machine_id, split_cfg,
|
||||
coscli_path, cos_config_path, hb_interval, global_processed)
|
||||
else:
|
||||
_process_shard(shard, processor, machine_id, split_cfg,
|
||||
coscli_path, cos_config_path, hb_interval, global_processed)
|
||||
except Exception as e:
|
||||
logging.error(f"✗ 分片处理出错(锁保留, 重启可续做): {e}")
|
||||
time.sleep(5)
|
||||
finally:
|
||||
if used_long:
|
||||
job_cos.set_active_root_override(None) # 这批长视频处理完/出错, 下一轮先回去看主队列
|
||||
finally:
|
||||
stop.set()
|
||||
logging.info("worker 退出。")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
167
scripts/apply_update.sh
Normal file
167
scripts/apply_update.sh
Normal file
@@ -0,0 +1,167 @@
|
||||
#!/usr/bin/env bash
|
||||
# 一键更新脚本(在线上【项目根目录】运行)。
|
||||
# 作用: 用 _new/ 里的新代码覆盖改过的文件, 并把新配置项【合并】进现有 config.json
|
||||
# (只加缺失的键, 绝不动你已有的密钥/路径/已调过的值), 然后自检。
|
||||
# 用法: 解压更新包到项目根目录后, 执行 bash apply_update.sh
|
||||
set -u
|
||||
|
||||
ROOT="$(pwd)"
|
||||
TS="$(date +%Y%m%d_%H%M%S)"
|
||||
BAK="_backup_${TS}"
|
||||
NEW="_new"
|
||||
|
||||
echo "==> 项目目录: $ROOT"
|
||||
|
||||
if [ ! -d "tool" ] || [ ! -f "config/config.json" ]; then
|
||||
echo "✗ 当前目录不像项目根(缺 tool/ 或 config/config.json)。请 cd 到项目根再运行。"
|
||||
exit 1
|
||||
fi
|
||||
if [ ! -d "$NEW" ]; then
|
||||
echo "✗ 找不到 $NEW/ 目录。请先把更新包解压到项目根目录。"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 0. 更新前先把本机 数据.jsonl + 超长待重跑列表 备份到 COS(不等每天10点的自动备份)。
|
||||
# 更新代码本身不会碰 数据.jsonl, 这一步纯粹是"顺手让全局数据保持最新"(比如全局去重要用),
|
||||
# COS 访问失败不阻断更新, 只提示。
|
||||
echo "==> 更新前备份 数据.jsonl / 超长待重跑列表 到 COS"
|
||||
python3 tools/upload_daily.py || echo " ⚠ 备份失败(不影响本次更新继续), 稍后可手动重跑: python3 tools/upload_daily.py"
|
||||
|
||||
FILES="main.py run_worker.py tool/process.py tool/merge.py tool/getsrt.py tool/whisper_manager.py tool/VideoProcessing/multithread_algo.py tool/Ocr/tencentcloud_ocr.py tool/job_cos.py config/config_utils.py"
|
||||
|
||||
# 1. 备份
|
||||
echo "==> 备份到 $BAK/"
|
||||
mkdir -p "$BAK/tool/VideoProcessing" "$BAK/tool/Ocr" "$BAK/config"
|
||||
for f in $FILES; do
|
||||
[ -f "$f" ] && cp -p "$f" "$BAK/$f"
|
||||
done
|
||||
cp -p config/config.json "$BAK/config/config.json"
|
||||
|
||||
# 2. 覆盖代码文件
|
||||
echo "==> 更新代码文件"
|
||||
for f in $FILES; do
|
||||
if [ ! -f "$NEW/$f" ]; then
|
||||
echo "✗ 更新包缺少 $NEW/$f, 中止(已备份, 未改动)"; exit 1
|
||||
fi
|
||||
cp -p "$NEW/$f" "$f"
|
||||
echo " ✓ $f"
|
||||
done
|
||||
|
||||
# 2b. 附带的助手脚本(有就拷, 没有不报错)
|
||||
for f in tools/拆分超长.py tools/seed_jobs.py tools/upload_daily.py tools/release_machine.py tools/patch_config.py; do
|
||||
if [ -f "$NEW/$f" ]; then
|
||||
mkdir -p "$(dirname "$f")"
|
||||
cp -p "$NEW/$f" "$f" && echo " ✓ $f"
|
||||
fi
|
||||
done
|
||||
|
||||
# 3. 合并配置(只加缺失键, 不覆盖已有值; 报告与推荐值不同的键)
|
||||
echo "==> 合并 config.json 新配置项(保留你已有的密钥/路径/设置)"
|
||||
python3 - "$ROOT/config/config.json" <<'PYEOF'
|
||||
import json, sys
|
||||
p = sys.argv[1]
|
||||
d = json.load(open(p, encoding="utf-8"))
|
||||
au = d.setdefault("audio", {})
|
||||
sf = d.setdefault("splitframe", {})
|
||||
sub = d.setdefault("subtitle", {})
|
||||
ocr = d.setdefault("tencentcloud", {}).setdefault("ocr", {})
|
||||
dp = d.setdefault("dispatch", {})
|
||||
_bkt = (d.get("tencentcloud", {}).get("cos", {}) or {}).get("bucket", "asr-dig-data-1253181912")
|
||||
|
||||
NEW_KEYS = [
|
||||
(ocr, "max_attempts", 4),
|
||||
(ocr, "timeout", 30),
|
||||
(ocr, "dispatch_log_every", 1000),
|
||||
(au, "cpu_threads", 2),
|
||||
(sf, "crop_bottom_ratio", None),
|
||||
(sub, "min_duration_seconds", 300),
|
||||
(sub, "no_composite", False),
|
||||
(sub, "merge_workers", 8),
|
||||
(sub, "max_tiles_per_composite", 24),
|
||||
(sub, "ocr_max_image_side", 4000),
|
||||
(sub, "cv_detect", True),
|
||||
(sub, "cv_use_region", False),
|
||||
(sub, "filter_static", True),
|
||||
(sub, "static_frac", 0.25),
|
||||
(sub, "filter_lowconf", True),
|
||||
(sub, "lowconf_thr", 80),
|
||||
(sub, "lowconf_filter_frac", 0.6),
|
||||
(sub, "lowconf_min_det", 200),
|
||||
(sub, "strip_static_prefix", False),
|
||||
(sub, "filter_watermark_pos", True),
|
||||
(sub, "wm_dominant_frac", 0.25),
|
||||
(sub, "wm_min_count", 20),
|
||||
(sub, "split_far_boxes", True),
|
||||
(sub, "max_duration_seconds", 5400),
|
||||
(sub, "long_video_list", "./超长待重跑.txt"),
|
||||
(sub, "ocr_no_drop", True),
|
||||
(sub, "ocr_max_passes", 5),
|
||||
(sub, "ocr_retry_sleep", 2),
|
||||
(sub, "ocr_fail_list", "./OCR失败待重跑.txt"),
|
||||
(sub, "no_subtitle_reasons", ["字幕太少","冗余文字过多","冗余文字太多","无字幕区域","非字幕内容","置信度太低","未检测","特殊字体","视频过短","视频过长"]),
|
||||
(sf, "ffmpeg_timeout", 3600),
|
||||
(sf, "ffmpeg_threads", "auto"),
|
||||
(sf, "gpu_skip_indices", []),
|
||||
(sf, "gpu_auto_select", True),
|
||||
(sf, "gpu_busy_mem_mb", 4096),
|
||||
(dp, "enabled", True),
|
||||
(dp, "source", "cos"),
|
||||
(dp, "run_long_video", False),
|
||||
(dp, "auto_fallback_long_video", True),
|
||||
(dp, "global_dedup_on_startup", True),
|
||||
(dp, "machine_id", ""),
|
||||
(dp, "jobs_root", "cos://%s/_jobs" % _bkt),
|
||||
(dp, "shard_size", 2000),
|
||||
(dp, "keep_alive", True),
|
||||
(dp, "idle_poll_seconds", 60),
|
||||
(dp, "claim_shuffle", True),
|
||||
(dp, "claim_match", ""),
|
||||
(dp, "heartbeat_seconds", 300),
|
||||
(dp, "daily_backup_path", "cos://%s/_jobs/daily" % _bkt),
|
||||
(dp, "long_list_backup_path", "cos://%s/_jobs/long_video" % _bkt),
|
||||
(dp, "monitor_root", "cos://%s/_jobs/monitor" % _bkt),
|
||||
(dp, "long_list_clear_after_upload", True),
|
||||
(dp, "long_list_min_batch", 1000),
|
||||
(dp, "daily_backup_hour", 10),
|
||||
(dp, "stop_flag_key", "_jobs/STOP"),
|
||||
]
|
||||
added, differ = [], []
|
||||
for sec, k, v in NEW_KEYS:
|
||||
if k not in sec:
|
||||
sec[k] = v; added.append((k, v))
|
||||
elif sec[k] != v:
|
||||
differ.append((k, sec[k], v))
|
||||
|
||||
json.dump(d, open(p, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
|
||||
print(" 新增键:", ", ".join(f"{k}={v}" for k, v in added) or "(无, 已都存在)")
|
||||
if differ:
|
||||
print(" ⚠ 以下键你线上已有值, 未改动(如想用推荐值请手动改):")
|
||||
for k, cur, rec in differ:
|
||||
print(f" {k}: 当前={cur} 推荐={rec}")
|
||||
PYEOF
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "✗ 配置合并失败, 正在回滚 config.json"
|
||||
cp -p "$BAK/config/config.json" config/config.json
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# 4. 自检
|
||||
echo "==> 自检 import"
|
||||
if python3 -c "import main, run_worker, tool.job_cos, tool.merge, tool.process, tool.getsrt, tool.whisper_manager, tool.VideoProcessing.multithread_algo; print(' ✓ import OK')"; then
|
||||
:
|
||||
else
|
||||
echo "✗ import 失败! 正在回滚所有改动"
|
||||
for f in $FILES; do cp -p "$BAK/$f" "$f"; done
|
||||
cp -p "$BAK/config/config.json" config/config.json
|
||||
echo "已回滚到更新前。备份在 $BAK/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "============================================="
|
||||
echo "✓ 更新完成。备份在: $BAK/"
|
||||
echo " 回退: 把 $BAK/ 里的文件拷回原位即可。"
|
||||
echo ""
|
||||
echo "下一步: 先小批量验证, 没问题再上量。"
|
||||
echo " python main.py --source-list ./text2 --max-workers 32 --executor process"
|
||||
echo "============================================="
|
||||
108
scripts/benchmark.sh
Normal file
108
scripts/benchmark.sh
Normal file
@@ -0,0 +1,108 @@
|
||||
#!/usr/bin/env bash
|
||||
# 新机器上线前的算力/IO摸底测试(跟 scripts/diag.sh 不是一回事 —— diag.sh 是排查"已经在跑但
|
||||
# 出问题"的机器；这个是给"还没开始跑正式任务、想先估算这台机器大概能跑多快"的新机器用的)。
|
||||
# 测: CPU/内存/GPU(含真实硬解测试)/磁盘顺序读写速度/到OCR端点的网络延迟, 汇总成一份文件。
|
||||
#
|
||||
# 用法(项目根目录): bash scripts/benchmark.sh
|
||||
# 产物: benchmark_YYYYmmdd_HHMMSS.txt (把这个文件发回来)
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit 1
|
||||
OUT="benchmark_$(date +%Y%m%d_%H%M%S).txt"
|
||||
PY="$(command -v python || command -v python3)"
|
||||
FFMPEG="$($PY -c "from config.config_utils import get_ffmpeg; print(get_ffmpeg())" 2>/dev/null || command -v ffmpeg)"
|
||||
|
||||
exec > >(tee "$OUT") 2>&1
|
||||
|
||||
echo "================ 算力/IO摸底 $(date) ================"
|
||||
echo "主机: $(hostname)"
|
||||
echo ""
|
||||
|
||||
echo "========== 1. CPU =========="
|
||||
echo "[核数] $(nproc)"
|
||||
if command -v lscpu >/dev/null 2>&1; then
|
||||
lscpu | grep -E "Model name|CPU\(s\)|MHz|Thread|Socket"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========== 2. 内存 =========="
|
||||
free -h
|
||||
echo ""
|
||||
|
||||
echo "========== 3. GPU =========="
|
||||
if command -v nvidia-smi >/dev/null 2>&1; then
|
||||
echo "[列表]"
|
||||
nvidia-smi --query-gpu=index,name,memory.total,memory.used,utilization.gpu,driver_version --format=csv
|
||||
echo ""
|
||||
echo "[真实硬解测试: ffmpeg合成测试图案, 用CPU软编码生成素材(不碰GPU), 再用 -hwaccel cuda 解码,"
|
||||
echo " 每张卡各测一遍 —— 生产流程只用GPU【解码】下载好的视频, 从不用GPU编码, 所以这里故意不测nvenc编码,"
|
||||
echo " 只测跟生产完全一致的硬解路径]"
|
||||
if [ -x "$FFMPEG" ] || command -v "$FFMPEG" >/dev/null 2>&1; then
|
||||
n_gpu=$(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l)
|
||||
testfile="/tmp/_bench_src.mp4"
|
||||
"$FFMPEG" -hide_banner -loglevel error -f lavfi -i "testsrc=size=1920x1080:rate=30:duration=10" \
|
||||
-c:v libx264 -preset veryfast -f mp4 "$testfile" -y 2>&1 | tail -3
|
||||
if [ -f "$testfile" ]; then
|
||||
for dev in $(seq 0 $((n_gpu - 1))); do
|
||||
echo " -- GPU $dev --"
|
||||
t0=$(date +%s.%N)
|
||||
"$FFMPEG" -hide_banner -loglevel error -hwaccel cuda -hwaccel_device "$dev" \
|
||||
-i "$testfile" -f null - 2>&1 | tail -5
|
||||
t1=$(date +%s.%N)
|
||||
echo " 硬解读回耗时: $(echo "$t1 - $t0" | bc 2>/dev/null || echo "?")s (10秒素材, 越接近0越好, 明显>10s说明硬解没吃上或GPU很忙; 有报错说明硬解不可用, 会自动回退CPU软解)"
|
||||
done
|
||||
rm -f "$testfile"
|
||||
else
|
||||
echo " CPU软编码生成测试素材都失败了(跟GPU无关, 检查ffmpeg本身/libx264), 跳过硬解测试"
|
||||
fi
|
||||
else
|
||||
echo " 找不到 ffmpeg, 跳过硬解测试"
|
||||
fi
|
||||
else
|
||||
echo " 没有 nvidia-smi(不是GPU机器, 或驱动没装)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========== 4. 磁盘顺序读写速度(当前目录所在盘) =========="
|
||||
TESTFILE="./_bench_io.tmp"
|
||||
echo "[写] 顺序写1GB"
|
||||
sync
|
||||
dd if=/dev/zero of="$TESTFILE" bs=1M count=1024 conv=fdatasync 2>&1 | tail -3
|
||||
echo "[读] 顺序读(先清缓存再读更准, 没权限清缓存就直接读)"
|
||||
( sync && echo 3 > /proc/sys/vm/drop_caches ) 2>/dev/null
|
||||
dd if="$TESTFILE" of=/dev/null bs=1M 2>&1 | tail -3
|
||||
rm -f "$TESTFILE"
|
||||
echo ""
|
||||
echo "[磁盘剩余空间]"
|
||||
df -h .
|
||||
echo ""
|
||||
|
||||
echo "========== 5. 网络: 到OCR端点的延迟/连通率 =========="
|
||||
if [ -f config/config.json ] && [ -n "$PY" ]; then
|
||||
"$PY" tools/查端点存活.py --tcp --timeout 3 2>&1
|
||||
else
|
||||
echo " 跳过(找不到 python 或 config/config.json)"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========== 6. 网络: 到COS的连通性 =========="
|
||||
if command -v coscli >/dev/null 2>&1 || [ -x ./coscli-linux-amd64 ]; then
|
||||
COSCLI="$(command -v coscli || echo ./coscli-linux-amd64)"
|
||||
COSCFG="$($PY -c "from config.config_utils import config;print((config.get_splitframe_config() or {}).get('cosconfig_path','.cos.yaml'))" 2>/dev/null || echo .cos.yaml)"
|
||||
JOBS_ROOT="$($PY -c "from config.config_utils import config;print((config.get_dispatch_config() or {}).get('jobs_root',''))" 2>/dev/null)"
|
||||
if [ -n "$JOBS_ROOT" ]; then
|
||||
echo "[列 $JOBS_ROOT 耗时]"
|
||||
t0=$(date +%s.%N)
|
||||
"$COSCLI" -c "$COSCFG" ls "$JOBS_ROOT/" 2>&1 | tail -5
|
||||
t1=$(date +%s.%N)
|
||||
echo " 耗时: $(echo "$t1 - $t0" | bc 2>/dev/null || echo "?")s (纯网络往返延迟参考, 不是下载带宽)"
|
||||
else
|
||||
echo " dispatch.jobs_root 未配置, 跳过"
|
||||
fi
|
||||
else
|
||||
echo " 找不到 coscli, 跳过"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "================ 摸底结束 ================"
|
||||
echo ""
|
||||
echo ">>> 报告已保存到: $OUT (把这个文件发回来)"
|
||||
48
scripts/build_bundle.sh
Normal file
48
scripts/build_bundle.sh
Normal file
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 一键构建离线部署包（在【能联网】的打包机上跑）
|
||||
#
|
||||
# setup_server.sh(装环境) → make_portable.sh(自包含) → pack.sh(打包) 三步合一
|
||||
#
|
||||
# 用法:
|
||||
# bash build_bundle.sh cpu # CPU 版包：johnvansickle ffmpeg，老 glibc 也能跑
|
||||
# bash build_bundle.sh gpu # GPU 版包：BtbN cuda ffmpeg，目标机需 glibc>=2.28
|
||||
# bash build_bundle.sh gpu --whisper-gpu # GPU 版 + 把 CUDA 库打进包(目标机无系统 CUDA 也能 whisper GPU)
|
||||
#
|
||||
# 产物: /tmp/ocr_offline_bundle.tar.gz
|
||||
# 拷到目标机后: bash deploy.sh /tmp/ocr_offline_bundle.tar.gz /opt
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
VARIANT="${1:-cpu}" # cpu | gpu
|
||||
WHISPER_GPU_FLAG=0
|
||||
[ "${2:-}" = "--whisper-gpu" ] && WHISPER_GPU_FLAG=1
|
||||
|
||||
log() { echo -e "\n\033[1;36m##### $* #####\033[0m"; }
|
||||
|
||||
case "$VARIANT" in
|
||||
cpu) FFV=cpu ;;
|
||||
gpu) FFV=gpu ;;
|
||||
*) echo "用法: bash build_bundle.sh [cpu|gpu] [--whisper-gpu]"; exit 1 ;;
|
||||
esac
|
||||
|
||||
SD="$(cd "$(dirname "$0")" && pwd)"
|
||||
log "构建 $VARIANT 版离线包 (whisper_gpu=$WHISPER_GPU_FLAG)"
|
||||
|
||||
# 1) 装基础环境(已装过会跳过大部分)
|
||||
log "1/3 setup_server.sh（装 gcc/uv/python/依赖）"
|
||||
bash "$SD/setup_server.sh"
|
||||
|
||||
# 2) 转自包含（python+模型+ffmpeg 进项目）
|
||||
log "2/3 make_portable.sh（FFMPEG_VARIANT=$FFV）"
|
||||
FFMPEG_VARIANT="$FFV" WHISPER_GPU="$WHISPER_GPU_FLAG" bash "$SD/make_portable.sh"
|
||||
|
||||
# 3) 打包（按变体命名，cpu/gpu 两个包不会互相覆盖）
|
||||
OUT="/tmp/ocr_bundle_${VARIANT}.tar.gz"
|
||||
log "3/3 pack.sh（打成 tar → $OUT）"
|
||||
bash "$SD/pack.sh" "$OUT"
|
||||
|
||||
echo
|
||||
echo "✅ 完成。产物: $OUT"
|
||||
echo " 拷到目标机后: bash $SD/deploy.sh $OUT /opt"
|
||||
75
scripts/deploy.sh
Normal file
75
scripts/deploy.sh
Normal file
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 离线机部署脚本（在【不能联网】的机器上跑）
|
||||
#
|
||||
# 自动识别打包模式：
|
||||
# A) 自包含包（含项目内 python/）：解压到【任意目标路径】，默认当前目录。
|
||||
# bash deploy.sh 包.tar.gz [目标父目录] # 例: bash deploy.sh bundle.tar.gz /opt
|
||||
# B) venv 包：解压到 / 还原到相同绝对路径（要求同路径/同用户/同架构）。
|
||||
# bash deploy.sh 包.tar.gz
|
||||
#
|
||||
# 前提：与打包机【同 CPU 架构(x86_64)】且【glibc 不比打包机老】。
|
||||
# 离线机不需要 uv / python / gcc / 联网。
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
BUNDLE="${1:-/tmp/ocr_offline_bundle.tar.gz}"
|
||||
TARGET="${2:-$(pwd)}" # 自包含模式的解压目标父目录
|
||||
log() { echo -e "\n\033[1;32m==> $*\033[0m"; }
|
||||
warn() { echo -e "\033[1;33m[警告] $*\033[0m"; }
|
||||
|
||||
[ -f "$BUNDLE" ] || { echo "找不到包: $BUNDLE"; exit 1; }
|
||||
|
||||
log "[1/3] 兼容性 + 模式识别"
|
||||
echo " 架构: $(uname -m)（需与打包机一致，一般 x86_64）"
|
||||
echo " glibc: $(ldd --version 2>/dev/null | head -1 | awk '{print $NF}')（不能比打包机老）"
|
||||
|
||||
# 看包里第一个条目：以 / 或 data/ 之类绝对路径开头 = venv 模式；以纯目录名开头 = 自包含
|
||||
# 注意：head 关管道会让 tar 收到 SIGPIPE 返回非零，配合 set -e/pipefail 会误退出，故加 || true
|
||||
FIRST="$(tar tzf "$BUNDLE" 2>/dev/null | head -1 || true)"
|
||||
echo " 包内首条目: $FIRST"
|
||||
|
||||
PROJDIR=""
|
||||
if printf '%s' "$FIRST" | grep -qE '^(/|root/|home/|data/|usr/|opt/)'; then
|
||||
# ---------- venv 模式：解压到 / ----------
|
||||
log "[2/3] venv 模式 → 解压到 / (还原到原绝对路径)"
|
||||
tar xzf "$BUNDLE" -C /
|
||||
# 项目目录 = 第一条目里到 ocr_data 那一段的绝对路径(取含项目名的目录)
|
||||
PROJDIR="/$(printf '%s' "$FIRST" | sed -E 's#(.*/[^/]*ocr[_-]?data)/.*#\1#')"
|
||||
warn "venv 模式：必须解压到相同路径/同用户，否则 venv 失效"
|
||||
else
|
||||
# ---------- 自包含模式：解压到目标目录 ----------
|
||||
log "[2/3] 自包含模式 → 解压到 $TARGET"
|
||||
mkdir -p "$TARGET"
|
||||
tar xzf "$BUNDLE" -C "$TARGET"
|
||||
TOP="$(printf '%s' "$FIRST" | cut -d/ -f1)"
|
||||
PROJDIR="$TARGET/$TOP"
|
||||
fi
|
||||
echo " 项目目录: $PROJDIR"
|
||||
|
||||
log "[3/3] 离线自检"
|
||||
cd "$PROJDIR" || { echo "进项目目录失败: $PROJDIR"; exit 1; }
|
||||
# shellcheck disable=SC1091
|
||||
source activate.sh
|
||||
|
||||
ok=1
|
||||
ck() { if eval "$2" >/dev/null 2>&1; then echo " ✓ $1"; else echo " ✗ $1"; ok=0; fi; }
|
||||
ck "python" "python --version"
|
||||
ck "import cv2(headless)" "python -c 'import cv2'"
|
||||
ck "import numpy" "python -c 'import numpy'"
|
||||
ck "import faster_whisper" "python -c 'import faster_whisper'"
|
||||
ck "import qcloud_cos" "python -c 'import qcloud_cos'"
|
||||
ck "ffmpeg(bin/)" "ffmpeg -version"
|
||||
ck "ffmpeg 支持 AV1" "ffmpeg -hide_banner -decoders | grep -iE 'dav1d|av1'"
|
||||
ck "coscli" "./coscli-linux-amd64 --help"
|
||||
MODEL="$(python -c "import json;print(json.load(open('config/config.json'))['audio'].get('model_size','small'))" 2>/dev/null || echo small)"
|
||||
ck "whisper 模型离线加载" "python -c \"from faster_whisper import WhisperModel; WhisperModel('$MODEL',device='cpu',compute_type='int8')\""
|
||||
|
||||
echo
|
||||
if [ "$ok" -eq 1 ]; then
|
||||
log "全部就绪 ✅ 可离线运行"
|
||||
echo " cd $PROJDIR && source activate.sh"
|
||||
echo " python main.py --source-list ./text2 --max-workers 12"
|
||||
else
|
||||
warn "有项目未通过(见上面 ✗)。常见原因：架构/glibc 不匹配；venv 模式下路径/用户不一致"
|
||||
fi
|
||||
152
scripts/diag.sh
Normal file
152
scripts/diag.sh
Normal file
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env bash
|
||||
# 一键诊断: 收集机器/进程/内存/OOM/日志/数据 全部信息到一个文件, 发回给我分析。
|
||||
# 用法(在项目根目录): bash scripts/diag.sh
|
||||
# 产物: diag_YYYYmmdd_HHMMSS.txt
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" || exit 1
|
||||
OUT="diag_$(date +%Y%m%d_%H%M%S).txt"
|
||||
PY="$(command -v python || command -v python3)"
|
||||
|
||||
# 所有输出同时进文件
|
||||
exec > >(tee "$OUT") 2>&1
|
||||
|
||||
echo "================ 诊断报告 $(date) ================"
|
||||
echo "主机: $(hostname) 目录: $(pwd)"
|
||||
echo ""
|
||||
|
||||
echo "========== 1. 机器资源 =========="
|
||||
echo "[内存 free -g]"; free -g
|
||||
echo ""
|
||||
echo "[CPU核数] $(nproc)"
|
||||
echo "[负载 uptime] $(uptime)"
|
||||
echo ""
|
||||
|
||||
echo "========== 2. OOM 检查(夜里有没有被系统杀) =========="
|
||||
echo "[dmesg killed process]"
|
||||
( dmesg -T 2>/dev/null || dmesg 2>/dev/null ) | grep -iE "killed process|out of memory|oom-kill" | tail -20 || echo " (无 / dmesg无权限)"
|
||||
echo "[/var/log/messages oom]"
|
||||
grep -iE "killed process|out of memory|oom" /var/log/messages 2>/dev/null | tail -10 || echo " (无 / 无权限)"
|
||||
echo ""
|
||||
|
||||
echo "========== 3. 进程状态 =========="
|
||||
echo "[main.py 进程数] $(ps -eo args 2>/dev/null | grep -c '[m]ain.py')"
|
||||
echo "[ffmpeg 进程数] $(ps -eo comm 2>/dev/null | grep -c '[f]fmpeg')"
|
||||
echo "[forkserver/worker] $(ps -eo args 2>/dev/null | grep -c '[m]ultiprocessing\|[f]orkserver')"
|
||||
echo ""
|
||||
echo "[吃内存最多的10个进程 RSS]"
|
||||
ps -eo pid,rss,etime,args --sort=-rss 2>/dev/null | head -11
|
||||
echo ""
|
||||
|
||||
echo "========== 4. Python / 环境 =========="
|
||||
echo "[python] $($PY --version 2>&1) 路径: $PY"
|
||||
echo "[which python] $(command -v python 2>/dev/null)"
|
||||
echo "[自包含python在不在] $(ls python/bin/python3 2>/dev/null && echo yes || echo no)"
|
||||
echo ""
|
||||
|
||||
echo "========== 5. config 关键值 =========="
|
||||
if [ -f config/config.json ]; then
|
||||
$PY - <<'PYEOF'
|
||||
import json
|
||||
try:
|
||||
d=json.load(open("config/config.json",encoding="utf-8"))
|
||||
s=d.get("subtitle",{}); sf=d.get("splitframe",{}); au=d.get("audio",{})
|
||||
print(" ocr_workers=%s no_composite=%s merge_workers=%s max_tiles=%s"%(
|
||||
s.get("ocr_workers"),s.get("no_composite"),s.get("merge_workers"),s.get("max_tiles_per_composite")))
|
||||
print(" cv_detect=%s filter_static=%s filter_watermark_pos=%s split_far_boxes=%s"%(
|
||||
s.get("cv_detect"),s.get("filter_static"),s.get("filter_watermark_pos"),s.get("split_far_boxes")))
|
||||
print(" min_duration=%s crop_bottom_ratio=%s ffmpeg_threads=%s use_gpu=%s"%(
|
||||
s.get("min_duration_seconds"),sf.get("crop_bottom_ratio"),sf.get("ffmpeg_threads"),sf.get("use_gpu")))
|
||||
print(" audio.model_size=%s cpu_threads=%s endpoints数=%s"%(
|
||||
au.get("model_size"),au.get("cpu_threads"),len(d.get("tencentcloud",{}).get("ocr",{}).get("endpoints",[]))))
|
||||
except Exception as e:
|
||||
print(" 读config失败:",e)
|
||||
PYEOF
|
||||
else
|
||||
echo " config/config.json 不存在"
|
||||
fi
|
||||
echo "[启动命令] $(ps -eo args 2>/dev/null | grep '[m]ain.py' | head -1)"
|
||||
echo ""
|
||||
|
||||
echo "========== 6. app.log 分析 =========="
|
||||
if [ -f app.log ]; then
|
||||
echo "[总行数] $(wc -l < app.log)"
|
||||
echo "[关键词计数]"
|
||||
for kw in 成功 无字幕 失败 暂停启动 抽帧完成 拼接完成 字幕完成 内存不足 BrokenProcess 崩溃 worker崩 Traceback Error 超时 timeout 冷却 端点; do
|
||||
c=$(grep -c "$kw" app.log 2>/dev/null)
|
||||
[ "${c:-0}" -gt 0 ] && printf " %-14s %s\n" "$kw" "$c"
|
||||
done
|
||||
echo ""
|
||||
echo "[最近 40 行]"
|
||||
tail -40 app.log
|
||||
else
|
||||
echo " app.log 不存在"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========== 7. 数据.jsonl 分析 =========="
|
||||
if [ -f 数据.jsonl ]; then
|
||||
$PY - <<'PYEOF'
|
||||
import json,statistics
|
||||
from collections import Counter,defaultdict
|
||||
from datetime import datetime,timedelta
|
||||
latest={}
|
||||
for line in open("数据.jsonl",encoding="utf-8"):
|
||||
line=line.strip()
|
||||
if not line: continue
|
||||
try: r=json.loads(line)
|
||||
except: continue
|
||||
v=r.get("video_cos_path")
|
||||
if v: latest[v]=r
|
||||
n=len(latest)
|
||||
st=Counter(r.get("status") for r in latest.values())
|
||||
print(" 去重视频 %d | 状态 %s"%(n,dict(st)))
|
||||
if n: print(" 成功率 %.1f%%"%(st.get("success",0)/n*100))
|
||||
# 阶段耗时(成功)
|
||||
tot=defaultdict(float); te=0
|
||||
for r in latest.values():
|
||||
if r.get("status")!="success": continue
|
||||
sd=r.get("stage_durations") or {}; te+=r.get("elapsed_seconds") or 0
|
||||
for k in ["split_frame","merge","getsrt"]:
|
||||
if k in sd: tot[k]+=sd[k]
|
||||
if te:
|
||||
print(" 阶段占比: 抽帧%.0f%% 拼接%.0f%% OCR+字幕%.0f%%"%(tot["split_frame"]/te*100,tot["merge"]/te*100,tot["getsrt"]/te*100))
|
||||
# OCR/帧
|
||||
rs=[ic.get("total_ocr_count")/ic.get("frame_count") for r in latest.values() if (ic:=r.get("image_counts") or {}).get("frame_count") and ic.get("total_ocr_count")]
|
||||
if rs: print(" OCR/帧 中位 %.3f"%statistics.median(rs))
|
||||
# 时长
|
||||
durs=sorted((r.get("duration") or 0) for r in latest.values() if r.get("duration"))
|
||||
if durs: print(" 视频时长 中位%.0fs max%.0fs | >20分钟 %d个"%(statistics.median(durs),max(durs),sum(1 for d in durs if d>1200)))
|
||||
# 单视频平均耗时
|
||||
els=[r.get("elapsed_seconds") or 0 for r in latest.values() if r.get("status")=="success"]
|
||||
if els: print(" 成功视频平均耗时 %.0fs 中位%.0fs"%(sum(els)/len(els),statistics.median(els)))
|
||||
# 每小时吞吐 + 有效并发(重叠法)
|
||||
busy=defaultdict(float); cnt=defaultdict(int)
|
||||
for r in latest.values():
|
||||
try: s=datetime.fromisoformat(r["process_start"]); e=datetime.fromisoformat(r["process_end"])
|
||||
except: continue
|
||||
if e<=s: continue
|
||||
cnt[s.strftime("%m-%d %H")]+=1
|
||||
cur=s
|
||||
while cur<e:
|
||||
hk=cur.strftime("%m-%d %H"); nx=cur.replace(minute=0,second=0,microsecond=0)+timedelta(hours=1)
|
||||
seg=min(e,nx); busy[hk]+=(seg-cur).total_seconds(); cur=seg
|
||||
print(" 每小时(完成数 / 有效并发):")
|
||||
for hk in sorted(busy)[-8:]:
|
||||
print(" %s : %d条 / 并发%.1f"%(hk,cnt[hk],busy[hk]/3600))
|
||||
PYEOF
|
||||
else
|
||||
echo " 数据.jsonl 不存在"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
echo "========== 8. 磁盘 / GPU =========="
|
||||
echo "[磁盘 df -h .]"; df -h . 2>/dev/null
|
||||
echo "[split/ 大小] $(du -sh split 2>/dev/null || echo '无')"
|
||||
echo "[正在处理的视频数 split/] $(ls split 2>/dev/null | wc -l)"
|
||||
if command -v nvidia-smi >/dev/null 2>&1; then
|
||||
echo "[GPU]"; nvidia-smi --query-gpu=index,memory.used,memory.total,utilization.gpu --format=csv,noheader 2>/dev/null
|
||||
fi
|
||||
echo ""
|
||||
echo "================ 诊断结束 ================"
|
||||
echo ""
|
||||
echo ">>> 报告已保存到: $OUT (把这个文件发回来)"
|
||||
127
scripts/make_portable.sh
Normal file
127
scripts/make_portable.sh
Normal file
@@ -0,0 +1,127 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 把环境改造成"项目内自包含"——Python、第三方库、模型全放进项目目录，
|
||||
# 之后整个项目文件夹可以【随便放到任意路径、任意用户】，不再依赖 /root 或固定路径。
|
||||
#
|
||||
# 在【能联网】的机器上跑一次（已 setup_server.sh 装好 .venv 后）：
|
||||
# bash make_portable.sh # 默认 ffmpeg 用 GPU 版(BtbN，含 cuda/nvdec，也兼容 CPU)
|
||||
# # 需运行机 glibc>=2.28；GPU 机器自动硬解、CPU 机器软解
|
||||
# FFMPEG_VARIANT=cpu bash make_portable.sh # ffmpeg 用 CPU 版(johnvansickle，全静态，老 glibc 也能跑)
|
||||
# 之后:
|
||||
# bash pack.sh # 打包(此时只需打项目目录，已自包含)
|
||||
# # 离线机解压到【任意路径】即可: source activate.sh && python main.py ...
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
PROJ="$(pwd)"
|
||||
PIP_MIRROR="${PIP_MIRROR:-https://repo.huaweicloud.com/repository/pypi/simple}"
|
||||
|
||||
log() { echo -e "\n\033[1;32m==> $*\033[0m"; }
|
||||
warn() { echo -e "\033[1;33m[警告] $*\033[0m"; }
|
||||
have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
[ -d .venv ] || { echo "没有 .venv，请先 bash setup_server.sh"; exit 1; }
|
||||
# shellcheck disable=SC1091
|
||||
source .venv/bin/activate
|
||||
have uv || { echo "需要 uv（本机改造用，离线机不需要）"; exit 1; }
|
||||
|
||||
# 0) 确保 bin/ffmpeg 可用。两种变体（用环境变量 FFMPEG_VARIANT 选）：
|
||||
# cpu (默认) = johnvansickle 静态版，含 dav1d(CPU 软解 AV1)，兼容老 glibc，哪都能跑。
|
||||
# gpu = BtbN 版，含 cuda/nvdec(GPU 硬解)，给 Ampere+ 卡的机器用；需较新 glibc(>=2.28)。
|
||||
# 用法: FFMPEG_VARIANT=gpu bash make_portable.sh
|
||||
# 默认下 GPU 版(含 cuda/nvdec，也兼容 CPU 软解)。要纯 CPU 版用 FFMPEG_VARIANT=cpu。
|
||||
FFMPEG_VARIANT="${FFMPEG_VARIANT:-gpu}"
|
||||
FF_NEED=1
|
||||
if [ "$FFMPEG_VARIANT" = "gpu" ]; then
|
||||
FF_URL="${FFMPEG_GPU_URL:-https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz}"
|
||||
# GPU 版(BtbN)要 glibc>=2.28 才能运行；本机若太老仅警告(可能在给别的新机器打包)
|
||||
# 注意：管道里 head 会让前面命令 SIGPIPE，配合 set -e/pipefail 会误退出，故加 || true
|
||||
_glibc="$(ldd --version 2>/dev/null | grep -oE '[0-9]+\.[0-9]+' | head -1 || true)"
|
||||
if [ -n "$_glibc" ] && [ "$(printf '%s\n2.28\n' "$_glibc" | sort -V | head -1)" != "2.28" ]; then
|
||||
warn "本机 glibc=$_glibc < 2.28，GPU 版 ffmpeg 在本机跑不了(GLIBC_2.28 not found)。"
|
||||
warn "仅当打包给 glibc>=2.28 的机器时才用 GPU 版；本机自用请: FFMPEG_VARIANT=cpu bash make_portable.sh"
|
||||
fi
|
||||
# 用 grep(不带 -q)读完整输出再判，避免 grep -q 提前关管道触发 SIGPIPE+pipefail 误判
|
||||
if [ -x "$PROJ/bin/ffmpeg" ] && "$PROJ/bin/ffmpeg" -hide_banner -hwaccels 2>/dev/null | grep -i cuda >/dev/null; then
|
||||
log "[0/4] bin/ffmpeg 已是 GPU 版(支持 cuda)，跳过"; FF_NEED=0
|
||||
else
|
||||
log "[0/4] 下载 GPU 版 ffmpeg (BtbN，含 cuda/nvdec) → ./bin"
|
||||
fi
|
||||
else
|
||||
FF_URL="${FFMPEG_STATIC_URL:-https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz}"
|
||||
if [ -x "$PROJ/bin/ffmpeg" ] && "$PROJ/bin/ffmpeg" -hide_banner -decoders 2>/dev/null | grep -iE 'dav1d|av1' >/dev/null; then
|
||||
log "[0/4] bin/ffmpeg 已支持 AV1(CPU 版)，跳过"; FF_NEED=0
|
||||
else
|
||||
log "[0/4] 下载 CPU 版 ffmpeg (johnvansickle，含 dav1d) → ./bin"
|
||||
fi
|
||||
fi
|
||||
if [ "$FF_NEED" -eq 1 ]; then
|
||||
mkdir -p "$PROJ/bin"; tmp="$(mktemp -d)"
|
||||
if have curl; then curl -L "$FF_URL" -o "$tmp/ff.tar.xz"; else wget -O "$tmp/ff.tar.xz" "$FF_URL"; fi
|
||||
tar xf "$tmp/ff.tar.xz" -C "$tmp"
|
||||
# johnvansickle 在 <dir>/ffmpeg，BtbN 在 <dir>/bin/ffmpeg —— 直接全树找，两种都兼容
|
||||
ffb="$(find "$tmp" -type f -name ffmpeg | head -1)"
|
||||
ffp="$(find "$tmp" -type f -name ffprobe | head -1)"
|
||||
if [ -n "$ffb" ] && [ -n "$ffp" ]; then
|
||||
cp "$ffb" "$ffp" "$PROJ/bin/" && chmod +x "$PROJ/bin/ffmpeg" "$PROJ/bin/ffprobe"
|
||||
log "ffmpeg/ffprobe 已装到 ./bin（$FFMPEG_VARIANT 版）"
|
||||
else
|
||||
echo "解压后没找到 ffmpeg/ffprobe，请手动放入 ./bin"; exit 1
|
||||
fi
|
||||
rm -rf "$tmp"
|
||||
fi
|
||||
|
||||
# 1) 把 uv 的独立 Python 复制进项目 ./python（python-build-standalone 本身可重定位）
|
||||
log "[1/4] 复制独立 Python 到项目内 ./python"
|
||||
PYSRC="$(dirname "$(dirname "$(readlink -f .venv/bin/python)")")"
|
||||
echo " 源: $PYSRC"
|
||||
rm -rf "$PROJ/python"
|
||||
cp -a "$PYSRC" "$PROJ/python"
|
||||
PYBIN="$PROJ/python/bin/python3"
|
||||
[ -x "$PYBIN" ] || PYBIN="$(ls "$PROJ"/python/bin/python3.* | head -1)"
|
||||
echo " 项目内 Python: $PYBIN"
|
||||
# 去掉 uv 的"外部管理(EXTERNALLY-MANAGED)"标记，否则 uv 拒绝往这个 Python 里装包
|
||||
rm -f "$PROJ"/python/lib/python3.*/EXTERNALLY-MANAGED
|
||||
|
||||
# 2) 把依赖装进这个项目内 Python（脱离 .venv，用 headless opencv）
|
||||
log "[2/4] 安装依赖到项目内 Python"
|
||||
uv pip uninstall --python "$PYBIN" --break-system-packages opencv-python 2>/dev/null || true
|
||||
uv pip install --python "$PYBIN" --break-system-packages -r requirements.txt --index-url "$PIP_MIRROR" \
|
||||
--only-binary numpy,scipy,scikit-learn,opencv-python-headless,Pillow,av \
|
||||
|| uv pip install --python "$PYBIN" --break-system-packages -r requirements.txt --index-url "$PIP_MIRROR"
|
||||
|
||||
# WHISPER_GPU=1：把 CUDA 库(cublas/cudnn)也装进项目 Python，这样目标机【无需系统 CUDA】
|
||||
# 也能让 whisper 跑 GPU(ctranslate2 从 venv 里找库；activate.sh 会把它加进 LD_LIBRARY_PATH)。
|
||||
if [ "${WHISPER_GPU:-0}" = "1" ]; then
|
||||
log " 额外装 CUDA 库(nvidia-cublas/cudnn-cu12)给 whisper GPU 用"
|
||||
uv pip install --python "$PYBIN" --break-system-packages \
|
||||
nvidia-cublas-cu12 nvidia-cudnn-cu12 --index-url "$PIP_MIRROR" \
|
||||
|| warn "CUDA 库安装失败；目标机若无系统 CUDA，whisper 将回退 CPU"
|
||||
fi
|
||||
|
||||
# 3) 把 whisper 模型缓存搬进项目 ./models（用 HF_HOME 指过去）
|
||||
log "[3/4] 预下载 whisper 模型到项目内 ./models"
|
||||
MODEL="$(python -c "import json;print(json.load(open('config/config.json'))['audio'].get('model_size','small'))")"
|
||||
mkdir -p "$PROJ/models"
|
||||
HF_HOME="$PROJ/models" HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 \
|
||||
HF_ENDPOINT="${HF_ENDPOINT:-https://hf-mirror.com}" \
|
||||
HF_HUB_DISABLE_XET=1 \
|
||||
"$PYBIN" -c "from faster_whisper import WhisperModel; WhisperModel('$MODEL',device='cpu',compute_type='int8'); print('模型已放进 ./models')"
|
||||
|
||||
# 4) 删掉旧 .venv（不再需要，已被项目内 Python 取代）
|
||||
log "[4/4] 清理旧 .venv"
|
||||
deactivate 2>/dev/null || true
|
||||
rm -rf "$PROJ/.venv"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
完成 ✅ 现在项目已自包含，可随便放任意路径/用户：
|
||||
./python/ 独立 Python + 所有第三方库
|
||||
./bin/ ffmpeg / ffprobe
|
||||
./models/ whisper 模型
|
||||
coscli-linux-amd64
|
||||
|
||||
验证: source activate.sh && python -c "import cv2,faster_whisper;print('OK')"
|
||||
打包: bash pack.sh
|
||||
EOF
|
||||
33
scripts/monitor.sh
Normal file
33
scripts/monitor.sh
Normal file
@@ -0,0 +1,33 @@
|
||||
#!/usr/bin/env bash
|
||||
# 每隔 N 秒采集 CPU/IO/负载/网络,带时间戳写入一个文件。
|
||||
# 用法: bash monitor.sh [间隔秒,默认5] [输出文件,默认 monitor.log]
|
||||
# 停止: Ctrl-C 或 kill $(cat monitor.pid)
|
||||
INTERVAL="${1:-5}"
|
||||
OUT="${2:-monitor.log}"
|
||||
echo $$ > monitor.pid
|
||||
|
||||
echo "===== 监控开始 $(date) 间隔${INTERVAL}s 输出 $OUT =====" | tee -a "$OUT"
|
||||
command -v iostat >/dev/null 2>&1 && HAS_IOSTAT=1 || HAS_IOSTAT=0
|
||||
command -v sar >/dev/null 2>&1 && HAS_SAR=1 || HAS_SAR=0
|
||||
[ "$HAS_IOSTAT" = 0 ] && echo "(无 iostat, 只记 vmstat; 装: yum/apt install sysstat)" | tee -a "$OUT"
|
||||
|
||||
trap 'echo "===== 监控停止 $(date) =====" >> "$OUT"; rm -f monitor.pid; exit 0' INT TERM
|
||||
|
||||
while true; do
|
||||
{
|
||||
echo "----- $(date '+%Y-%m-%d %H:%M:%S') -----"
|
||||
echo "[vmstat] r=运行队列 b=阻塞(等IO) wa=iowait% id=空闲%"
|
||||
vmstat 1 2 | tail -1
|
||||
if [ "$HAS_IOSTAT" = 1 ]; then
|
||||
echo "[iostat] %util>90=磁盘吃满 w/s=写次数 w_await=写延迟ms"
|
||||
# 取第二份报告(间隔值, 非开机累计), 只留设备行
|
||||
iostat -dx 1 2 | awk '/Device/{c++} c==2'
|
||||
fi
|
||||
if [ "$HAS_SAR" = 1 ]; then
|
||||
echo "[net] 网卡收发 kB/s:"
|
||||
sar -n DEV 1 1 | awk '/Average:/ && $2!="lo" && $2!="IFACE" {printf " %s rx=%s tx=%s kB/s\n",$2,$5,$6}'
|
||||
fi
|
||||
echo ""
|
||||
} >> "$OUT" 2>&1
|
||||
sleep "$INTERVAL"
|
||||
done
|
||||
84
scripts/pack.sh
Normal file
84
scripts/pack.sh
Normal file
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 离线部署打包脚本（在【能联网】的机器上跑）
|
||||
#
|
||||
# 两种模式自动识别：
|
||||
# A) 自包含模式（已跑过 make_portable.sh，项目内有 ./python + ./models）
|
||||
# → 打包里只含项目这一个文件夹，离线机解压到【任意路径/任意用户】即可。
|
||||
# B) venv 模式（只有 .venv）
|
||||
# → 打项目 + /root 下的 uv Python + HF 缓存，离线机要解压到相同路径。
|
||||
#
|
||||
# 用法: bash pack.sh
|
||||
# 产物: /tmp/ocr_offline_bundle.tar.gz
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
PROJECT_DIR="$(pwd)"
|
||||
OUT="${1:-/tmp/ocr_offline_bundle.tar.gz}"
|
||||
|
||||
log() { echo -e "\n\033[1;32m==> $*\033[0m"; }
|
||||
warn() { echo -e "\033[1;33m[警告] $*\033[0m"; }
|
||||
have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
PBASE="$(basename "$PROJECT_DIR")"
|
||||
PPARENT="$(dirname "$PROJECT_DIR")"
|
||||
|
||||
# 运行时数据排除（部署不需要，打进去又慢又大）。用项目目录名作前缀，两种模式通用。
|
||||
RT_DIRS=(split video composite coscli_output srt info info_error log)
|
||||
EXCLUDES=(--exclude='*.log' --exclude='__pycache__' --exclude='*.pyc'
|
||||
--exclude="$PBASE/数据.jsonl")
|
||||
for d in "${RT_DIRS[@]}"; do EXCLUDES+=(--exclude="$PBASE/$d"); done
|
||||
|
||||
# 选最快的压缩器：pigz 多线程(输出仍是标准 gzip，离线机普通 tar xzf 就能解)
|
||||
if have pigz; then
|
||||
COMPRESS="pigz"; log "用 pigz 多线程压缩(快)"
|
||||
else
|
||||
COMPRESS="gzip -1"; warn "未装 pigz，用 gzip -1；想更快可先 yum install -y pigz"
|
||||
fi
|
||||
|
||||
if [ -x "$PROJECT_DIR/python/bin/python3" ]; then
|
||||
# ---------- A) 自包含：只打项目目录(用相对名)，可解压到任意路径 ----------
|
||||
log "自包含模式（项目内含 ./python + ./models）"
|
||||
[ -d "$PROJECT_DIR/models" ] || warn "没看到 ./models，确认 make_portable.sh 跑完整了"
|
||||
tar "${EXCLUDES[@]}" --use-compress-program="$COMPRESS" -cf "$OUT" \
|
||||
-C "$PPARENT" "$PBASE"
|
||||
MODE="自包含（解压到任意路径/用户）"
|
||||
else
|
||||
# ---------- B) venv：打项目(绝对路径) + 外部 Python + HF 缓存 ----------
|
||||
[ -d .venv ] || { echo "既没有 ./python 也没有 .venv，请先 setup_server.sh 或 make_portable.sh"; exit 1; }
|
||||
# shellcheck disable=SC1091
|
||||
source .venv/bin/activate
|
||||
|
||||
log "venv 模式：确保 headless opencv + 预下载模型"
|
||||
uv pip uninstall opencv-python 2>/dev/null || true
|
||||
uv pip install opencv-python-headless \
|
||||
--index-url "${PIP_MIRROR:-https://repo.huaweicloud.com/repository/pypi/simple}" 2>/dev/null \
|
||||
|| echo "（headless 可能已装，忽略）"
|
||||
MODEL="$(python -c "import json;print(json.load(open('config/config.json'))['audio'].get('model_size','small'))")"
|
||||
HF_HUB_OFFLINE=0 TRANSFORMERS_OFFLINE=0 HF_ENDPOINT="${HF_ENDPOINT:-https://hf-mirror.com}" \
|
||||
python -c "from faster_whisper import WhisperModel; WhisperModel('$MODEL',device='cpu',compute_type='int8'); print('模型已缓存')"
|
||||
|
||||
PYDIR="$(dirname "$(dirname "$(readlink -f .venv/bin/python)")")"
|
||||
HFCACHE="${HF_HOME:-$HOME/.cache/huggingface}"
|
||||
echo " uv Python: $PYDIR 模型缓存: $HFCACHE"
|
||||
|
||||
log "打包项目(绝对路径) + Python + 模型缓存"
|
||||
# 项目用绝对路径(还原到原位)，外部两目录同理
|
||||
EXABS=(--exclude='*.log' --exclude='__pycache__' --exclude='*.pyc'
|
||||
--exclude="${PROJECT_DIR#/}/数据.jsonl")
|
||||
for d in "${RT_DIRS[@]}"; do EXABS+=(--exclude="${PROJECT_DIR#/}/$d"); done
|
||||
tar "${EXABS[@]}" --use-compress-program="$COMPRESS" -cf "$OUT" \
|
||||
-C / "${PROJECT_DIR#/}" -C / "${PYDIR#/}" -C / "${HFCACHE#/}"
|
||||
MODE="venv（离线机需解压到相同路径/用户）"
|
||||
fi
|
||||
|
||||
echo
|
||||
log "完成 ✅ $OUT ($(du -h "$OUT" | cut -f1))"
|
||||
echo "模式: $MODE"
|
||||
cat <<EOF
|
||||
|
||||
下一步：
|
||||
scp $OUT root@离线机:/tmp/
|
||||
bash deploy.sh /tmp/$(basename "$OUT") # 自包含模式可加目标路径: bash deploy.sh 包 /opt
|
||||
EOF
|
||||
26
scripts/set_ffmpeg.sh
Normal file
26
scripts/set_ffmpeg.sh
Normal file
@@ -0,0 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# 一键切换项目 bin/ffmpeg 的版本(本机自用 cpu / 打 GPU 包 gpu)
|
||||
# bash temp/set_ffmpeg.sh cpu # johnvansickle 全静态，任何 glibc 都能跑(本机自用)
|
||||
# bash temp/set_ffmpeg.sh gpu # BtbN 含 cuda/nvdec，需 glibc>=2.28(给 GPU 机器)
|
||||
set -euo pipefail
|
||||
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # 项目根
|
||||
V="${1:-cpu}"
|
||||
BIN="$(pwd)/bin"; mkdir -p "$BIN"
|
||||
have(){ command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
case "$V" in
|
||||
cpu) URL="https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz" ;;
|
||||
gpu) URL="https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-linux64-gpl.tar.xz" ;;
|
||||
*) echo "用法: bash temp/set_ffmpeg.sh [cpu|gpu]"; exit 1 ;;
|
||||
esac
|
||||
|
||||
echo "下载 $V 版 ffmpeg ..."
|
||||
tmp="$(mktemp -d)"
|
||||
if have curl; then curl -L "$URL" -o "$tmp/ff.tar.xz"; else wget -O "$tmp/ff.tar.xz" "$URL"; fi
|
||||
tar xf "$tmp/ff.tar.xz" -C "$tmp"
|
||||
ffb="$(find "$tmp" -type f -name ffmpeg | head -1)"
|
||||
ffp="$(find "$tmp" -type f -name ffprobe | head -1)"
|
||||
cp "$ffb" "$ffp" "$BIN/"
|
||||
chmod +x "$BIN/ffmpeg" "$BIN/ffprobe"
|
||||
rm -rf "$tmp"
|
||||
echo "已切到 $V 版: $("$BIN/ffmpeg" -version 2>&1 | head -1 || echo '本机跑不了(若是 gpu 版+老 glibc，正常)')"
|
||||
244
scripts/setup_server.sh
Normal file
244
scripts/setup_server.sh
Normal file
@@ -0,0 +1,244 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 视频字幕提取系统 — 服务器一键环境配置脚本
|
||||
#
|
||||
# 安装/配置：gcc 编译工具链、ffmpeg(含 AV1/dav1d 解码)、uv、Python 虚拟环境、
|
||||
# 项目依赖，并处理常见环境坑(opencv libGL、UTF-8 locale、HF/pip 镜像、
|
||||
# OMP 线程、文件句柄上限)。
|
||||
#
|
||||
# 用法(在项目根目录):
|
||||
# bash setup_server.sh
|
||||
# 跑完后:
|
||||
# source activate.sh # 激活环境(设置 venv + 各种环境变量)
|
||||
# python main.py --source-list ./text2 --max-workers 12
|
||||
#
|
||||
# 幂等：可重复运行，已装的会跳过。
|
||||
# =============================================================================
|
||||
set -euo pipefail
|
||||
|
||||
# ----------------------------- 可调参数 --------------------------------------
|
||||
PYTHON_VERSION="${PYTHON_VERSION:-3.11}"
|
||||
PIP_MIRROR="${PIP_MIRROR:-https://repo.huaweicloud.com/repository/pypi/simple}"
|
||||
HF_ENDPOINT_URL="${HF_ENDPOINT_URL:-https://hf-mirror.com}"
|
||||
FFMPEG_STATIC_URL="${FFMPEG_STATIC_URL:-https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz}"
|
||||
PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
VENV_DIR="$PROJECT_DIR/.venv"
|
||||
|
||||
cd "$PROJECT_DIR"
|
||||
|
||||
# 让本脚本(及自检)优先用项目 bin/ 里的 ffmpeg(我们装的静态版)
|
||||
export PATH="$PROJECT_DIR/bin:$PATH"
|
||||
|
||||
# ----------------------------- 工具函数 --------------------------------------
|
||||
log() { echo -e "\n\033[1;32m==> $*\033[0m"; }
|
||||
warn() { echo -e "\033[1;33m[警告] $*\033[0m"; }
|
||||
err() { echo -e "\033[1;31m[错误] $*\033[0m" >&2; }
|
||||
have() { command -v "$1" >/dev/null 2>&1; }
|
||||
|
||||
# root 则不用 sudo，否则用 sudo
|
||||
if [ "$(id -u)" -eq 0 ]; then SUDO=""; else SUDO="sudo"; fi
|
||||
|
||||
# 检测包管理器
|
||||
PKG=""
|
||||
if have dnf; then PKG="dnf"
|
||||
elif have yum; then PKG="yum"
|
||||
elif have apt-get; then PKG="apt"
|
||||
else warn "未识别的包管理器，跳过系统包安装(请确保已手动装好 gcc/ffmpeg 等)"; fi
|
||||
|
||||
# 解压后 coscli/bin 里的二进制常丢失 +x，先无条件加执行权限（放最前，确保一定执行）
|
||||
for b in coscli-linux-amd64 coscli bin/ffmpeg bin/ffprobe; do
|
||||
[ -f "$PROJECT_DIR/$b" ] && chmod +x "$PROJECT_DIR/$b" 2>/dev/null || true
|
||||
done
|
||||
log "已给 coscli / bin 二进制加执行权限"
|
||||
|
||||
# =============================================================================
|
||||
log "[1/7] 安装系统依赖(gcc 工具链 + opencv/运行库)"
|
||||
# =============================================================================
|
||||
# 关键环境坑：
|
||||
# - opencv-python 在无图形界面的服务器上 import 会报 libGL.so.1 缺失 → 装 mesa-libGL/libgl1
|
||||
# - 还需要 glib2(libgthread-2.0) 否则 opencv 也可能报错
|
||||
# - gcc/gcc-c++/make：少数包没有 wheel 时要本地编译
|
||||
case "$PKG" in
|
||||
dnf|yum)
|
||||
# openblas-devel：万一 scipy/numpy 走源码编译时需要(优先还是用 wheel)
|
||||
$SUDO $PKG install -y gcc gcc-c++ gcc-gfortran make git tar xz which \
|
||||
mesa-libGL glib2 openblas-devel || warn "部分系统包安装失败，继续"
|
||||
;;
|
||||
apt)
|
||||
$SUDO apt-get update -y || true
|
||||
$SUDO apt-get install -y build-essential gfortran git tar xz-utils curl \
|
||||
libgl1 libglib2.0-0 libopenblas-dev || warn "部分系统包安装失败，继续"
|
||||
;;
|
||||
esac
|
||||
|
||||
# =============================================================================
|
||||
log "[2/7] 安装 ffmpeg(含 AV1/dav1d 解码) → 项目目录 bin/"
|
||||
# =============================================================================
|
||||
# 直接下载静态构建(johnvansickle，自带 dav1d)到项目的 bin/，代码会自动优先用它
|
||||
# (config_utils 启动时把 bin/ 提到 PATH 最前)。不依赖系统 ffmpeg，也不用 sudo。
|
||||
# 注意：CPU 版(无 NVDEC)；要 GPU 硬解需另装 cuda 版 ffmpeg。
|
||||
FFMPEG_DIR="$PROJECT_DIR/bin"
|
||||
LOCAL_FFMPEG="$FFMPEG_DIR/ffmpeg"
|
||||
need_ffmpeg=1
|
||||
if [ -x "$LOCAL_FFMPEG" ] && "$LOCAL_FFMPEG" -hide_banner -decoders 2>/dev/null | grep -qiE 'dav1d|av1'; then
|
||||
log "项目自带 ffmpeg 已存在且支持 AV1: $LOCAL_FFMPEG，跳过下载"
|
||||
need_ffmpeg=0
|
||||
fi
|
||||
if [ "$need_ffmpeg" -eq 1 ]; then
|
||||
if have curl || have wget; then
|
||||
mkdir -p "$FFMPEG_DIR"
|
||||
tmp="$(mktemp -d)"
|
||||
log "下载 ffmpeg 静态包到 $FFMPEG_DIR ..."
|
||||
if have curl; then curl -L "$FFMPEG_STATIC_URL" -o "$tmp/ff.tar.xz"
|
||||
else wget -O "$tmp/ff.tar.xz" "$FFMPEG_STATIC_URL"; fi
|
||||
tar -xf "$tmp/ff.tar.xz" -C "$tmp"
|
||||
ffdir="$(find "$tmp" -maxdepth 1 -type d -name 'ffmpeg-*' | head -1)"
|
||||
if [ -n "$ffdir" ]; then
|
||||
cp "$ffdir/ffmpeg" "$ffdir/ffprobe" "$FFMPEG_DIR/"
|
||||
chmod +x "$FFMPEG_DIR/ffmpeg" "$FFMPEG_DIR/ffprobe"
|
||||
log "ffmpeg/ffprobe 已装到 $FFMPEG_DIR"
|
||||
else
|
||||
err "解压后未找到 ffmpeg 目录，请手动安装 ffmpeg"
|
||||
fi
|
||||
rm -rf "$tmp"
|
||||
else
|
||||
err "没有 curl/wget，无法下载 ffmpeg，请手动安装"
|
||||
fi
|
||||
fi
|
||||
|
||||
# =============================================================================
|
||||
log "[3/7] 安装 uv"
|
||||
# =============================================================================
|
||||
if ! have uv; then
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
# uv 默认装到 ~/.local/bin 或 ~/.cargo/bin
|
||||
export PATH="$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
|
||||
fi
|
||||
have uv || { err "uv 安装失败，请检查网络"; exit 1; }
|
||||
log "uv 版本: $(uv --version)"
|
||||
|
||||
# =============================================================================
|
||||
log "[4/7] 用 uv 安装 Python ${PYTHON_VERSION}"
|
||||
# =============================================================================
|
||||
uv python install "$PYTHON_VERSION" || warn "uv python install 失败(可能已装)"
|
||||
|
||||
# =============================================================================
|
||||
log "[5/7] 创建虚拟环境并安装依赖"
|
||||
# =============================================================================
|
||||
# 优先用 pyproject.toml + uv.lock(规范、可复现)；没有则回退 requirements.txt。
|
||||
# pyproject 的 [tool.uv] 已配置：禁止 av/numpy/sklearn 本地编译 + 清华镜像。
|
||||
# 用 uv venv + uv pip install(现代解析器)。--only-binary 让它自动挑有 wheel 的版本，
|
||||
# 避免某些受限镜像把新版本只给源码包导致本地编译/报错(不像 uv sync 的 no-build 会硬失败)。
|
||||
[ -d "$VENV_DIR" ] || uv venv --python "$PYTHON_VERSION" "$VENV_DIR"
|
||||
# shellcheck disable=SC1091
|
||||
source "$VENV_DIR/bin/activate"
|
||||
if [ -f requirements.txt ]; then
|
||||
uv pip install -r requirements.txt --index-url "$PIP_MIRROR" \
|
||||
--only-binary numpy,scipy,scikit-learn,opencv-python-headless,Pillow,av \
|
||||
|| uv pip install -r requirements.txt --index-url "$PIP_MIRROR"
|
||||
else
|
||||
warn "未找到 requirements.txt，跳过依赖安装"
|
||||
fi
|
||||
|
||||
# 激活 venv 供后续自检使用
|
||||
# shellcheck disable=SC1091
|
||||
[ -f "$VENV_DIR/bin/activate" ] && source "$VENV_DIR/bin/activate"
|
||||
log "Python: $(python --version 2>&1) 位置: $(command -v python)"
|
||||
|
||||
# =============================================================================
|
||||
log "[6/7] 写出环境变量脚本 activate.sh"
|
||||
# =============================================================================
|
||||
# 常见环境坑统一在这里处理，跑项目前 source 一下即可
|
||||
# 用单引号 heredoc 让脚本体保持字面(不在生成时展开)，实现"自动定位所在目录"，
|
||||
# 这样整个项目可以搬到任意路径，activate.sh 仍能找到自己的 .venv / bin。
|
||||
cat > "$PROJECT_DIR/activate.sh" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
# 跑项目前先: source activate.sh —— 自动定位项目目录，项目可放任意路径
|
||||
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
# Python：优先项目内自包含的 ./python(make_portable.sh 产物，可随便放)；否则用 .venv
|
||||
if [ -x "$HERE/python/bin/python3" ]; then
|
||||
export PATH="$HERE/bin:$HERE/python/bin:$PATH"
|
||||
else
|
||||
export VIRTUAL_ENV="$HERE/.venv"
|
||||
export PATH="$HERE/bin:$VIRTUAL_ENV/bin:$HOME/.local/bin:$HOME/.cargo/bin:$PATH"
|
||||
fi
|
||||
|
||||
# UTF-8：项目有中文文件名(数据.jsonl)。优先选系统存在的 UTF-8 locale，避免 setlocale 警告；
|
||||
# 并强制 Python 用 UTF-8(即使系统没有 UTF-8 locale 也能正确处理中文文件名)。
|
||||
if locale -a 2>/dev/null | grep -qiE '^C\.(utf-?8|UTF-?8)$'; then
|
||||
export LANG=C.UTF-8 LC_ALL=C.UTF-8
|
||||
elif locale -a 2>/dev/null | grep -qiE 'en_US\.(utf-?8|UTF-?8)'; then
|
||||
export LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8
|
||||
fi
|
||||
export PYTHONUTF8=1
|
||||
export PYTHONIOENCODING=utf-8
|
||||
|
||||
# HuggingFace 镜像（联网下模型时用）
|
||||
export HF_ENDPOINT=__HF_ENDPOINT__
|
||||
# 模型缓存：项目内有 ./models 就用它(自包含包)，否则用默认 ~/.cache/huggingface
|
||||
[ -d "$HERE/models" ] && export HF_HOME="$HERE/models"
|
||||
# 只在模型【确实已缓存】时才强制离线，否则允许联网下载(避免"没模型又禁网"的死锁)
|
||||
if ls "${HF_HOME:-$HOME/.cache/huggingface}"/hub/models--*faster-whisper* >/dev/null 2>&1; then
|
||||
export HF_HUB_OFFLINE=1
|
||||
export TRANSFORMERS_OFFLINE=1
|
||||
fi
|
||||
|
||||
# 若项目自带 CUDA 库(WHISPER_GPU=1 打包)，加进 LD_LIBRARY_PATH，让 ctranslate2 找到(whisper GPU)
|
||||
_nvlibs="$(ls -d "$HERE"/python/lib/python*/site-packages/nvidia/*/lib 2>/dev/null | tr '\n' ':')"
|
||||
[ -n "$_nvlibs" ] && export LD_LIBRARY_PATH="${_nvlibs}${LD_LIBRARY_PATH:-}"
|
||||
|
||||
# 限制数值库线程：防止 whisper/numpy/openblas 每个都抢满核(多视频并发时互相踩)
|
||||
export OMP_NUM_THREADS=2
|
||||
export OPENBLAS_NUM_THREADS=2
|
||||
export MKL_NUM_THREADS=2
|
||||
|
||||
echo "环境已激活: $(python --version 2>&1) | $HERE"
|
||||
EOF
|
||||
sed -i "s|__HF_ENDPOINT__|$HF_ENDPOINT_URL|" "$PROJECT_DIR/activate.sh"
|
||||
chmod +x "$PROJECT_DIR/activate.sh"
|
||||
|
||||
# 提高文件句柄上限(高并发 + 海量小帧文件)，当前会话生效；永久需改 /etc/security/limits.conf
|
||||
ulimit -n 65535 2>/dev/null || warn "ulimit 提升失败(非致命)，如需永久生效请改 /etc/security/limits.conf"
|
||||
|
||||
|
||||
# =============================================================================
|
||||
log "[7/7] 环境自检"
|
||||
# =============================================================================
|
||||
ok=1
|
||||
check() { if eval "$2" >/dev/null 2>&1; then echo " ✓ $1"; else echo " ✗ $1"; ok=0; fi; }
|
||||
|
||||
check "gcc" "gcc --version"
|
||||
check "ffmpeg" "ffmpeg -version"
|
||||
check "ffprobe" "ffprobe -version"
|
||||
check "ffmpeg 支持 AV1" "ffmpeg -hide_banner -decoders | grep -iE 'dav1d|av1'"
|
||||
check "uv" "uv --version"
|
||||
check "python" "python --version"
|
||||
check "import cv2" "python -c 'import cv2'"
|
||||
check "import numpy" "python -c 'import numpy'"
|
||||
check "import sklearn" "python -c 'import sklearn'"
|
||||
check "import av(PyAV)" "python -c 'import av'"
|
||||
check "import faster_whisper" "python -c 'import faster_whisper'"
|
||||
check "import qcloud_cos" "python -c 'import qcloud_cos'"
|
||||
|
||||
# GPU 检测(可选，有则抽帧能硬解加速)
|
||||
if have nvidia-smi && nvidia-smi >/dev/null 2>&1; then
|
||||
echo " ✓ 检测到 NVIDIA GPU(抽帧可硬解加速)"
|
||||
else
|
||||
echo " · 未检测到 GPU(抽帧用 CPU，正常)"
|
||||
fi
|
||||
|
||||
echo
|
||||
if [ "$ok" -eq 1 ]; then
|
||||
log "全部就绪 ✅"
|
||||
else
|
||||
warn "有项目未通过(见上面 ✗)，请按提示排查"
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
|
||||
下一步：
|
||||
source activate.sh
|
||||
python main.py --source-list ./text2 --max-workers 12
|
||||
|
||||
EOF
|
||||
52
scripts/stop.sh
Normal file
52
scripts/stop.sh
Normal file
@@ -0,0 +1,52 @@
|
||||
#!/usr/bin/env bash
|
||||
# 停止本项目所有进程(主程序 + 进程池worker + ffmpeg子进程 + coscli)。
|
||||
# 用法: bash stop.sh 普通停止
|
||||
# bash stop.sh -y 不确认直接停
|
||||
set -u
|
||||
|
||||
echo "==> 即将停止的进程:"
|
||||
ps -eo pid,etime,rss,args 2>/dev/null | grep -E "main\.py|run_worker\.py|multiprocessing|forkserver|[f]fmpeg|[c]oscli" | grep -v grep | grep -v "stop.sh" || echo " (没找到运行中的进程)"
|
||||
echo ""
|
||||
|
||||
if [ "${1:-}" != "-y" ]; then
|
||||
read -r -p "确认全部杀掉? [y/N] " ans
|
||||
case "$ans" in
|
||||
y|Y) ;;
|
||||
*) echo "已取消"; exit 0 ;;
|
||||
esac
|
||||
fi
|
||||
|
||||
echo "==> 1. 杀主程序 + 进程池worker"
|
||||
pkill -9 -f "main.py" 2>/dev/null && echo " ✓ main.py"
|
||||
pkill -9 -f "run_worker.py" 2>/dev/null && echo " ✓ run_worker.py"
|
||||
pkill -9 -f "multiprocessing" 2>/dev/null && echo " ✓ multiprocessing worker"
|
||||
pkill -9 -f "forkserver" 2>/dev/null && echo " ✓ forkserver"
|
||||
|
||||
echo "==> 2. 杀 ffmpeg 子进程(不会自动死, 必须单独杀, 否则占GPU/CPU)"
|
||||
pkill -9 ffmpeg 2>/dev/null && echo " ✓ ffmpeg"
|
||||
pkill -9 ffprobe 2>/dev/null && echo " ✓ ffprobe"
|
||||
|
||||
echo "==> 3. 杀残留 coscli(正在传输的)"
|
||||
pkill -9 -f "coscli" 2>/dev/null && echo " ✓ coscli"
|
||||
|
||||
sleep 1
|
||||
echo ""
|
||||
echo "==> 检查残留:"
|
||||
LEFT=$(ps -eo pid,args 2>/dev/null | grep -E "main\.py|run_worker\.py|multiprocessing|forkserver|[f]fmpeg|[c]oscli" | grep -v grep | grep -v "stop.sh")
|
||||
if [ -z "$LEFT" ]; then
|
||||
echo " ✓ 全部已停止"
|
||||
else
|
||||
echo " ⚠️ 仍有残留, 再杀一次:"
|
||||
echo "$LEFT"
|
||||
echo "$LEFT" | awk '{print $1}' | xargs -r kill -9 2>/dev/null
|
||||
echo " 已强制杀掉"
|
||||
fi
|
||||
|
||||
# 可选: 看GPU是否释放
|
||||
if command -v nvidia-smi >/dev/null 2>&1; then
|
||||
echo ""
|
||||
echo "==> GPU 占用(确认ffmpeg/whisper已释放):"
|
||||
nvidia-smi --query-gpu=index,memory.used,utilization.gpu --format=csv,noheader 2>/dev/null
|
||||
fi
|
||||
echo ""
|
||||
echo "停止完成。重启: bash scripts/启动.sh 30"
|
||||
164
scripts/启动.sh
Normal file
164
scripts/启动.sh
Normal file
@@ -0,0 +1,164 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 一键启动(防 SIGHUP/外部kill 脱离终端 + 调好参数 + 启动前自检 + 防重复启动)
|
||||
#
|
||||
# 两种模式(MODE 环境变量切换):
|
||||
# MODE=worker (默认) —— 常驻 worker, 从 COS 自动认领任务(run_worker.py)。不需要本地 text2。
|
||||
# MODE=local —— 旧模式, 处理本地 ./text2 下的任务txt(main.py)。
|
||||
#
|
||||
# 用法:
|
||||
# bash scripts/启动.sh # 默认 worker 模式启动
|
||||
# bash scripts/启动.sh 32 # 临时把 max-workers 改成 32
|
||||
# MODE=local bash scripts/启动.sh # 用旧的本地 text2 模式
|
||||
# MAX_WORKERS=20 MEM_GUARD=40 bash scripts/启动.sh
|
||||
#
|
||||
# 停止: bash scripts/stop.sh -y
|
||||
# =============================================================================
|
||||
set -u
|
||||
cd "$(cd "$(dirname "$0")/.." && pwd)"
|
||||
ROOT="$(pwd)"
|
||||
|
||||
# ---- 可调参数(命令行第1个参数 / 环境变量 均可覆盖) ----
|
||||
# MODE 未显式指定时, 读 config.dispatch.source 决定: local -> 本地text2, 其余 -> COS常驻
|
||||
if [ -z "${MODE:-}" ]; then
|
||||
_src="$(python -c "from config.config_utils import config; print((config.get_dispatch_config() or {}).get('source','cos'))" 2>/dev/null || echo cos)"
|
||||
[ "$_src" = "local" ] && MODE="local" || MODE="worker"
|
||||
fi
|
||||
MODE="${MODE:-worker}" # worker=COS认领常驻 / local=本地text2
|
||||
MAX_WORKERS="${1:-${MAX_WORKERS:-25}}" # 0=不传给python,由main.py/run_worker.py自动按本机CPU核数选
|
||||
# 共享机/跟OCR服务同机部署的, 建议显式传小一点的值(如24), 别用自动
|
||||
MAX_TASKS="${MAX_TASKS:-0}" # 0=不回收worker(避免3.11回收缩水把并发缩到0)
|
||||
MEM_GUARD="${MEM_GUARD:-30}" # 可用内存低于此(G)暂停启动新视频
|
||||
SOURCE_LIST="${SOURCE_LIST:-./text2}" # 仅 local 模式: 任务txt目录
|
||||
EXECUTOR="${EXECUTOR:-process}" # process=进程池(绕GIL) / thread
|
||||
LOG="${LOG:-app.log}"
|
||||
NICE="${NICE:-15}" # 处理进程CPU优先级(0~19,越大越低)
|
||||
LOG_MAX_MB="${LOG_MAX_MB:-200}" # $LOG 超过这个大小(MB)就在启动前轮转一次, 0=不轮转
|
||||
|
||||
# ---- 启动前轮转一次日志(常驻进程一跑数天/数周, >>追加写不会自己变小) ----
|
||||
_rotate_log() {
|
||||
local f="$1" maxmb="$2"
|
||||
[ "$maxmb" -le 0 ] && return 0
|
||||
[ -f "$f" ] || return 0
|
||||
local sz_mb=$(( $(stat -c%s "$f" 2>/dev/null || echo 0) / 1024 / 1024 ))
|
||||
if [ "$sz_mb" -ge "$maxmb" ]; then
|
||||
local ts; ts="$(date +%Y%m%d_%H%M%S)"
|
||||
mv "$f" "${f}.${ts}" 2>/dev/null && echo "[日志轮转] $f (${sz_mb}M) → ${f}.${ts}, 重新开始写"
|
||||
ls -t "${f}".[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -f # 只留最近5份轮转备份
|
||||
fi
|
||||
}
|
||||
_rotate_log "$LOG" "$LOG_MAX_MB"
|
||||
|
||||
# 按模式确定入口与进程匹配模式
|
||||
if [ "$MODE" = "worker" ]; then
|
||||
ENTRY="run_worker.py"
|
||||
PROC_PAT='python .*run_worker\.py'
|
||||
else
|
||||
ENTRY="main.py"
|
||||
PROC_PAT='python .*main\.py'
|
||||
fi
|
||||
|
||||
MAX_WORKERS_DISP="$MAX_WORKERS"
|
||||
[ "$MAX_WORKERS" = "0" ] && MAX_WORKERS_DISP="0(自动按CPU核数选, 实际值见下面python启动日志)"
|
||||
|
||||
echo "=================================================="
|
||||
echo " 启动 $ENTRY (MODE=$MODE)"
|
||||
echo " max-workers=$MAX_WORKERS_DISP max-tasks-per-child=$MAX_TASKS mem-guard=${MEM_GUARD}G"
|
||||
echo " executor=$EXECUTOR nice=$NICE(让OCR服务优先) 日志→$LOG"
|
||||
[ "$MODE" = "local" ] && echo " source-list=$SOURCE_LIST"
|
||||
echo "=================================================="
|
||||
|
||||
# ---- 1. 防重复启动 ----
|
||||
if pgrep -af "$PROC_PAT" 2>/dev/null | grep -v 'multiprocessing' | grep -v 'grep' >/dev/null; then
|
||||
echo "✗ 已经有 $ENTRY 在运行了, 不重复启动。"
|
||||
echo " 查看: pgrep -af '$PROC_PAT' | grep -v multiprocessing"
|
||||
echo " 如要重启: 先 bash scripts/stop.sh -y"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---- 2. 启动前自检 ----
|
||||
fail=0
|
||||
need() { command -v "$1" >/dev/null 2>&1 && echo " ✓ $1" || { echo " ✗ 缺少 $1 (需在 PATH 或项目目录)"; fail=1; }; }
|
||||
echo "[依赖检查]"
|
||||
need python
|
||||
need ffmpeg
|
||||
need ffprobe
|
||||
if command -v coscli >/dev/null 2>&1; then echo " ✓ coscli"
|
||||
elif [ -x "./coscli-linux-amd64" ]; then echo " ✓ ./coscli-linux-amd64"
|
||||
else echo " ✗ 找不到 coscli / ./coscli-linux-amd64"; fail=1; fi
|
||||
|
||||
echo "[配置/任务检查]"
|
||||
[ -f config/config.json ] && echo " ✓ config/config.json" || { echo " ✗ 缺 config/config.json"; fail=1; }
|
||||
|
||||
if [ "$MODE" = "worker" ]; then
|
||||
# worker 模式: 任务来自 COS, 不需要本地 text2; 检查 dispatch 配置与机器ID + COS 队列可达
|
||||
chk=$(python - <<'PY' 2>&1
|
||||
import sys
|
||||
try:
|
||||
from config.config_utils import config
|
||||
from tool import job_cos
|
||||
dp = config.get_dispatch_config() or {}
|
||||
if not dp.get("jobs_root"):
|
||||
print("NO_JOBS_ROOT"); sys.exit(0)
|
||||
mid = job_cos.get_machine_id()
|
||||
todo = job_cos.list_todo() # 顺带验证 COS 队列可达
|
||||
print("OK\t%s\t%d" % (mid, len(todo)))
|
||||
except Exception as e:
|
||||
print("ERR\t%s" % e)
|
||||
PY
|
||||
)
|
||||
case "$chk" in
|
||||
OK*) mid=$(echo "$chk" | cut -f2); n=$(echo "$chk" | cut -f3)
|
||||
echo " ✓ dispatch 配置OK 机器ID=$mid todo剩余分片=$n"
|
||||
[ "$n" = "0" ] && echo " ⚠ todo 队列为空(worker会空闲等待; 先 python tools/seed_jobs.py 灌任务)";;
|
||||
NO_JOBS_ROOT) echo " ✗ config.dispatch.jobs_root 未配置(取任务地址)"; fail=1;;
|
||||
*) echo " ✗ 访问 COS 任务队列失败: ${chk#ERR }"; fail=1;;
|
||||
esac
|
||||
else
|
||||
if ls "$SOURCE_LIST"/*.txt >/dev/null 2>&1; then
|
||||
n=$(ls "$SOURCE_LIST"/*.txt 2>/dev/null | wc -l)
|
||||
echo " ✓ $SOURCE_LIST 有 $n 个任务txt"
|
||||
else
|
||||
echo " ✗ $SOURCE_LIST 下没有 .txt 任务文件"; fail=1
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ "$fail" -ne 0 ]; then
|
||||
echo ""
|
||||
echo "✗ 自检未通过, 已中止(没启动)。修好上面 ✗ 的项再跑。"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# ---- 3. 启动(setsid+nohup+disown 三保险, SSH断开也杀不到) ----
|
||||
echo ""
|
||||
echo "[启动中...]"
|
||||
if [ "$MODE" = "worker" ]; then
|
||||
setsid nohup nice -n "$NICE" python run_worker.py \
|
||||
--max-workers "$MAX_WORKERS" \
|
||||
--max-tasks-per-child "$MAX_TASKS" \
|
||||
--mem-guard-gb "$MEM_GUARD" \
|
||||
--executor "$EXECUTOR" >> "$LOG" 2>&1 &
|
||||
else
|
||||
setsid nohup nice -n "$NICE" python main.py \
|
||||
--source-list "$SOURCE_LIST" \
|
||||
--max-workers "$MAX_WORKERS" \
|
||||
--max-tasks-per-child "$MAX_TASKS" \
|
||||
--mem-guard-gb "$MEM_GUARD" \
|
||||
--executor "$EXECUTOR" >> "$LOG" 2>&1 &
|
||||
fi
|
||||
disown 2>/dev/null || true
|
||||
|
||||
# ---- 4. 等几秒确认起来了 ----
|
||||
sleep 6
|
||||
if pgrep -af "$PROC_PAT" 2>/dev/null | grep -v 'multiprocessing' | grep -v 'grep' >/dev/null; then
|
||||
echo "✓ 已启动。日志末尾:"
|
||||
tail -n 5 "$LOG" 2>/dev/null | sed 's/^/ /'
|
||||
echo ""
|
||||
echo " 实时看日志: tail -f $LOG"
|
||||
echo " 数worker: pgrep -af multiprocessing | wc -l (稍等会接近 $MAX_WORKERS_DISP)"
|
||||
echo " 停止: bash scripts/stop.sh -y"
|
||||
else
|
||||
echo "✗ 启动后没检测到进程, 查日志看原因:"
|
||||
tail -n 20 "$LOG" 2>/dev/null | sed 's/^/ /'
|
||||
exit 1
|
||||
fi
|
||||
121
scripts/守护重启.sh
Normal file
121
scripts/守护重启.sh
Normal file
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env bash
|
||||
# =============================================================================
|
||||
# 守护重启: main.py 被外部 kill / 异常退出后, 自动拉起(共享机上挡不住别人kill, 就让它自己回来)。
|
||||
# 【带明确开关, 想关随时关, 不会和你抢】:
|
||||
# 关闭原理 = 放一个"暂停标记文件" .guard_paused, 守护看到它就什么都不做。
|
||||
#
|
||||
# 用法:
|
||||
# bash scripts/守护重启.sh install # 安装(每分钟检测一次, 没在跑就启动)
|
||||
# bash scripts/守护重启.sh off # ★想停就用这个★ 暂停自动重启 + 停掉当前进程
|
||||
# bash scripts/守护重启.sh on # 重新开启自动重启(下一分钟内会被拉起)
|
||||
# bash scripts/守护重启.sh status # 看状态(装没装/开没开/在没在跑)
|
||||
# bash scripts/守护重启.sh uninstall # 彻底卸载(删掉定时任务, 进程不动)
|
||||
# bash scripts/守护重启.sh tick # (cron 内部调用, 别手动跑)
|
||||
#
|
||||
# 启动参数想改, 改下面 START= 这一行即可。
|
||||
# =============================================================================
|
||||
set -u
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
PAUSE="$ROOT/.guard_paused" # 这个文件存在 = 暂停自动重启
|
||||
LOCK="$ROOT/.guard.lock"
|
||||
GLOG="$ROOT/guard.log"
|
||||
CRON_TAG="# ocr-guard" # crontab 里的标识, 用于增删去重
|
||||
|
||||
# ★ 启动命令(按之前调优建议; 想改并发/参数改这里) ★
|
||||
# MODE=worker(默认): 常驻 worker 从 COS 认领任务; MODE=local: 旧的本地 text2 模式。
|
||||
# nice -n 15: 让 co-located 的 OCR 服务优先抢 CPU; max-tasks-per-child 20: 本机内存偏紧
|
||||
# (44G可用/184G总量), 定期回收worker释放常驻内存; 3.11回收有缩水风险(见CLAUDE.md), 但20这个
|
||||
# 量级配合 --mem-guard-gb 观察下来风险可控, 内存不紧的机器可以改回0(不回收)。
|
||||
if [ -z "${MODE:-}" ]; then
|
||||
_src="$(python -c "from config.config_utils import config; print((config.get_dispatch_config() or {}).get('source','cos'))" 2>/dev/null || echo cos)"
|
||||
[ "$_src" = "local" ] && MODE="local" || MODE="worker"
|
||||
fi
|
||||
MODE="${MODE:-worker}"
|
||||
if [ "$MODE" = "worker" ]; then
|
||||
START='nice -n 15 python run_worker.py --max-workers 30 --max-tasks-per-child 20 --mem-guard-gb 30 --executor process'
|
||||
PROC_PAT='python .*run_worker\.py'
|
||||
else
|
||||
START='nice -n 15 python main.py --source-list ./text2 --max-workers 30 --max-tasks-per-child 20 --mem-guard-gb 30 --executor process'
|
||||
PROC_PAT='python .*main\.py'
|
||||
fi
|
||||
|
||||
# 判断父进程是否在跑(forkserver worker 命令行不含入口文件名, 不会误判)
|
||||
running() {
|
||||
pgrep -af "$PROC_PAT" 2>/dev/null | grep -v 'multiprocessing' | grep -v 'grep' >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# app.log 轮转: 进程可能正在往它里面 >> 追加写, 不能直接 mv(会导致进程还在写旧inode, 新文件
|
||||
# 看起来永远是空的) —— 用 copytruncate: 先 cp 一份出去, 再原地截断; O_APPEND 的fd是按"当前
|
||||
# 文件大小"找写入位置的, 原地截断后进程会自动从offset 0继续写这个(同一个)文件, 不受影响。
|
||||
_rotate_applog() {
|
||||
local f="$ROOT/app.log" maxmb=200
|
||||
[ -f "$f" ] || return 0
|
||||
local sz_mb=$(( $(stat -c%s "$f" 2>/dev/null || echo 0) / 1024 / 1024 ))
|
||||
if [ "$sz_mb" -ge "$maxmb" ]; then
|
||||
local ts; ts="$(date +%Y%m%d_%H%M%S)"
|
||||
cp "$f" "${f}.${ts}" 2>/dev/null && : > "$f" \
|
||||
&& echo "$(date '+%F %T') [日志轮转] app.log(${sz_mb}M) → app.log.${ts}, 原地清空续写" >> "$GLOG"
|
||||
ls -t "${f}".[0-9]* 2>/dev/null | tail -n +6 | xargs -r rm -f # 只留最近5份
|
||||
fi
|
||||
}
|
||||
|
||||
case "${1:-status}" in
|
||||
install)
|
||||
( crontab -l 2>/dev/null | grep -vF "$CRON_TAG"
|
||||
echo "* * * * * cd $ROOT && bash scripts/守护重启.sh tick $CRON_TAG" ) | crontab -
|
||||
rm -f "$PAUSE"
|
||||
echo "✓ 守护已安装(每分钟检测), 自动重启: 开启"
|
||||
echo " 想停: bash scripts/守护重启.sh off"
|
||||
;;
|
||||
|
||||
uninstall)
|
||||
crontab -l 2>/dev/null | grep -vF "$CRON_TAG" | crontab -
|
||||
echo "✓ 已卸载守护定时任务(不再自动重启)。当前进程未动。"
|
||||
echo " 如要同时停进程: bash scripts/stop.sh -y"
|
||||
;;
|
||||
|
||||
off)
|
||||
touch "$PAUSE"
|
||||
echo "✓ 已暂停自动重启(标记 $PAUSE)。正在停止当前进程..."
|
||||
bash "$ROOT/scripts/stop.sh" -y
|
||||
echo "✓ 已停止, 且守护不会再把它拉起。恢复运行: bash scripts/守护重启.sh on"
|
||||
;;
|
||||
|
||||
on)
|
||||
rm -f "$PAUSE"
|
||||
echo "✓ 自动重启已开启。若当前没在跑, 一分钟内会被拉起。"
|
||||
;;
|
||||
|
||||
tick)
|
||||
# 暂停中 → 什么都不做(这就是 off 之所以能稳关掉的原因)
|
||||
[ -f "$PAUSE" ] && exit 0
|
||||
# 防并发: 同一时刻只允许一个 tick 启动
|
||||
exec 9>"$LOCK"
|
||||
flock -n 9 || exit 0
|
||||
_rotate_applog # 每分钟顺带检查一次, 跟要不要重启无关(常驻不重启时全靠这个兜底)
|
||||
running && exit 0
|
||||
echo "$(date '+%F %T') 检测到未运行 → 启动" >> "$GLOG"
|
||||
setsid nohup $START >> "$ROOT/app.log" 2>&1 &
|
||||
;;
|
||||
|
||||
status)
|
||||
echo "项目目录: $ROOT"
|
||||
if crontab -l 2>/dev/null | grep -qF "$CRON_TAG"; then
|
||||
echo "守护定时任务: ✓ 已安装(每分钟检测)"
|
||||
else
|
||||
echo "守护定时任务: ✗ 未安装"
|
||||
fi
|
||||
if [ -f "$PAUSE" ]; then
|
||||
echo "自动重启: ⏸ 已暂停 (.guard_paused 存在 → 不会拉起)"
|
||||
else
|
||||
echo "自动重启: ▶ 开启"
|
||||
fi
|
||||
if running; then echo "main.py: 运行中"; else echo "main.py: 未运行"; fi
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "用法: install | on | off | uninstall | status | tick"
|
||||
;;
|
||||
esac
|
||||
0
tool/Ocr/__init__.py
Normal file
0
tool/Ocr/__init__.py
Normal file
372
tool/Ocr/tencentcloud_ocr.py
Normal file
372
tool/Ocr/tencentcloud_ocr.py
Normal file
@@ -0,0 +1,372 @@
|
||||
# -*- encoding: utf-8 -*-
|
||||
'''
|
||||
@File : tencentcloud_ocr.py
|
||||
@Time : @
|
||||
@Author : eason
|
||||
@Version : 1.0
|
||||
@Contact :
|
||||
@License : (C)Copyright 2021-2024
|
||||
@Desc : 将图片转换为base64编码，并使用内部Youtu OCR服务进行文字识别（app_id自动生成）
|
||||
'''
|
||||
|
||||
import json
|
||||
import os
|
||||
import base64
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
import requests
|
||||
from config.config_utils import config # 如果不再需要配置可以移除，但保留其他配置可能使用
|
||||
|
||||
|
||||
# ---- OCR 端点（多 IP 负载均衡）配置 ----
|
||||
OCR_PATH = "/youtu/ocrapi/hpgeneralocr_v3"
|
||||
OCR_DEFAULT_PORT = 18081 # config 里只填 IP 时自动补的端口
|
||||
DEFAULT_ENDPOINTS = ["http://asr.ocr.general.smart.l20.production.polaris:18081"]
|
||||
WEIGHTS_FILE = "./config/ocr_weights.json"
|
||||
|
||||
|
||||
def normalize_endpoint(e):
|
||||
"""把 'IP' / 'IP:port' / 'http://IP:port' 统一成 'http://IP:port'。
|
||||
config 里只填裸 IP 时，自动补 http:// 和默认端口。"""
|
||||
if not e:
|
||||
return None
|
||||
e = str(e).strip().rstrip("/")
|
||||
if not e:
|
||||
return None
|
||||
if not (e.startswith("http://") or e.startswith("https://")):
|
||||
e = "http://" + e
|
||||
host = e.split("://", 1)[1].split("/", 1)[0] # 取 host[:port] 部分
|
||||
if ":" not in host: # 没写端口就补默认端口
|
||||
e = e + f":{OCR_DEFAULT_PORT}"
|
||||
return e
|
||||
|
||||
|
||||
def img2base64(img_path: str) -> str:
|
||||
"""将图片转换为base64编码"""
|
||||
with open(img_path, 'rb') as f:
|
||||
image_base64 = base64.b64encode(f.read()).decode()
|
||||
return image_base64
|
||||
|
||||
|
||||
def get_ocr_endpoints():
|
||||
"""从配置读取 OCR 端点列表，自动归一化（只填 IP 也行）；缺省回退到内置默认。"""
|
||||
ocr_cfg = config.get_tencent_ocr_config() or {}
|
||||
eps = ocr_cfg.get("endpoints")
|
||||
if isinstance(eps, str):
|
||||
eps = [eps]
|
||||
if not eps:
|
||||
eps = DEFAULT_ENDPOINTS
|
||||
return [u for u in (normalize_endpoint(e) for e in eps) if u]
|
||||
|
||||
|
||||
def load_endpoint_weights():
|
||||
"""读取压测产出的 QPS 权重文件 {base_url: qps}；不存在则返回 {}。"""
|
||||
if not os.path.exists(WEIGHTS_FILE):
|
||||
return {}
|
||||
try:
|
||||
with open(WEIGHTS_FILE, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
return {k.rstrip("/"): float(v) for k, v in data.items() if float(v) > 0}
|
||||
except (json.JSONDecodeError, ValueError, TypeError, OSError):
|
||||
return {}
|
||||
|
||||
|
||||
class OcrDispatcher:
|
||||
"""线程安全的 OCR 端点分发器：用 P-of-d-Choices(默认d=3)做【负载/延迟感知】派发——
|
||||
随机取 sample_size 个端点, 把请求发给其中【在途更少 × 更快】的那个。慢/积压端点自动少分。
|
||||
每个进程独立按后端真实快慢反应 → 无需任何协调/中心LB, 也能在 120 台 OCR 上涌现出全局均衡。
|
||||
d 从 2 提到 3(甚至更多): 端点数越多, 随机抽2个都恰好较差的概率越高, 抽样数多一个能明显
|
||||
收窄这个方差, 之后收益递减, 不用无限往上加。
|
||||
失败端点冷却摘除(冷却时长带随机抖动, 防止多个进程对同一端点同时解冷却又一起打回去,
|
||||
造成"整体一起失败-一起恢复-一起打爆"的同步震荡)。多个并发 OCR 线程共享一个实例。"""
|
||||
|
||||
def __init__(self, endpoints, weights=None, cooldown=30.0, sample_size=3):
|
||||
self.endpoints = list(endpoints)
|
||||
# PID 打乱(历史原因保留): 让各进程初始顺序不同; 现在 P-of-d 是随机取样, 影响不大但无害。
|
||||
try:
|
||||
random.Random(os.getpid()).shuffle(self.endpoints)
|
||||
except Exception:
|
||||
pass
|
||||
weights = weights or {}
|
||||
# 压测权重(QPS): 作为"处理能力", 强端点 score 更低 → 多分。无权重默认 1.0。
|
||||
self.weights = {e: max(float(weights.get(e, 1.0)), 0.0001) for e in self.endpoints}
|
||||
self.cooldown = cooldown
|
||||
self.sample_size = max(2, int(sample_size)) # 每次候选抽样个数(P-of-d 的 d)
|
||||
self._lock = threading.Lock()
|
||||
self._inflight = {e: 0 for e in self.endpoints} # 当前在途请求数(本进程视角)
|
||||
self._lat = {e: 0.05 for e in self.endpoints} # 延迟 EWMA(秒), 初始乐观→新端点也会被试
|
||||
self._cooldown_until = {e: 0.0 for e in self.endpoints}
|
||||
self._rng = random.Random(os.getpid() ^ 0x9E3779B1)
|
||||
# —— 派发计数(验证分发是否均匀): 每端点累计派发数, 每 _dump_every 次落盘一次本进程文件 ——
|
||||
self._count = {e: 0 for e in self.endpoints}
|
||||
self._since_dump = 0
|
||||
try:
|
||||
self._dump_every = int((config.get_tencent_ocr_config() or {}).get('dispatch_log_every', 0) or 0)
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
self._dump_every = 0
|
||||
|
||||
def _score(self, e):
|
||||
"""负载分: (在途+1) × 延迟EWMA / 权重。越低越优先。"""
|
||||
return (self._inflight[e] + 1) * self._lat[e] / self.weights[e]
|
||||
|
||||
def pick(self):
|
||||
"""P-of-d: 随机不重复抽 sample_size(默认3) 个可用端点, 选负载分最低的;
|
||||
选中即占一个在途名额(必须配对 done())。全部冷却时退化为全部可用。"""
|
||||
now = time.time()
|
||||
snapshot = None
|
||||
with self._lock:
|
||||
available = [e for e in self.endpoints if self._cooldown_until[e] <= now]
|
||||
if not available:
|
||||
available = list(self.endpoints) # 全挂了也得发, 给一次机会
|
||||
if len(available) == 1:
|
||||
chosen = available[0]
|
||||
else:
|
||||
k = min(self.sample_size, len(available))
|
||||
cands = self._rng.sample(available, k) # 不重复抽样, 比各自 choice() 更有效(不会抽到同一个)
|
||||
chosen = min(cands, key=self._score)
|
||||
self._inflight[chosen] = self._inflight.get(chosen, 0) + 1
|
||||
self._count[chosen] = self._count.get(chosen, 0) + 1
|
||||
if self._dump_every > 0:
|
||||
self._since_dump += 1
|
||||
if self._since_dump >= self._dump_every:
|
||||
self._since_dump = 0
|
||||
snapshot = (dict(self._count),
|
||||
{e: round(self._lat[e], 4) for e in self._lat})
|
||||
if snapshot is not None: # 落盘在锁外, 不阻塞其它线程派发
|
||||
self._write_dump(snapshot[0], snapshot[1])
|
||||
return chosen
|
||||
|
||||
def _write_dump(self, counts, lats):
|
||||
"""把本进程的累计派发计数写到 ./ocr_dispatch/<pid>.json(覆盖式, 单进程单文件无竞争)。"""
|
||||
try:
|
||||
d = "./ocr_dispatch"
|
||||
os.makedirs(d, exist_ok=True)
|
||||
path = os.path.join(d, f"{os.getpid()}.json")
|
||||
data = {"pid": os.getpid(), "ts": time.time(),
|
||||
"total": sum(counts.values()), "counts": counts, "lat_ewma": lats}
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False)
|
||||
os.replace(tmp, path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def done(self, endpoint, latency=None, ok=True):
|
||||
"""请求结束回调: 释放在途名额; 成功则更新延迟EWMA, 失败则冷却摘除(冷却时长带±30%抖动,
|
||||
避免同一时刻失败的多个进程在同一时刻解冷却、一起冲回去打同一个刚失败的端点)。"""
|
||||
with self._lock:
|
||||
if self._inflight.get(endpoint, 0) > 0:
|
||||
self._inflight[endpoint] -= 1
|
||||
if endpoint not in self._cooldown_until:
|
||||
return
|
||||
if ok:
|
||||
if latency is not None:
|
||||
self._lat[endpoint] = 0.7 * self._lat[endpoint] + 0.3 * max(latency, 0.001)
|
||||
self._cooldown_until[endpoint] = 0.0
|
||||
else:
|
||||
jitter = self._rng.uniform(0.7, 1.3)
|
||||
self._cooldown_until[endpoint] = time.time() + self.cooldown * jitter
|
||||
|
||||
# —— 兼容旧接口(内部都走 done) ——
|
||||
def report_failure(self, endpoint):
|
||||
self.done(endpoint, ok=False)
|
||||
|
||||
def report_success(self, endpoint):
|
||||
self.done(endpoint, ok=True)
|
||||
|
||||
|
||||
_dispatcher = None
|
||||
_dispatcher_lock = threading.Lock()
|
||||
|
||||
|
||||
def get_dispatcher():
|
||||
"""惰性单例分发器（首次调用时读取端点+权重）。"""
|
||||
global _dispatcher
|
||||
if _dispatcher is None:
|
||||
with _dispatcher_lock:
|
||||
if _dispatcher is None:
|
||||
eps = get_ocr_endpoints()
|
||||
_dispatcher = OcrDispatcher(eps, load_endpoint_weights())
|
||||
return _dispatcher
|
||||
|
||||
|
||||
def _build_payload(image_base64: str) -> dict:
|
||||
timestamp = int(time.time() * 1000)
|
||||
rand_suffix = random.randint(100000, 999999)
|
||||
app_id = f"{timestamp}{rand_suffix}"
|
||||
return {
|
||||
"app_id": app_id,
|
||||
"image": image_base64,
|
||||
"ocr_template": "OCR",
|
||||
"options": {"pdf_page_index": 0, "detect_text": True},
|
||||
}
|
||||
|
||||
|
||||
# 跨进程【全局 OCR 并发上限】信号量。进程池下由 main 在父进程创建、经 initializer 注入每个 worker;
|
||||
# 不管多少进程×多少线程,全局同时在飞的 OCR 请求数被这个信号量硬性卡住,防止打爆端点。
|
||||
# 为 None 时不限流(线程池模式 / 未注入)。
|
||||
_ocr_semaphore = None
|
||||
|
||||
|
||||
def set_ocr_semaphore(sem):
|
||||
"""由 worker 进程初始化时调用,注入全局并发信号量。"""
|
||||
global _ocr_semaphore
|
||||
_ocr_semaphore = sem
|
||||
|
||||
|
||||
# 每个进程一个持久 Session(连接池/keep-alive 复用连接)。
|
||||
# 原来用 requests.post() 每次新建+关闭连接 → 高并发下海量 TIME_WAIT → 本地临时端口耗尽
|
||||
# → "Cannot assign requested address" / 连接超时(表现成"所有端点超时", 其实是本机端口用完)。
|
||||
# 复用连接后, 到每个端点只保持少量长连接, 端口不再churn。惰性创建 → 进程池下各进程独立 Session。
|
||||
_session = None
|
||||
_session_lock = threading.Lock()
|
||||
|
||||
|
||||
def _get_session():
|
||||
global _session
|
||||
if _session is None:
|
||||
with _session_lock:
|
||||
if _session is None:
|
||||
s = requests.Session()
|
||||
adapter = requests.adapters.HTTPAdapter(
|
||||
pool_connections=256, # 连接池数(≥端点数, 每端点一个池)
|
||||
pool_maxsize=64, # 每端点最大并发连接(复用)
|
||||
max_retries=0,
|
||||
pool_block=False)
|
||||
s.mount("http://", adapter)
|
||||
s.mount("https://", adapter)
|
||||
_session = s
|
||||
return _session
|
||||
|
||||
|
||||
def ocr_request_once(base_url: str, image_base64: str, timeout: int = 30) -> str:
|
||||
"""向指定端点发一次 OCR 请求，返回原始响应文本；失败抛异常。
|
||||
受全局信号量限流;用持久 Session 复用连接(防端口耗尽)。"""
|
||||
url = base_url.rstrip("/") + OCR_PATH
|
||||
headers = {"Content-Type": "application/json"}
|
||||
sem = _ocr_semaphore
|
||||
if sem is not None:
|
||||
sem.acquire()
|
||||
try:
|
||||
resp = _get_session().post(url, headers=headers, json=_build_payload(image_base64),
|
||||
timeout=timeout)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
finally:
|
||||
if sem is not None:
|
||||
sem.release()
|
||||
|
||||
|
||||
def _normalize_youtu_to_tencent(raw: str) -> str:
|
||||
"""将 Youtu OCR 返回格式转为项目其他模块期望的腾讯云 OCR 格式。"""
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return raw
|
||||
|
||||
# 已经是腾讯云格式则直接返回
|
||||
if "TextDetections" in data:
|
||||
return raw
|
||||
|
||||
if data.get("error_code") != 0:
|
||||
return json.dumps({"TextDetections": [], "Language": "zh",
|
||||
"RequestId": data.get("session_id", "")}, ensure_ascii=False)
|
||||
|
||||
text_detections = []
|
||||
for recognize_item in data.get("recognize_list", []):
|
||||
item_list = recognize_item.get("item_content", {}).get("item_list", [])
|
||||
for item in item_list:
|
||||
content = item.get("content", "")
|
||||
confidence = int(item.get("conf", 0) * 100) # 0-1 → 0-100
|
||||
|
||||
vertex = item.get("vertex", {})
|
||||
if vertex:
|
||||
polygon = [
|
||||
{"X": vertex.get("x1", 0), "Y": vertex.get("y1", 0)},
|
||||
{"X": vertex.get("x2", 0), "Y": vertex.get("y2", 0)},
|
||||
{"X": vertex.get("x3", 0), "Y": vertex.get("y3", 0)},
|
||||
{"X": vertex.get("x4", 0), "Y": vertex.get("y4", 0)},
|
||||
]
|
||||
else:
|
||||
coord = item.get("coord", {})
|
||||
x, y = coord.get("x", 0), coord.get("y", 0)
|
||||
w, h = coord.get("width", 0), coord.get("height", 0)
|
||||
polygon = [
|
||||
{"X": x, "Y": y},
|
||||
{"X": x + w, "Y": y},
|
||||
{"X": x + w, "Y": y + h},
|
||||
{"X": x, "Y": y + h},
|
||||
]
|
||||
|
||||
text_detections.append({
|
||||
"DetectedText": content,
|
||||
"Confidence": confidence,
|
||||
"Polygon": polygon,
|
||||
})
|
||||
|
||||
return json.dumps({
|
||||
"TextDetections": text_detections,
|
||||
"Language": "zh",
|
||||
"RequestId": data.get("session_id", ""),
|
||||
}, ensure_ascii=False)
|
||||
|
||||
|
||||
def _get_ocr_call_params():
|
||||
"""从 config 读【单次OCR的重试上限/超时】, 带默认。
|
||||
封顶重试是为了防"卡1小时": 端点都超时时, 一张图最坏耗时 = max_attempts × timeout,
|
||||
而不是原来的 len(endpoints)(119) × 30s ≈ 3570s。"""
|
||||
try:
|
||||
cfg = config.get_tencent_ocr_config() or {}
|
||||
max_attempts = int(cfg.get("max_attempts", 4) or 4)
|
||||
timeout = int(cfg.get("timeout", 12) or 12)
|
||||
except (ValueError, TypeError, AttributeError):
|
||||
max_attempts, timeout = 4, 12
|
||||
return max(1, max_attempts), max(1, timeout)
|
||||
|
||||
|
||||
def youtu_ocr_api(image_base64: str) -> str:
|
||||
"""
|
||||
调用内部 Youtu OCR 服务进行文字识别。
|
||||
按 QPS 权重自动在多个 OCR IP 间分发；某端点失败时摘除冷却并换下一个端点重试。
|
||||
重试次数封顶(config: ocr.max_attempts), 单次超时可配(ocr.timeout), 避免端点全超时时拖死 worker。
|
||||
"""
|
||||
dispatcher = get_dispatcher()
|
||||
max_attempts, timeout = _get_ocr_call_params()
|
||||
# 最多试 max_attempts 个端点(且不超过端点总数), 而不是把 119 个全试一遍
|
||||
attempts = min(max(1, len(dispatcher.endpoints)), max_attempts)
|
||||
last_err = None
|
||||
for _ in range(attempts):
|
||||
url = dispatcher.pick()
|
||||
t0 = time.time()
|
||||
try:
|
||||
raw = ocr_request_once(url, image_base64, timeout=timeout)
|
||||
# 把真实延迟反馈给分发器 → 慢端点延迟EWMA升高 → 后续自动少分(负载感知核心)
|
||||
dispatcher.done(url, latency=time.time() - t0, ok=True)
|
||||
return _normalize_youtu_to_tencent(raw)
|
||||
except requests.exceptions.RequestException as e:
|
||||
last_err = e
|
||||
dispatcher.done(url, latency=time.time() - t0, ok=False)
|
||||
continue
|
||||
print(f"Youtu OCR 请求失败（已尝试 {attempts} 个端点, 单次超时 {timeout}s）: {last_err}")
|
||||
return ""
|
||||
|
||||
|
||||
# 为了保持原函数名兼容，将新接口赋值给原名称
|
||||
tencentcloud_ocr_api = youtu_ocr_api
|
||||
|
||||
|
||||
def tencent_ocr_video_all_frames(pic_dir_path: str,
|
||||
jsonl_save_path: str = 'frames_ocr.jsonl') -> list:
|
||||
"""
|
||||
对一个视频抽帧后的全部图片进行 ocr处理并保存起来
|
||||
"""
|
||||
with open(jsonl_save_path, "w", encoding='utf-8') as f:
|
||||
for root, _, files in os.walk(pic_dir_path):
|
||||
for pic_name in sorted(files):
|
||||
if pic_name.endswith(('jpg')):
|
||||
pic_path = os.path.join(root, pic_name)
|
||||
img_base64 = img2base64(img_path=pic_path)
|
||||
json_string = tencentcloud_ocr_api(image_base64=img_base64)
|
||||
f.write("pic_path: {}\n".format(pic_path))
|
||||
f.write(json_string + "\n")
|
||||
0
tool/VideoProcessing/__init__.py
Normal file
0
tool/VideoProcessing/__init__.py
Normal file
480
tool/VideoProcessing/multithread_algo.py
Normal file
480
tool/VideoProcessing/multithread_algo.py
Normal file
@@ -0,0 +1,480 @@
|
||||
from tool.log.log_manager import logger
|
||||
from config.config_utils import config, get_ffmpeg, get_ffprobe
|
||||
import shutil
|
||||
import subprocess
|
||||
import json
|
||||
import cv2
|
||||
import os
|
||||
import itertools
|
||||
import threading
|
||||
import time
|
||||
|
||||
|
||||
def extract_frames(
|
||||
video_path,
|
||||
video_name,
|
||||
start_frame,
|
||||
end_frame,
|
||||
output_dir,
|
||||
frames_to_extract,
|
||||
frame_prefix=""
|
||||
):
|
||||
"""
|
||||
进行抽帧
|
||||
:param video_path: 视频地址
|
||||
:param video_name: 视频名称
|
||||
:param start_frame: 开始的帧
|
||||
:param end_frame: 结束的帧
|
||||
:param output_dir: 输出地址
|
||||
:param frames_to_extract: 帧时间列表
|
||||
:param frame_prefix:
|
||||
:return: 抽取的帧数
|
||||
"""
|
||||
cap = cv2.VideoCapture(video_path)
|
||||
try:
|
||||
extracted_count = 0
|
||||
for current_time in frames_to_extract:
|
||||
# 只抽取在当前线程范围内且预定义的帧
|
||||
if start_frame <= current_time < end_frame:
|
||||
current_ms = 1000 * current_time
|
||||
cap.set(cv2.CAP_PROP_POS_MSEC, current_ms)
|
||||
ret, frame = cap.read()
|
||||
if not ret:
|
||||
print(f"{current_time}无ret")
|
||||
break
|
||||
frame_filename = os.path.join(
|
||||
output_dir,
|
||||
f"{video_name}{frame_prefix}_{3 * int(current_time) + int((current_time - int(current_time) + 0.1) / 0.33)}.jpg"
|
||||
)
|
||||
cv2.imwrite(frame_filename, frame)
|
||||
extracted_count += 1
|
||||
logger.log(f"线程 {start_frame}-{end_frame}: 抽取了{extracted_count}帧", program="split_frame")
|
||||
return extracted_count
|
||||
except Exception as e:
|
||||
logger.log(f"线程 {start_frame}-{end_frame} 抽帧异常: {e}", level="error", program="split_frame")
|
||||
raise
|
||||
finally:
|
||||
cap.release()
|
||||
|
||||
|
||||
def _run_extract_frames(result_container, index, *args, **kwargs):
|
||||
"""包装函数，捕获子线程异常并存储到共享容器"""
|
||||
try:
|
||||
result_container[index] = extract_frames(*args, **kwargs)
|
||||
except Exception as e:
|
||||
result_container[index] = e
|
||||
|
||||
|
||||
def split_video_get_frame_images(
|
||||
video_path,
|
||||
video_name,
|
||||
num_segments,
|
||||
output_dir,
|
||||
interval_seconds=1.0
|
||||
):
|
||||
"""
|
||||
拆分视频，获取视频的帧图片
|
||||
:param video_path:
|
||||
:param video_name:
|
||||
:param num_segments:
|
||||
:param output_dir:
|
||||
:param interval_seconds:
|
||||
:return:
|
||||
"""
|
||||
cap = cv2.VideoCapture(video_path)
|
||||
total_frames = cap.get(cv2.CAP_PROP_FRAME_COUNT)
|
||||
cap_fps = cap.get(cv2.CAP_PROP_FPS)
|
||||
print("total frames:", total_frames)
|
||||
print("fps:", cap_fps)
|
||||
video_duration = total_frames / cap_fps
|
||||
cap.release()
|
||||
frames_to_extract = []
|
||||
if interval_seconds == 0.33:
|
||||
total_seconds = int(video_duration)
|
||||
for seconds in range(total_seconds + 1):
|
||||
for i in range(3):
|
||||
target_time = seconds + i * 0.33
|
||||
if target_time < total_frames:
|
||||
frames_to_extract.append(target_time)
|
||||
# 去重排序
|
||||
frames_to_extract = sorted(set(frames_to_extract))
|
||||
else:
|
||||
t = 0.0
|
||||
while t < video_duration:
|
||||
frames_to_extract.append(t)
|
||||
t += interval_seconds
|
||||
frames_to_extract = sorted(set(frames_to_extract))
|
||||
frames_to_extract = [tt for tt in frames_to_extract if tt < video_duration]
|
||||
expected_frames = len(frames_to_extract)
|
||||
logger.log(f"【视频统计】 视频时长：{video_duration:.2f}秒, 统一计算抽帧位置：{expected_frames}", program="split_frame")
|
||||
|
||||
# 分段处理
|
||||
frames_per_segment = video_duration // num_segments
|
||||
threads = []
|
||||
thread_results = {}
|
||||
|
||||
for i in range(num_segments):
|
||||
start_frame = frames_per_segment * i
|
||||
end_frame = (i + 1) * frames_per_segment if i < num_segments - 1 else video_duration
|
||||
thread = threading.Thread(
|
||||
target=_run_extract_frames,
|
||||
args=(thread_results, i, video_path, video_name, start_frame, end_frame, output_dir, frames_to_extract, "_first")
|
||||
)
|
||||
threads.append(thread)
|
||||
thread.start()
|
||||
|
||||
for thread in threads:
|
||||
thread.join()
|
||||
|
||||
# 检查子线程异常
|
||||
for i, result in thread_results.items():
|
||||
if isinstance(result, Exception):
|
||||
raise RuntimeError(f"线程 {i} 抽帧失败: {result}") from result
|
||||
|
||||
return expected_frames, video_duration
|
||||
|
||||
|
||||
def get_video_codec(video_path: str) -> str:
|
||||
"""通过 ffprobe 获取视频编码格式"""
|
||||
cmd = [
|
||||
get_ffprobe(), '-v', 'error', '-select_streams', 'v:0',
|
||||
'-show_entries', 'stream=codec_name',
|
||||
'-of', 'default=noprint_wrappers=1:nokey=1', video_path
|
||||
]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True)
|
||||
return result.stdout.strip().lower()
|
||||
|
||||
|
||||
# ==================== GPU 硬解检测（结果缓存，只查一次）====================
|
||||
_gpu_lock = threading.Lock()
|
||||
_gpu_checked = False
|
||||
_gpu_available = False
|
||||
_gpu_count = 0 # 检测到的 GPU 张数（多卡轮询用）
|
||||
_gpu_rr = itertools.count() # 多卡轮询计数器（next() 在 CPython 下原子）
|
||||
# GPU 无法解码的编码：首次失败后记下，后续该编码直接走 CPU。
|
||||
# 假设多卡同型号（同样支持/不支持某编码），故按编码全局标记。
|
||||
_gpu_bad_codecs = set()
|
||||
|
||||
|
||||
def gpu_decode_available() -> bool:
|
||||
"""检测本机是否有可用的 NVIDIA GPU 硬解（nvidia-smi 可用 + ffmpeg 支持 cuda），
|
||||
并数出 GPU 张数。结果缓存，整个进程只真正检测一次。"""
|
||||
global _gpu_checked, _gpu_available, _gpu_count
|
||||
if _gpu_checked:
|
||||
return _gpu_available
|
||||
with _gpu_lock:
|
||||
if _gpu_checked:
|
||||
return _gpu_available
|
||||
avail = False
|
||||
count = 0
|
||||
try:
|
||||
r = subprocess.run(['nvidia-smi', '-L'], capture_output=True, text=True, timeout=10)
|
||||
if r.returncode == 0:
|
||||
count = sum(1 for ln in r.stdout.splitlines() if ln.strip().startswith('GPU '))
|
||||
r2 = subprocess.run([get_ffmpeg(), '-hide_banner', '-hwaccels'],
|
||||
capture_output=True, text=True, timeout=10)
|
||||
avail = count > 0 and 'cuda' in (r2.stdout or '').lower()
|
||||
except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
|
||||
avail, count = False, 0
|
||||
_gpu_available = avail
|
||||
_gpu_count = count
|
||||
_gpu_checked = True
|
||||
logger.log(
|
||||
f"GPU 硬解检测: {'可用，抽帧优先用 GPU(cuda)' if avail else '不可用，使用 CPU 软解'}"
|
||||
f"（检测到 {count} 张卡）", program="split_frame")
|
||||
return _gpu_available
|
||||
|
||||
|
||||
# —— 自动探测空闲 GPU（避开被 co-located OCR 服务占用的卡）——
|
||||
_gpu_free_cache = {"ts": 0.0, "free": None} # free=空闲卡索引集合; None=查不到(让上层回退)
|
||||
_gpu_free_lock = threading.Lock()
|
||||
|
||||
|
||||
def _query_gpu_mem_used() -> dict:
|
||||
"""nvidia-smi 查每张卡【已用显存(MB)】, 返回 {index: mem_mb}; 失败返回 {}。"""
|
||||
try:
|
||||
r = subprocess.run(
|
||||
['nvidia-smi', '--query-gpu=index,memory.used',
|
||||
'--format=csv,noheader,nounits'],
|
||||
capture_output=True, text=True, timeout=5)
|
||||
out = {}
|
||||
for ln in r.stdout.splitlines():
|
||||
parts = [p.strip() for p in ln.split(',')]
|
||||
if len(parts) >= 2 and parts[0].isdigit():
|
||||
try:
|
||||
out[int(parts[0])] = float(parts[1])
|
||||
except ValueError:
|
||||
pass
|
||||
return out
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
|
||||
def _get_free_gpus(busy_mem_mb: float, ttl: float = 20.0):
|
||||
"""返回【空闲】(已用显存 < busy_mem_mb)的 GPU 索引集合; 带 TTL 缓存, 不频繁调 nvidia-smi。
|
||||
返回 None 表示查不到(nvidia-smi 失败), 让上层回退到原有逻辑。"""
|
||||
now = time.time()
|
||||
with _gpu_free_lock:
|
||||
if _gpu_free_cache["free"] is not None and (now - _gpu_free_cache["ts"]) < ttl:
|
||||
return _gpu_free_cache["free"]
|
||||
mem = _query_gpu_mem_used()
|
||||
free = {i for i, m in mem.items() if m < busy_mem_mb} if mem else None
|
||||
_gpu_free_cache["free"] = free
|
||||
_gpu_free_cache["ts"] = now
|
||||
return free
|
||||
|
||||
|
||||
def _next_gpu_device() -> int:
|
||||
"""多卡轮询取下一张 GPU 索引；单卡或未知时返回 0。
|
||||
- gpu_reserve_last_for_whisper: 留最后一张给 whisper（抽帧只用前 N-1 张）
|
||||
- gpu_skip_indices: 手动【跳过】这些卡
|
||||
- gpu_auto_select: 自动探测, 只往【空闲卡】(显存占用 < gpu_busy_mem_mb)上分配,
|
||||
自动避开被 OCR 服务占着、抽帧慢25倍的卡。查不到/全忙时回退正常轮询。"""
|
||||
if _gpu_count <= 1:
|
||||
return 0
|
||||
n = _gpu_count
|
||||
skip = set()
|
||||
auto = False
|
||||
busy_mem = 4096.0
|
||||
try:
|
||||
cfg = config.get_splitframe_config()
|
||||
if cfg.get('gpu_reserve_last_for_whisper', True):
|
||||
n = _gpu_count - 1 # 留最后一张给 whisper
|
||||
for i in (cfg.get('gpu_skip_indices') or []):
|
||||
try:
|
||||
skip.add(int(i))
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
auto = bool(cfg.get('gpu_auto_select', False))
|
||||
busy_mem = float(cfg.get('gpu_busy_mem_mb', 4096))
|
||||
except Exception:
|
||||
pass
|
||||
# 候选 = 前 n 张里去掉手动跳过的卡
|
||||
candidates = [i for i in range(max(1, n)) if i not in skip]
|
||||
# 自动探测: 进一步只保留空闲卡(若探测到至少一张空闲); 全忙或查不到则维持原候选
|
||||
if auto:
|
||||
free = _get_free_gpus(busy_mem)
|
||||
if free:
|
||||
free_cands = [i for i in candidates if i in free]
|
||||
if free_cands:
|
||||
candidates = free_cands
|
||||
if not candidates:
|
||||
candidates = list(range(max(1, n))) # 兜底, 不至于不抽帧
|
||||
return candidates[next(_gpu_rr) % len(candidates)]
|
||||
|
||||
|
||||
def _clean_temp(temp_pattern: str):
|
||||
"""清理某次抽帧产生的临时帧（GPU 失败回退前用）。"""
|
||||
d = os.path.dirname(temp_pattern)
|
||||
prefix = os.path.basename(temp_pattern).split('%')[0]
|
||||
try:
|
||||
for f in os.listdir(d):
|
||||
if f.startswith(prefix):
|
||||
try:
|
||||
os.remove(os.path.join(d, f))
|
||||
except OSError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _resolve_ffmpeg_threads(raw) -> int:
|
||||
"""splitframe.ffmpeg_threads 支持字面数字，或 "auto"：按 CPU核数/并发进程数 自动摊分，
|
||||
避免 30 台核数不一样的机器都得手改这个数。并发进程数从环境变量 OCR_MAX_WORKERS 读
|
||||
(main.py/run_worker.py 启动时按实际生效的 --max-workers 写入，进程池 forkserver 子进程会继承)；
|
||||
读不到(如手动 import 调试)按 4 估算。0/未配置 = 不限制(交给 ffmpeg 默认行为)。"""
|
||||
if isinstance(raw, str) and raw.strip().lower() == 'auto':
|
||||
cpu = os.cpu_count() or 4
|
||||
try:
|
||||
workers = int(os.environ.get('OCR_MAX_WORKERS', '0') or 0)
|
||||
except ValueError:
|
||||
workers = 0
|
||||
workers = workers if workers > 0 else 4
|
||||
return max(1, cpu // workers)
|
||||
try:
|
||||
return int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return 0
|
||||
|
||||
|
||||
def _run_ffmpeg_extract(video_path: str, fps: float, temp_pattern: str, codec: str) -> str:
|
||||
"""执行抽帧：有 GPU 且该编码可硬解则用 GPU，失败/不支持自动回退 CPU。返回 'gpu' 或 'cpu'。"""
|
||||
cfg = config.get_splitframe_config()
|
||||
use_gpu = cfg.get('use_gpu', True)
|
||||
# 明确不让 GPU 解的编码(如 V100 解不了 AV1 → 配 ["av1"] 直接走 CPU，省掉一次失败探测)
|
||||
skip_codecs = {str(c).lower() for c in (cfg.get('gpu_skip_codecs') or [])}
|
||||
ffmpeg_threads = _resolve_ffmpeg_threads(cfg.get('ffmpeg_threads', 0))
|
||||
|
||||
# 字幕固定在下半屏时，抽帧阶段就裁掉上半部，帧变小 → 下游每帧处理(merge读帧/CV/OCR)成本按比例下降。
|
||||
# crop_bottom_ratio=保留底部比例(如 0.55=保留下 55%)；null/缺省=不裁(全帧)。
|
||||
vf = f'fps={fps:.6f}'
|
||||
crop_r = cfg.get('crop_bottom_ratio')
|
||||
try:
|
||||
crop_r = float(crop_r) if crop_r is not None else None
|
||||
except (TypeError, ValueError):
|
||||
crop_r = None
|
||||
if crop_r is not None and 0.0 < crop_r < 1.0:
|
||||
vf = f'{vf},crop=iw:ih*{crop_r:.4f}:0:ih*{1.0 - crop_r:.4f}'
|
||||
out_opts = ['-vf', vf, '-q:v', '2', '-y', temp_pattern, '-loglevel', 'error']
|
||||
|
||||
# ① GPU 硬解：支持该编码的卡上用 GPU 解码（多卡轮询分配到不同卡）。
|
||||
# 解码在 GPU、自动下载到内存，fps/JPEG 编码在 CPU。
|
||||
# 跳过条件：配置黑名单(gpu_skip_codecs，如 av1) 或 之前解失败过的编码(_gpu_bad_codecs)。
|
||||
# 其他能解的编码（如 H264/HEVC）走 GPU；某编码首次解失败也会记下、之后转 CPU。
|
||||
# ffmpeg 抽帧超时(秒): 防止坏视频/网络读卡死时 worker 永久阻塞。默认1小时,够超长视频抽帧。
|
||||
try:
|
||||
ff_timeout = int(cfg.get('ffmpeg_timeout', 3600))
|
||||
except (TypeError, ValueError):
|
||||
ff_timeout = 3600
|
||||
|
||||
if (use_gpu and codec not in skip_codecs and codec not in _gpu_bad_codecs
|
||||
and gpu_decode_available()):
|
||||
dev = _next_gpu_device()
|
||||
gpu_cmd = [get_ffmpeg()]
|
||||
if ffmpeg_threads > 0:
|
||||
gpu_cmd += ['-threads', str(ffmpeg_threads)]
|
||||
gpu_cmd += ['-hwaccel', 'cuda', '-hwaccel_device', str(dev),
|
||||
'-i', video_path] + out_opts
|
||||
try:
|
||||
subprocess.run(gpu_cmd, check=True, capture_output=True, timeout=ff_timeout)
|
||||
return f'gpu{dev}'
|
||||
except subprocess.TimeoutExpired:
|
||||
logger.log(f"GPU 抽帧超时(>{ff_timeout}s)，回退 CPU: {os.path.basename(video_path)}",
|
||||
level="warning", program="split_frame")
|
||||
_clean_temp(temp_pattern)
|
||||
except subprocess.CalledProcessError as e:
|
||||
with _gpu_lock:
|
||||
_gpu_bad_codecs.add(codec)
|
||||
err = (e.stderr or b'')
|
||||
err = err.decode(errors='ignore') if isinstance(err, bytes) else str(err)
|
||||
logger.log(f"GPU 硬解失败(编码 {codec or '未知'})，该编码后续改用 CPU: {err[:200]}",
|
||||
level="warning", program="split_frame")
|
||||
_clean_temp(temp_pattern) # 清掉半成品再回退
|
||||
|
||||
# ② CPU 软解（限解码线程，避免多视频并发互抢）
|
||||
cmd = [get_ffmpeg()]
|
||||
if ffmpeg_threads > 0:
|
||||
cmd += ['-threads', str(ffmpeg_threads)]
|
||||
cmd += ['-i', video_path] + out_opts
|
||||
# 捕获 stderr：失败时把 ffmpeg 真实报错带出来(否则只看到 exit 255 不知道原因)
|
||||
try:
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=ff_timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
_clean_temp(temp_pattern)
|
||||
raise RuntimeError(f"ffmpeg(cpu) 抽帧超时(>{ff_timeout}s): {os.path.basename(video_path)}")
|
||||
if r.returncode != 0:
|
||||
raise RuntimeError(
|
||||
f"ffmpeg(cpu) 退出码 {r.returncode}: {(r.stderr or '').strip()[-600:]}")
|
||||
return 'cpu'
|
||||
|
||||
|
||||
def split_frame_ffmpeg(video_path: str, video_name: str, output_dir: str,
|
||||
interval_seconds: float) -> tuple:
|
||||
"""
|
||||
使用 FFmpeg 提取帧（有 GPU 则硬解加速，否则软解；用于 AV1 等 OpenCV 无法解码的格式）。
|
||||
输出文件命名与 OpenCV 方案保持一致。
|
||||
"""
|
||||
# 获取视频时长与编码格式
|
||||
probe = subprocess.run([
|
||||
get_ffprobe(), '-v', 'error', '-select_streams', 'v:0',
|
||||
'-show_entries', 'stream=duration,codec_name',
|
||||
'-of', 'json', video_path
|
||||
], capture_output=True, text=True)
|
||||
# ffprobe 失败(返回码非0或无输出)时给清楚的错，避免后面 json.loads("") 报看不懂的
|
||||
# "Expecting value"。常见原因：bin/ffprobe 是 GPU 版但本机 glibc 太老(<2.28)跑不起来。
|
||||
if probe.returncode != 0 or not (probe.stdout or "").strip():
|
||||
raise RuntimeError(
|
||||
f"ffprobe 执行失败(返回码 {probe.returncode})；"
|
||||
f"常见原因：bin/ffprobe 与本机不兼容(如 GPU 版 ffmpeg 需 glibc>=2.28)。"
|
||||
f"stderr: {(probe.stderr or '').strip()[:300]}")
|
||||
info = json.loads(probe.stdout)
|
||||
stream0 = info['streams'][0] if info.get('streams') else {}
|
||||
video_duration = float(stream0.get('duration', 0) or 0)
|
||||
codec = (stream0.get('codec_name') or '').lower()
|
||||
# 流时长缺失(某些封装(尤其AV1/部分mp4)只在容器层有duration, 流里没有)时兜底查容器时长——
|
||||
# 不然 video_duration=0 会导致 frames_to_extract 算成空列表(下面 for 循环整个不执行), 但
|
||||
# 实际 ffmpeg 硬解还是会正常跑完整个视频(耗时不受这个影响), 抽完的帧因为"期望帧数=0"在
|
||||
# 后面的重命名循环里被当成多余的全部删掉 —— 表现为"抽帧完成(0帧, 耗时正常, 视频0s)",
|
||||
# 花了几分钟GPU时间白干。跟 tool/process.py 的 _probe_duration() 是同一个坑、同一个修法。
|
||||
if video_duration <= 0:
|
||||
try:
|
||||
fmt_probe = subprocess.run([
|
||||
get_ffprobe(), '-v', 'error',
|
||||
'-show_entries', 'format=duration', '-of', 'json', video_path
|
||||
], capture_output=True, text=True)
|
||||
fmt_info = json.loads(fmt_probe.stdout or '{}')
|
||||
video_duration = float((fmt_info.get('format') or {}).get('duration', 0) or 0)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 计算目标时间点（与 OpenCV 方案一致）
|
||||
frames_to_extract = []
|
||||
if interval_seconds == 0.33:
|
||||
for sec in range(int(video_duration) + 1):
|
||||
for i in range(3):
|
||||
t = sec + i * 0.33
|
||||
if t < video_duration:
|
||||
frames_to_extract.append(t)
|
||||
else:
|
||||
t = 0.0
|
||||
while t < video_duration:
|
||||
frames_to_extract.append(t)
|
||||
t += interval_seconds
|
||||
frames_to_extract = sorted(set(frames_to_extract))
|
||||
expected = len(frames_to_extract)
|
||||
|
||||
# 一次性用 FFmpeg 提取所有帧（fps 滤镜）
|
||||
# 注意：interval 0.33 是"每秒3帧"的约定值，真正频率应为 3.0。
|
||||
# 若用 1/0.33=3.0303，会比真正 3fps 快约1%，导致第N帧实际抽自 N×0.33 秒、
|
||||
# 而 SRT 按 N/3≈N×0.3333 秒标注，产生随帧号累积的时间漂移（18分钟处可差~10秒）。
|
||||
fps = 3.0 if interval_seconds == 0.33 else 1.0 / interval_seconds
|
||||
temp_pattern = os.path.join(output_dir, f"__tmp_{video_name}_%08d.jpg")
|
||||
|
||||
# 有 GPU 用 GPU 硬解，否则 CPU 软解（CPU 路径按 ffmpeg_threads 限解码线程）
|
||||
backend = _run_ffmpeg_extract(video_path, fps, temp_pattern, codec)
|
||||
|
||||
# 按期望时间点重命名
|
||||
temp_files = sorted(
|
||||
f for f in os.listdir(output_dir) if f.startswith(f"__tmp_{video_name}_")
|
||||
)
|
||||
extracted = 0
|
||||
for i, tmp in enumerate(temp_files):
|
||||
if i >= len(frames_to_extract):
|
||||
os.remove(os.path.join(output_dir, tmp))
|
||||
continue
|
||||
ct = frames_to_extract[i]
|
||||
frame_idx = 3 * int(ct) + int((ct - int(ct) + 0.1) / 0.33)
|
||||
new_name = os.path.join(output_dir, f"{video_name}_first_{frame_idx}.jpg")
|
||||
os.rename(os.path.join(output_dir, tmp), new_name)
|
||||
extracted += 1
|
||||
|
||||
logger.log(f"FFmpeg 抽帧完成({backend}): 应抽{expected}帧, 实际{extracted}帧", program="split_frame")
|
||||
return expected, video_duration, backend
|
||||
|
||||
|
||||
def split_frame(video_path, extract_interval=0.33):
|
||||
"""
|
||||
进行视频分割
|
||||
:param video_path: 视频本地位置
|
||||
:param extract_interval: 分割间隔，默认0.33间隔，1秒3张图
|
||||
:return:
|
||||
"""
|
||||
logger.start_program("split_frame")
|
||||
video_name = os.path.basename(video_path).partition(".")[0]
|
||||
to_save_dir = os.path.join(os.getcwd(), "split", video_name)
|
||||
os.makedirs(to_save_dir, exist_ok=True)
|
||||
|
||||
logger.log(f"【 正在抽帧处理 】{video_name}, 抽帧间隔：{extract_interval}秒", program="split_frame")
|
||||
start = time.time()
|
||||
|
||||
# 统一使用 FFmpeg 单次顺序解码（比 OpenCV 逐帧 seek 快 3-5 倍）
|
||||
expected_frames, video_duration, backend = split_frame_ffmpeg(
|
||||
video_path, video_name, to_save_dir, extract_interval)
|
||||
|
||||
end = time.time()
|
||||
|
||||
actual_frames = len([f for f in os.listdir(to_save_dir) if f.endswith('.jpg')])
|
||||
logger.log(f"【抽帧完成】: 应抽{expected_frames}帧, 实际抽取{actual_frames}帧, 耗时{end - start:.2f}秒", program="split_frame")
|
||||
logger.end_program("split_frame")
|
||||
return video_duration, expected_frames, actual_frames, backend
|
||||
|
||||
0
tool/__init__.py
Normal file
0
tool/__init__.py
Normal file
BIN
tool/__pycache__/__init__.cpython-311.pyc
Normal file
BIN
tool/__pycache__/__init__.cpython-311.pyc
Normal file
Binary file not shown.
BIN
tool/__pycache__/job_cos.cpython-311.pyc
Normal file
BIN
tool/__pycache__/job_cos.cpython-311.pyc
Normal file
Binary file not shown.
2024
tool/getsrt.py
Normal file
2024
tool/getsrt.py
Normal file
File diff suppressed because it is too large
Load Diff
814
tool/job_cos.py
Normal file
814
tool/job_cos.py
Normal file
@@ -0,0 +1,814 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""COS 任务分发 / 成功记录采集。
|
||||
|
||||
核心原语: COS PutObject 带 ForbidOverwrite(x-cos-forbid-overwrite:true) —— 同名对象
|
||||
只能被创建一次, 并发写同一个锁文件只有第一个成功(其余 409)。以此实现跨机器原子认领,
|
||||
无需任何额外服务(Redis/DB)。
|
||||
|
||||
策略: 粘性认领(sticky)。认领某分片的机器是它的唯一所有者;机器重启后只续做自己名下
|
||||
未完成的分片, 别的机器永不抢占 -> 零重复。机器永久报废时, 人工跑 tools/release_machine.py
|
||||
把它名下的任务放回队列。
|
||||
|
||||
COS 布局(jobs_root 下):
|
||||
todo/<shard>.txt 待处理分片(每行一个视频 COS 路径)。完成才删, 是"任务存在性"的唯一凭证。
|
||||
claimed/<shard>.lock 认领锁, 内容 JSON {machine_id, host, claim_ts, hb_ts}。
|
||||
done/<shard>.done 完成标记, 内容 JSON {machine_id, stats, end_ts}。
|
||||
独立位置(各自 config 字段, 可指向别处):
|
||||
daily_backup_path/<日期>/数据_<机器号>.jsonl 每日 数据.jsonl 全量快照备份。
|
||||
daily_backup_path/<日期>/已处理_<机器号>.txt 上面jsonl的轻量版(只有video_cos_path,
|
||||
一行一个), 供 global_processed_set() 全局去重用。
|
||||
long_list_backup_path/todo/超长待重跑_<机器号>_<日期>_<时分秒>.txt 超长视频重跑队列(本身就是另一套 jobs_root)。
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
from qcloud_cos.cos_exception import CosServiceError
|
||||
|
||||
from config.config_utils import config
|
||||
|
||||
# 老版本 cos-python-sdk-v5 的 put_object 不认识 ForbidOverwrite 参数(其 maplist 没有该映射),
|
||||
# 直接传会报 "No Parameter Named ForbidOverwrite"。这里把映射补进去, 使 ForbidOverwrite='true'
|
||||
# 注入 x-cos-forbid-overwrite:true 请求头 —— 认领锁的原子性(同名对象只能创建一次)全靠它。
|
||||
# 不升级 SDK(线上离线机不便升级), 版本无关, 幂等。
|
||||
try:
|
||||
from qcloud_cos import cos_comm as _cos_comm
|
||||
_cos_comm.maplist.setdefault("ForbidOverwrite", "x-cos-forbid-overwrite")
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# ---------------------------------------------------------------- 客户端 ----
|
||||
|
||||
_CLIENT = None
|
||||
|
||||
|
||||
def client():
|
||||
"""模块级单例 CosS3Client(复用项目现有 COS 凭据)。"""
|
||||
global _CLIENT
|
||||
if _CLIENT is None:
|
||||
c = config.get_cos_config()
|
||||
_CLIENT = CosS3Client(CosConfig(
|
||||
Region=c["region"], SecretId=c["secret_id"], SecretKey=c["secret_key"],
|
||||
))
|
||||
return _CLIENT
|
||||
|
||||
|
||||
def parse_cos_url(url: str):
|
||||
"""cos://bucket/a/b -> ('bucket', 'a/b')。无 cos:// 前缀时按 bucket 配置补全。"""
|
||||
url = url.strip()
|
||||
if url.startswith("cos://"):
|
||||
rest = url[len("cos://"):]
|
||||
bucket, _, key = rest.partition("/")
|
||||
return bucket, key.strip("/")
|
||||
return config.get_cos_config()["bucket"], url.strip("/")
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- 配置 ----
|
||||
|
||||
def _cfg():
|
||||
return config.get_dispatch_config() or {}
|
||||
|
||||
|
||||
def _detect_ip() -> str:
|
||||
"""取本机对外默认路由的网卡IP:用 UDP socket connect 一个外部地址触发内核选路由
|
||||
(不会真的发包), 拿到的本地地址天然跳过 docker 网桥(如172.17.0.1)/回环这类非默认路由
|
||||
接口 —— 通常就是 SSH 连进来那张网卡的IP(前提:机器只有一条默认路由出口, 常见的
|
||||
'一张主网卡+docker网桥'部署; 若有多张真实网卡且SSH走的不是默认路由那张, 不适用,
|
||||
建议改用 `ip route get 8.8.8.8` 在目标机上核实后再决定)。
|
||||
取不到再退化到 `hostname -I` 第一个; 都拿不到返回空串, 上层回退主机名。
|
||||
仅在 IP 固定(非 DHCP)的机器上用它兜底身份才安全。"""
|
||||
try:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
try:
|
||||
s.connect(("8.8.8.8", 80))
|
||||
ip = s.getsockname()[0]
|
||||
finally:
|
||||
s.close()
|
||||
if ip and not ip.startswith("127."):
|
||||
return ip
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
r = subprocess.run(['hostname', '-I'], capture_output=True, text=True, timeout=5)
|
||||
ips = (r.stdout or '').split()
|
||||
return ips[0] if ips else ''
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return ''
|
||||
|
||||
|
||||
def get_machine_id() -> str:
|
||||
"""机器ID 取值优先级: config.dispatch.machine_id > 环境变量 MACHINE_ID >
|
||||
本机固定IP(默认路由网卡, 见 _detect_ip) > 主机名。
|
||||
必须【唯一且重启后稳定】—— 锁靠它判断'这片是不是我的'。克隆机场景下主机名常常相同,
|
||||
固定IP各机器天然不同,可以直接兜底;但如果 IP 是 DHCP 会变的,不要依赖这个,
|
||||
换IP等于换身份,旧身份名下未完成的锁会变孤儿——那种环境请在 config 里显式填 machine_id。"""
|
||||
mid = (_cfg().get("machine_id") or "").strip()
|
||||
if mid:
|
||||
return mid
|
||||
mid = (os.environ.get("MACHINE_ID") or "").strip()
|
||||
if mid:
|
||||
return mid
|
||||
ip = _detect_ip()
|
||||
if ip:
|
||||
return ip
|
||||
return socket.gethostname()
|
||||
|
||||
|
||||
def is_ocr_colocated() -> bool:
|
||||
"""本机是不是 OCR 服务共置机: 本机默认路由IP是否出现在 config.tencentcloud.ocr.endpoints
|
||||
列表里。这批端点IP本来就是各台跑OCR服务的机器, 如果本机自己也在这个列表里, 说明这台机器
|
||||
除了跑抽帧/OCR客户端之外, 本身还扛着一个OCR服务, 需要多留CPU给它, 不能被抽帧吃满。
|
||||
只覆盖"OCR端点=本机IP"这一种共置方式; 其它不出现在这个列表里的共享方式(比如机器上还跑
|
||||
了别的无关服务)探测不到, 那种情况还是得手动传小一点的 --max-workers。"""
|
||||
try:
|
||||
endpoints = set((config.get_tencent_ocr_config() or {}).get("endpoints") or [])
|
||||
except Exception:
|
||||
return False
|
||||
if not endpoints:
|
||||
return False
|
||||
ip = _detect_ip()
|
||||
return bool(ip) and ip in endpoints
|
||||
|
||||
|
||||
MAX_AUTO_WORKERS = 30 # auto_max_workers() 的硬上限, 见函数docstring③
|
||||
|
||||
|
||||
def auto_max_workers(cpu: int, disk_guard_gb: float = 0.0) -> int:
|
||||
"""--max-workers 未指定时的自动选择:
|
||||
① 默认按CPU核数; 如果 is_ocr_colocated() 判断本机还跑着OCR服务, 打对折让出一半CPU。
|
||||
② 再按本地磁盘算一次上限, 取①②里更小的那个 —— 核多盘小的机器(比如高核数但磁盘只有
|
||||
100多G)按纯CPU数选出的并发数,同时下载/抽帧/拼合成图会把磁盘写爆;`--disk-guard-gb`只挡
|
||||
"新启动"挡不住"已经在跑的", 真正决定磁盘峰值占用的是并发数本身, 所以从源头把并发数卡住:
|
||||
磁盘上限 = (可用磁盘 - disk_guard_gb这层安全垫) ÷ 每视频预估峰值占用(GB)
|
||||
每视频预估占用读 config.dispatch.est_disk_gb_per_video, 默认 2.0G —— 按 subtitle 配置常见的
|
||||
30~2400秒短视频估的(2400秒=3fps≈7200帧, 裁剪后的帧+合成图+源视频下载, 峰值同时存在时
|
||||
量级在1~2G上下; 视频普遍更短/分辨率更低可以调小, 更长/更高清要调大, 保守起见默认按上限估)。
|
||||
disk_guard_gb 传0(不用磁盘保护)时不再额外扣这层安全垫, 直接用当前可用磁盘算。
|
||||
③ 最后再套一个硬上限 MAX_AUTO_WORKERS(30) —— 超高核数机器(如64/128核)按纯CPU数选出的
|
||||
并发数会远超实际有意义的范围(单机瓶颈通常先卡在OCR接口/磁盘IO/内存, 而不是CPU), 且并发
|
||||
数太高会让 ffmpeg_threads="auto" 摊出来的每进程线程数过低(见 multithread_algo.py)。
|
||||
cpu<=0(测不到核数, 比如非Linux)时按10兜底。"""
|
||||
cpu = cpu if cpu and cpu > 0 else 10
|
||||
workers = cpu
|
||||
if is_ocr_colocated():
|
||||
workers = max(1, cpu // 2)
|
||||
logging.info(f"[资源探测] 本机IP在 tencentcloud.ocr.endpoints 里, 判断为OCR共置机, "
|
||||
f"max-workers 自动打对折让资源给OCR服务: {cpu}核 → {workers}")
|
||||
|
||||
try:
|
||||
free_gb = shutil.disk_usage(".").free / 1024 / 1024 / 1024
|
||||
budget_gb = free_gb - max(0.0, disk_guard_gb)
|
||||
est_gb = float(_cfg().get("est_disk_gb_per_video", 2.0) or 2.0)
|
||||
if budget_gb <= 0:
|
||||
logging.warning(f"[资源探测] 可用磁盘({free_gb:.1f}G)已经不够留给磁盘保护垫"
|
||||
f"({disk_guard_gb}G), max-workers 收紧到1, 建议先清理磁盘")
|
||||
workers = 1
|
||||
elif est_gb > 0:
|
||||
disk_cap = max(1, int(budget_gb / est_gb))
|
||||
if disk_cap < workers:
|
||||
logging.info(f"[资源探测] 按磁盘算的并发上限更小, max-workers 收紧: "
|
||||
f"{workers} → {disk_cap}(可用磁盘{free_gb:.1f}G - 保护垫{disk_guard_gb}G = "
|
||||
f"{budget_gb:.1f}G预算 ÷ 每视频预估{est_gb}G)")
|
||||
workers = disk_cap
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
if workers > MAX_AUTO_WORKERS:
|
||||
logging.info(f"[资源探测] max-workers 超过硬上限, 收紧: {workers} → {MAX_AUTO_WORKERS}")
|
||||
workers = MAX_AUTO_WORKERS
|
||||
return workers
|
||||
|
||||
|
||||
_root_override = {"value": None} # run_worker.py 用: 主队列空闲时临时切去认领长视频队列
|
||||
|
||||
|
||||
def set_active_root_override(root):
|
||||
"""临时覆盖 _active_jobs_root() 的返回值(仅当前进程内生效, 不改config/不用重启)。
|
||||
传 None 清除覆盖、恢复按 config 判断。给 run_worker.py 做'主队列没活时顺手兼跑长视频队列'用:
|
||||
覆盖生效期间, claim/list_todo/download_shard/mark_done/release 等全部原语都会转而操作
|
||||
覆盖指向的这个队列根, 跟专职长视频机(run_long_video=true)走的是同一套代码路径, 只是切换是
|
||||
动态的、按需的, 而不是启动时定死。"""
|
||||
_root_override["value"] = root
|
||||
|
||||
|
||||
def _active_jobs_root():
|
||||
"""本 worker 实际认领的队列根, 优先级:
|
||||
_root_override(运行时临时切换) > run_long_video=true(专跑超长视频, 用 long_list_backup_path)
|
||||
> jobs_root(默认主队列)。"""
|
||||
if _root_override["value"]:
|
||||
return _root_override["value"]
|
||||
c = _cfg()
|
||||
if c.get("run_long_video"):
|
||||
root = c.get("long_list_backup_path")
|
||||
if root:
|
||||
return root
|
||||
return c.get("jobs_root") or ""
|
||||
|
||||
|
||||
def _bucket_base():
|
||||
"""从【当前生效的队列根】解析 (bucket, base_prefix)。base 末尾无 '/'。
|
||||
只给 _key()(todo/claimed/done 分片路径)用——这个"当前生效"是有意跟着
|
||||
run_long_video/_root_override 变的(分片本来就该落在当前认领的那个队列下)。"""
|
||||
root = _active_jobs_root()
|
||||
if not root:
|
||||
raise RuntimeError("config.dispatch 未配置队列地址(jobs_root / long_list_backup_path)")
|
||||
bucket, key = parse_cos_url(root)
|
||||
return bucket, key.strip("/")
|
||||
|
||||
|
||||
def _primary_bucket_base():
|
||||
"""固定解析自 jobs_root(主队列), 不受 run_long_video / 运行时 _root_override 影响 ——
|
||||
daily_backup_path / monitor_root 没显式配置时的兜底基准必须用这个, 不能用 _bucket_base()
|
||||
(它会跟 _active_jobs_root() 一起变): 专职长视频机(run_long_video=true)跑起来后
|
||||
_active_jobs_root() 长期指向 long_list_backup_path, 如果兜底基准也跟着它变, 算出来的
|
||||
monitor_root/daily_backup_path 就会落到 .../long_video/monitor、.../long_video/daily 这种
|
||||
跟其它机器不一样的路径下 —— fleet_monitor.py 和 global_processed_set() 只认标准的
|
||||
monitor_root/daily_backup_path, 读不到这些"另起炉灶"的路径, 表现为这台机器的小时摘要/
|
||||
去重快照对外"消失了"(实际数据都在, 只是传错地方)。"""
|
||||
root = _cfg().get("jobs_root") or ""
|
||||
if not root:
|
||||
raise RuntimeError("config.dispatch.jobs_root 未配置")
|
||||
bucket, key = parse_cos_url(root)
|
||||
return bucket, key.strip("/")
|
||||
|
||||
|
||||
def _key(kind: str, shard: str = "") -> str:
|
||||
"""拼某类对象的 key。kind ∈ {todo, claimed, done}。"""
|
||||
_, base = _bucket_base()
|
||||
ext = {"todo": ".txt", "claimed": ".lock", "done": ".done"}[kind]
|
||||
if shard:
|
||||
return f"{base}/{kind}/{shard}{ext}"
|
||||
return f"{base}/{kind}/"
|
||||
|
||||
|
||||
# --------------------------------------------------------------- COS 原语 ----
|
||||
|
||||
def _put(key: str, data: bytes, forbid_overwrite: bool = False):
|
||||
bkt, _ = _bucket_base()
|
||||
kwargs = {}
|
||||
if forbid_overwrite:
|
||||
kwargs["ForbidOverwrite"] = "true" # 经 maplist 注入 x-cos-forbid-overwrite 头
|
||||
client().put_object(Bucket=bkt, Body=data, Key=key, **kwargs)
|
||||
|
||||
|
||||
def _get_bytes(key: str):
|
||||
bkt, _ = _bucket_base()
|
||||
resp = client().get_object(Bucket=bkt, Key=key)
|
||||
return resp["Body"].get_raw_stream().read()
|
||||
|
||||
|
||||
def _exists(key: str) -> bool:
|
||||
bkt, _ = _bucket_base()
|
||||
return client().object_exists(Bucket=bkt, Key=key)
|
||||
|
||||
|
||||
def _delete(key: str):
|
||||
bkt, _ = _bucket_base()
|
||||
try:
|
||||
client().delete_object(Bucket=bkt, Key=key)
|
||||
except CosServiceError as e:
|
||||
if e.get_status_code() != 404:
|
||||
raise
|
||||
|
||||
|
||||
def _list_shards(kind: str):
|
||||
"""列出某前缀下所有分片名(去掉前缀和扩展名)。"""
|
||||
bkt, _ = _bucket_base()
|
||||
prefix = _key(kind)
|
||||
ext = {"todo": ".txt", "claimed": ".lock", "done": ".done"}[kind]
|
||||
out, marker = [], ""
|
||||
while True:
|
||||
resp = client().list_objects(Bucket=bkt, Prefix=prefix, Marker=marker, MaxKeys=1000)
|
||||
for c in resp.get("Contents", []) or []:
|
||||
k = c["Key"]
|
||||
if k.endswith(ext):
|
||||
out.append(k[len(prefix):-len(ext)])
|
||||
if resp.get("IsTruncated") in ("true", True):
|
||||
marker = resp.get("NextMarker") or (resp["Contents"][-1]["Key"] if resp.get("Contents") else "")
|
||||
if not marker:
|
||||
break
|
||||
else:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
# ----------------------------------------------------------------- 接口 ----
|
||||
|
||||
def list_todo():
|
||||
return _list_shards("todo")
|
||||
|
||||
|
||||
def shard_matches(name: str, patterns) -> bool:
|
||||
"""分片名是否匹配 patterns(glob 通配, 如 'cos_urls_8*')。
|
||||
patterns 为空/None -> 认领全部(返回 True)。可传单个字符串或列表。
|
||||
匹配时对 '<名>' 和 '<名>.txt' 都试一遍, 写不写 .txt 都行。"""
|
||||
import fnmatch
|
||||
if not patterns:
|
||||
return True
|
||||
if isinstance(patterns, str):
|
||||
patterns = [patterns]
|
||||
cands = (name, name + ".txt")
|
||||
return any(fnmatch.fnmatch(c, p) for p in patterns for c in cands)
|
||||
|
||||
|
||||
def is_done(shard: str) -> bool:
|
||||
return _exists(_key("done", shard))
|
||||
|
||||
|
||||
def read_lock(shard: str):
|
||||
"""读认领锁内容(dict)。不存在返回 None。"""
|
||||
try:
|
||||
return json.loads(_get_bytes(_key("claimed", shard)).decode("utf-8"))
|
||||
except CosServiceError as e:
|
||||
if e.get_status_code() == 404:
|
||||
return None
|
||||
raise
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def claim(shard: str, machine_id: str) -> bool:
|
||||
"""原子认领: 写 claimed/<shard>.lock(禁止覆盖)。成功=True 表示这片归我;
|
||||
409(已被别人锁) -> False。"""
|
||||
now = datetime.now()
|
||||
body = json.dumps({
|
||||
"machine_id": machine_id, "host": socket.gethostname(),
|
||||
"claim_ts": now.isoformat(), "hb_ts": now.isoformat(),
|
||||
}, ensure_ascii=False).encode("utf-8")
|
||||
try:
|
||||
_put(_key("claimed", shard), body, forbid_overwrite=True)
|
||||
return True
|
||||
except CosServiceError as e:
|
||||
if e.get_status_code() == 409:
|
||||
return False # 已被别的机器抢到
|
||||
raise
|
||||
|
||||
|
||||
def heartbeat(shard: str, machine_id: str):
|
||||
"""刷新锁的心跳时间(允许覆盖, 仍写回 machine_id 以保持所有权信息)。"""
|
||||
lk = read_lock(shard) or {}
|
||||
lk.update({"machine_id": machine_id, "host": socket.gethostname(),
|
||||
"hb_ts": datetime.now().isoformat()})
|
||||
lk.setdefault("claim_ts", lk["hb_ts"])
|
||||
_put(_key("claimed", shard), json.dumps(lk, ensure_ascii=False).encode("utf-8"))
|
||||
|
||||
|
||||
def download_shard(shard: str):
|
||||
"""读 todo/<shard>.txt, 返回视频 COS 路径列表。"""
|
||||
text = _get_bytes(_key("todo", shard)).decode("utf-8")
|
||||
return [ln.strip() for ln in text.splitlines() if ln.strip()]
|
||||
|
||||
|
||||
def mark_done(shard: str, machine_id: str, stats: dict):
|
||||
_put(_key("done", shard), json.dumps({
|
||||
"machine_id": machine_id, "stats": stats,
|
||||
"end_ts": datetime.now().isoformat(),
|
||||
}, ensure_ascii=False).encode("utf-8"))
|
||||
|
||||
|
||||
def release(shard: str):
|
||||
"""跑完后清理: 删 todo 和 lock(保留 done 作为完成账)。"""
|
||||
_delete(_key("todo", shard))
|
||||
_delete(_key("claimed", shard))
|
||||
|
||||
|
||||
def my_open_claims(machine_id: str):
|
||||
"""我名下、尚未完成(无 done)的分片 —— 重启后续做用。"""
|
||||
out = []
|
||||
for shard in _list_shards("claimed"):
|
||||
lk = read_lock(shard)
|
||||
if lk and lk.get("machine_id") == machine_id and not is_done(shard):
|
||||
out.append(shard)
|
||||
return out
|
||||
|
||||
|
||||
# ------------------------------------------------------------- 优雅停机 ----
|
||||
|
||||
def stop_requested() -> bool:
|
||||
"""COS 上存在 stop_flag_key(默认 _jobs/STOP)则请求优雅停机。"""
|
||||
flag = _cfg().get("stop_flag_key")
|
||||
if not flag:
|
||||
_, base = _bucket_base()
|
||||
flag = f"{base}/STOP"
|
||||
bkt, _ = _bucket_base()
|
||||
try:
|
||||
return client().object_exists(Bucket=bkt, Key=flag.strip("/"))
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
# ----------------------------------------------------- 每日 数据.jsonl 备份 ----
|
||||
|
||||
# put_object 是"一次性读进内存 + 单个 HTTP 请求"上传, 底层 send_request 默认超时 30 秒 ——
|
||||
# 数据.jsonl 是全量快照, 产能高的机器跑几天就上百 MB/上 GB, 30 秒内传不完就
|
||||
# "The write operation timed out", 然后每几分钟重试又超时, 无限循环从没成功(实测 21.13.80.229)。
|
||||
# 超过这个大小的文件改用 coscli 上传(它支持分块/断点续传, 几 GB 也没问题); 小文件(如几KB的
|
||||
# 小时摘要)仍走 put_object —— 起 coscli 子进程有开销, 小文件没必要。
|
||||
_COSCLI_UPLOAD_THRESHOLD = 50 * 1024 * 1024 # 50MB
|
||||
|
||||
|
||||
def _coscli_put(local_path: str, cos_url: str) -> bool:
|
||||
"""用 coscli 把 local_path 上传到 cos_url(完整 cos:// 路径)。成功 True, 失败 False。
|
||||
coscli 支持分块上传, 大文件不像 put_object 那样受单请求超时限制。
|
||||
coscli 路径/配置从 splitframe 配置读(跟主流程上传 SRT 用的是同一个二进制和配置)。"""
|
||||
try:
|
||||
sf = config.get_splitframe_config() or {}
|
||||
coscli = sf.get("coscli_path", "./coscli-linux-amd64")
|
||||
coscfg = sf.get("cosconfig_path", ".cos.yaml")
|
||||
cmd = [coscli, "-c", coscfg, "cp", local_path, cos_url]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
|
||||
if r.returncode == 0:
|
||||
return True
|
||||
logging.warning(f"[备份] coscli 上传失败(将回退 put_object): {(r.stderr or '').strip()[:200]}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logging.warning(f"[备份] coscli 上传异常(将回退 put_object): {type(e).__name__}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def _upload_dated_file(local_path: str, dest_root: str, fname: str) -> str:
|
||||
"""把 local_path 上传到 dest_root/<日期>/fname。
|
||||
local_path 不存在 / dest_root 为空 -> 返回 ''。返回上传后的 cos:// 路径。
|
||||
大文件(> _COSCLI_UPLOAD_THRESHOLD)走 coscli 分块上传, 避免 put_object 30 秒超时;
|
||||
coscli 失败(二进制不在/权限等)自动回退 put_object 再试一次 —— 保证引入 coscli 不会
|
||||
比原来更差(最坏情况就是退回原行为)。"""
|
||||
if not local_path or not dest_root or not os.path.exists(local_path):
|
||||
return ""
|
||||
bkt, base_key = parse_cos_url(dest_root)
|
||||
date = datetime.now().strftime("%Y%m%d") # 按日期建文件夹
|
||||
key = f"{base_key.strip('/')}/{date}/{fname}"
|
||||
cos_url = f"cos://{bkt}/{key}"
|
||||
|
||||
size = os.path.getsize(local_path)
|
||||
if size > _COSCLI_UPLOAD_THRESHOLD:
|
||||
logging.info(f"[备份] {fname} 有 {size / 1024 / 1024:.0f}MB, 超过 "
|
||||
f"{_COSCLI_UPLOAD_THRESHOLD // 1024 // 1024}MB, 用 coscli 分块上传")
|
||||
if _coscli_put(local_path, cos_url):
|
||||
return cos_url
|
||||
# coscli 没成: 回退 put_object(大文件很可能还是超时, 但至少不比原来差, 且日志会记清楚)
|
||||
|
||||
with open(local_path, "rb") as f:
|
||||
data = f.read()
|
||||
client().put_object(Bucket=bkt, Body=data, Key=key)
|
||||
return cos_url
|
||||
|
||||
|
||||
def upload_daily_backup(jsonl_path: str, machine_id: str) -> str:
|
||||
"""把本地 数据.jsonl 上传到 dispatch.daily_backup_path/<日期>/数据_<机器号>.jsonl，
|
||||
顺带生成并上传一份轻量的"已处理清单"(见 upload_processed_list) 供 global_processed_set()
|
||||
用——两者是同一次备份动作的副产物，调这一个函数就够了。"""
|
||||
dest_root = _cfg().get("daily_backup_path")
|
||||
if not dest_root:
|
||||
bucket, base = _bucket_base()
|
||||
dest_root = f"cos://{bucket}/{base}/daily"
|
||||
dst = _upload_dated_file(jsonl_path, dest_root, f"数据_{machine_id}.jsonl")
|
||||
if dst:
|
||||
try:
|
||||
upload_processed_list(jsonl_path, machine_id)
|
||||
except Exception as e:
|
||||
logging.warning(f"[每日备份] 已处理清单上传失败(不影响 数据.jsonl 备份): {e}")
|
||||
return dst
|
||||
|
||||
|
||||
def upload_longlist_backup(machine_id: str) -> str:
|
||||
"""把本地超长待重跑列表(subtitle.long_video_list)上传到
|
||||
dispatch.long_list_backup_path/todo/<原文件名>_<机器号>_<日期>_<时分秒>.txt。
|
||||
放进 todo/ 子目录, 使其直接成为可认领的任务分片 —— 把别的机器的 jobs_root
|
||||
指向 long_list_backup_path, 它们就能自动认领这些超长视频重跑。
|
||||
文件名带【机器号+日期+时分秒】: 机器号保证各机器不互相覆盖; 时分秒是因为本函数是
|
||||
"传完即清空本地已传部分"的增量式设计(见下), 一天内不管自动备份触发几次、或人工额外
|
||||
手动跑几次(如 tools/upload_daily.py / apply_update.sh 更新前那一下), 每次快照的都只是
|
||||
"上次传完之后新增的那一小段" —— 如果 key 只精确到天, 同一天第二次上传会用 put_object
|
||||
直接覆盖同名对象, 把第一次已传但可能还没被任何机器认领处理的内容连本地带COS一起冲没。
|
||||
精确到秒后每次上传各自独立成对象, 不会互相覆盖, 只是分片数量会随调用次数变多(不影响使用,
|
||||
worker 一样能各自认领)。"""
|
||||
# 专跑长视频的机器(run_long_video)不再把超长列表传回同一队列, 防回环。
|
||||
if _cfg().get("run_long_video"):
|
||||
return ""
|
||||
local = (config.get_subtitle_config() or {}).get("long_video_list", "./超长待重跑.txt")
|
||||
if not local or not os.path.exists(local):
|
||||
return ""
|
||||
# 快照当前内容(文件 append-only); 全空则不传
|
||||
with open(local, "r", encoding="utf-8") as f:
|
||||
snapshot = f.readlines()
|
||||
if not any(ln.strip() for ln in snapshot):
|
||||
return ""
|
||||
dest_root = _cfg().get("long_list_backup_path")
|
||||
if not dest_root:
|
||||
bucket, base = _bucket_base()
|
||||
dest_root = f"cos://{bucket}/{base}/long_video"
|
||||
bkt, base_key = parse_cos_url(dest_root)
|
||||
stem = os.path.splitext(os.path.basename(local))[0] # 超长待重跑
|
||||
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
key = f"{base_key.strip('/')}/todo/{stem}_{machine_id}_{ts}.txt"
|
||||
client().put_object(Bucket=bkt, Body="".join(snapshot).encode("utf-8"), Key=key)
|
||||
# 上传成功后清空已上传部分: 只删前 len(snapshot) 行, 保留上传期间新追加的(append 在末尾),
|
||||
# 这样每天只传增量、跑完即了, 不会和已完成分片重叠。开关 long_list_clear_after_upload(默认开)。
|
||||
if _cfg().get("long_list_clear_after_upload", True):
|
||||
try:
|
||||
with open(local, "r", encoding="utf-8") as f:
|
||||
cur = f.readlines()
|
||||
with open(local, "w", encoding="utf-8") as f:
|
||||
f.writelines(cur[len(snapshot):])
|
||||
except OSError:
|
||||
pass
|
||||
return f"cos://{bkt}/{key}"
|
||||
|
||||
|
||||
def _daily_backup_bucket_base():
|
||||
"""daily_backup_path 配置解析为 (bucket, base_key)，缺省时与 upload_daily_backup 保持一致。
|
||||
兜底基准用 _primary_bucket_base()(固定挂 jobs_root), 不能用 _bucket_base() —— 见后者docstring
|
||||
里的解释, 否则专职长视频机会把每日备份传到 long_video 队列自己底下, 全局去重读不到。"""
|
||||
dest_root = _cfg().get("daily_backup_path")
|
||||
if not dest_root:
|
||||
bucket, base = _primary_bucket_base()
|
||||
dest_root = f"cos://{bucket}/{base}/daily"
|
||||
bucket, base = parse_cos_url(dest_root)
|
||||
return bucket, base.strip("/")
|
||||
|
||||
|
||||
def _extract_processed_paths(jsonl_path: str) -> list:
|
||||
"""从本地 数据.jsonl 逐行解析出 video_cos_path 列表(顺序不去重, 上传前用)。"""
|
||||
paths = []
|
||||
try:
|
||||
with open(jsonl_path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
vid = json.loads(line).get("video_cos_path")
|
||||
if vid:
|
||||
paths.append(vid)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
except OSError:
|
||||
pass
|
||||
return paths
|
||||
|
||||
|
||||
def upload_processed_list(jsonl_path: str, machine_id: str) -> str:
|
||||
"""从本地 数据.jsonl 提取 video_cos_path, 生成一份纯文本清单(一行一个路径, 不含状态/耗时等
|
||||
其它字段)上传到 daily_backup_path/<日期>/已处理_<机器号>.txt —— 跟 upload_daily_backup 上传的
|
||||
完整 jsonl 是同一份数据的两种形态, 前者给 global_processed_set() 用: 只关心"这个视频是否
|
||||
处理过"时, 拉这个纯清单比拉完整jsonl再挨行解析JSON 小得多、快得多(jsonl里每条记录还带
|
||||
timestamp/duration/image_counts/stage_durations 等一堆跟去重无关的字段)。"""
|
||||
dest_root = _cfg().get("daily_backup_path")
|
||||
if not dest_root:
|
||||
bucket, base = _bucket_base()
|
||||
dest_root = f"cos://{bucket}/{base}/daily"
|
||||
paths = _extract_processed_paths(jsonl_path)
|
||||
if not paths:
|
||||
return ""
|
||||
bkt, base_key = parse_cos_url(dest_root)
|
||||
date = datetime.now().strftime("%Y%m%d")
|
||||
key = f"{base_key.strip('/')}/{date}/已处理_{machine_id}.txt"
|
||||
client().put_object(Bucket=bkt, Body=("\n".join(paths) + "\n").encode("utf-8"), Key=key)
|
||||
return f"cos://{bkt}/{key}"
|
||||
|
||||
|
||||
def list_processed_list_files():
|
||||
"""列出 daily_backup_path 下所有 <日期>/已处理_<机器号>.txt，返回 [(日期, 机器号, key), ...]。"""
|
||||
bkt, base = _daily_backup_bucket_base()
|
||||
prefix = base + "/"
|
||||
out, marker = [], ""
|
||||
while True:
|
||||
resp = client().list_objects(Bucket=bkt, Prefix=prefix, Marker=marker, MaxKeys=1000)
|
||||
for c in resp.get("Contents", []) or []:
|
||||
rel = c["Key"][len(prefix):] # <日期>/已处理_<机器号>.txt
|
||||
parts = rel.split("/")
|
||||
if len(parts) == 2 and parts[1].startswith("已处理_") and parts[1].endswith(".txt"):
|
||||
out.append((parts[0], parts[1][len("已处理_"):-len(".txt")], c["Key"]))
|
||||
if resp.get("IsTruncated") in ("true", True):
|
||||
marker = resp.get("NextMarker") or (resp["Contents"][-1]["Key"] if resp.get("Contents") else "")
|
||||
if not marker:
|
||||
break
|
||||
else:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def global_processed_set(verbose: bool = False, all_dates: bool = False) -> set:
|
||||
"""聚合"已处理清单"(已处理_<机器号>.txt, 由 upload_processed_list 在每次备份时跟 数据.jsonl
|
||||
一起生成), 合并成全局已处理 video_cos_path 集合。只拉轻量txt清单、不下载/解析完整jsonl,
|
||||
省流量也省CPU。语义与本机 load_processed() 一致：不分 success/no_subtitle/error，只要出现过
|
||||
就算处理过。用于灌任务前过滤，避免同一视频被 30 台机器重复处理。
|
||||
|
||||
all_dates=False(默认, 日常灌任务用): 每台机器只取【最新一天】——数据.jsonl 是全量快照(本地
|
||||
从不清空), 取最新一份即覆盖该机器全部历史, 省得把每天的备份都下载一遍; 不同机器最新日期
|
||||
可能不同(比如有机器当天没开机备份), 各自独立取最新, 不要求同一天。
|
||||
|
||||
all_dates=True(机群已停机/一次性核对时用): 不做"每机器只取最新"这个假设, 把【所有日期】
|
||||
的清单全部下载合并——机群都停了就没有"之后还会补传更晚日期"这回事, 更保险的做法是把能拿到
|
||||
的所有历史清单全并起来判断"处理过没有", 而不是赌某一天的快照就是最全的(万一那台机器最新
|
||||
一天的快照因为某种原因不完整/被截断过, 只取最新反而会漏)。代价是下载的文件数从"机器数"
|
||||
变成"机器数×历史天数", 视频清单量小(轻量txt)时这个代价可以接受。"""
|
||||
files = list_processed_list_files()
|
||||
if not files:
|
||||
return set()
|
||||
|
||||
if all_dates:
|
||||
targets = [(date_str, machine_id, key) for date_str, machine_id, key in files]
|
||||
else:
|
||||
latest = {} # machine_id -> (日期, key)
|
||||
for date_str, machine_id, key in files:
|
||||
if machine_id not in latest or date_str > latest[machine_id][0]:
|
||||
latest[machine_id] = (date_str, key)
|
||||
targets = [(date_str, machine_id, key) for machine_id, (date_str, key) in latest.items()]
|
||||
|
||||
bkt, _ = _daily_backup_bucket_base()
|
||||
processed = set()
|
||||
for date_str, machine_id, key in targets:
|
||||
try:
|
||||
data = client().get_object(Bucket=bkt, Key=key)["Body"].get_raw_stream().read()
|
||||
except Exception as e:
|
||||
logging.warning(f"[全局去重] 读取 {key} 失败, 跳过: {e}")
|
||||
continue
|
||||
n = 0
|
||||
for line in data.decode("utf-8", errors="ignore").splitlines():
|
||||
line = line.strip()
|
||||
if line:
|
||||
processed.add(line)
|
||||
n += 1
|
||||
if verbose:
|
||||
logging.info(f"[全局去重] 机器 {machine_id}({date_str}): {n} 条")
|
||||
return processed
|
||||
|
||||
|
||||
def run_daily_backup_loop(machine_id: str, hour: int, stop, jsonl_path: str = "数据.jsonl",
|
||||
check_interval: int = 120):
|
||||
"""后台线程: 每天到 hour 点把 数据.jsonl + 超长列表上传 COS, 每天只传一次。
|
||||
main.py(本地模式) 和 run_worker.py(COS模式) 通用。stop 为 threading.Event, set() 即退出。"""
|
||||
last_date = None
|
||||
while not stop.wait(check_interval): # 每 check_interval 秒检查一次
|
||||
now = datetime.now()
|
||||
today = now.strftime("%Y%m%d")
|
||||
if now.hour >= hour and last_date != today:
|
||||
try:
|
||||
dst = upload_daily_backup(jsonl_path, machine_id)
|
||||
if dst:
|
||||
logging.info(f"[每日备份] 数据.jsonl 已上传 -> {dst}")
|
||||
else:
|
||||
logging.info("[每日备份] 本地无 数据.jsonl, 跳过")
|
||||
ll = upload_longlist_backup(machine_id)
|
||||
if ll:
|
||||
logging.info(f"[每日备份] 超长待重跑列表已上传 -> {ll}")
|
||||
last_date = today # 当天标记完成, 避免反复触发
|
||||
except Exception as e:
|
||||
logging.error(f"[每日备份] 上传失败(稍后重试): {e}")
|
||||
|
||||
|
||||
# ----------------------------------------------------- 每小时监控摘要 ----
|
||||
|
||||
def _monitor_bucket_base():
|
||||
"""monitor_root 配置解析为 (bucket, base_key)，缺省时 = jobs_root 的同级 monitor 目录
|
||||
(跟 _daily_backup_bucket_base 同一个派生模式)。兜底基准同样必须用 _primary_bucket_base()
|
||||
而不是 _bucket_base() —— 这就是本函数曾经的实际bug: 专职长视频机(run_long_video=true)
|
||||
没显式配 monitor_root 时, 旧版用 _bucket_base() 兜底会解析到【当前生效队列根】
|
||||
(_active_jobs_root(), 长视频机上长期等于 long_list_backup_path), 算出
|
||||
.../long_video/monitor/... 这种独立于其它机器的路径, fleet_monitor.py 只读标准
|
||||
monitor_root, 找不到这些机器的小时摘要, 表现为"全队机器数对不上"(实际数据都传了,
|
||||
只是传错地方, 数据没丢, 改这个函数后新上传的会去对的地方, 旧的仍留在错的路径下)。"""
|
||||
dest_root = _cfg().get("monitor_root")
|
||||
if not dest_root:
|
||||
bucket, base = _primary_bucket_base()
|
||||
dest_root = f"cos://{bucket}/{base}/monitor"
|
||||
bucket, base = parse_cos_url(dest_root)
|
||||
return bucket, base.strip("/")
|
||||
|
||||
|
||||
def compute_hourly_digest(jsonl_path: str, hour_dt: datetime):
|
||||
"""从本地 数据.jsonl 里筛出 process_start(缺失回退timestamp) 落在 hour_dt 所在整点小时的记录,
|
||||
算出这个小时的摘要。hour_dt 的分钟/秒会被忽略, 只取年月日时。
|
||||
本地jsonl不存在/读取失败 -> 返回 None(调用方应跳过上传)。
|
||||
这一小时确实一条记录都没处理(count==0, 比如COS任务队列暂时被抢空、worker在idle轮询)
|
||||
-> 仍然返回一份全0的摘要, 不当 None 处理 —— 队列见底idle是这套分布式调度系统的正常状态
|
||||
(见 run_worker.py 的 keep_alive/idle_poll_seconds), 如果这种情况也不传摘要, fleet_monitor.py
|
||||
的"机器掉线"检测会把"确认活着只是没活干"误判成"可能停机", 制造假警报。"""
|
||||
if not os.path.exists(jsonl_path):
|
||||
return None
|
||||
hour_prefix = hour_dt.strftime("%Y-%m-%dT%H") # 匹配 ISO 格式 process_start 的前13位
|
||||
count = success = no_subtitle = error = 0
|
||||
success_hours = 0.0
|
||||
try:
|
||||
with open(jsonl_path, "r", encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
ts = rec.get("process_start") or rec.get("timestamp") or ""
|
||||
if ts[:13] != hour_prefix:
|
||||
continue
|
||||
count += 1
|
||||
status = rec.get("status")
|
||||
if status == "success":
|
||||
success += 1
|
||||
try:
|
||||
success_hours += float(rec.get("duration") or 0) / 3600.0
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
elif status == "no_subtitle":
|
||||
no_subtitle += 1
|
||||
else:
|
||||
error += 1
|
||||
except OSError:
|
||||
return None
|
||||
return {
|
||||
"machine_id": None, # upload_hourly_digest 里填
|
||||
"hour": hour_dt.strftime("%Y-%m-%dT%H"),
|
||||
"count": count, "success": success, "no_subtitle": no_subtitle, "error": error,
|
||||
"success_hours": round(success_hours, 2),
|
||||
"uploaded_at": datetime.now().isoformat(),
|
||||
}
|
||||
|
||||
|
||||
def upload_hourly_digest(machine_id: str, jsonl_path: str, hour_dt: datetime) -> str:
|
||||
"""算好 hour_dt 那个整点小时的摘要并上传到
|
||||
monitor_root/hourly/<日期YYYYMMDD>/<小时HH>/<机器号>.json。
|
||||
即使这一小时count==0(队列见底idle)也会正常上传一份全0摘要——只有本地 数据.jsonl
|
||||
根本不存在/读取失败(digest为None, 比如worker刚起来还没写过jsonl)才跳过, 返回''。"""
|
||||
digest = compute_hourly_digest(jsonl_path, hour_dt)
|
||||
if digest is None:
|
||||
return ""
|
||||
digest["machine_id"] = machine_id
|
||||
bkt, base = _monitor_bucket_base()
|
||||
date_str = hour_dt.strftime("%Y%m%d")
|
||||
hour_str = hour_dt.strftime("%H")
|
||||
key = f"{base}/hourly/{date_str}/{hour_str}/{machine_id}.json"
|
||||
client().put_object(Bucket=bkt, Body=json.dumps(digest, ensure_ascii=False).encode("utf-8"), Key=key)
|
||||
return f"cos://{bkt}/{key}"
|
||||
|
||||
|
||||
def list_hourly_digest_files_for_hour(hour_dt: datetime) -> list:
|
||||
"""列出 monitor_root/hourly/<日期>/<小时>/ 下所有机器的摘要文件, 返回 [(机器号, key), ...]。"""
|
||||
bkt, base = _monitor_bucket_base()
|
||||
date_str = hour_dt.strftime("%Y%m%d")
|
||||
hour_str = hour_dt.strftime("%H")
|
||||
prefix = f"{base}/hourly/{date_str}/{hour_str}/"
|
||||
out, marker = [], ""
|
||||
while True:
|
||||
resp = client().list_objects(Bucket=bkt, Prefix=prefix, Marker=marker, MaxKeys=1000)
|
||||
for c in resp.get("Contents", []) or []:
|
||||
rel = c["Key"][len(prefix):]
|
||||
if rel.endswith(".json"):
|
||||
out.append((rel[:-len(".json")], c["Key"]))
|
||||
if resp.get("IsTruncated") in ("true", True):
|
||||
marker = resp.get("NextMarker") or (resp["Contents"][-1]["Key"] if resp.get("Contents") else "")
|
||||
if not marker:
|
||||
break
|
||||
else:
|
||||
break
|
||||
return out
|
||||
|
||||
|
||||
def download_hourly_digest(key: str) -> dict:
|
||||
"""下载解析单份小时摘要json, 返回dict。失败抛异常, 调用方负责捕获。"""
|
||||
bkt, _ = _monitor_bucket_base()
|
||||
data = client().get_object(Bucket=bkt, Key=key)["Body"].get_raw_stream().read()
|
||||
return json.loads(data.decode("utf-8"))
|
||||
|
||||
|
||||
def upload_monitor_report(local_path: str, hour_dt: datetime) -> str:
|
||||
"""把生成好的监控报告文件上传到 monitor_root/reports/<日期YYYYMMDD>_<小时HH>.md。"""
|
||||
if not os.path.exists(local_path):
|
||||
return ""
|
||||
bkt, base = _monitor_bucket_base()
|
||||
date_str = hour_dt.strftime("%Y%m%d")
|
||||
hour_str = hour_dt.strftime("%H")
|
||||
key = f"{base}/reports/{date_str}_{hour_str}.md"
|
||||
with open(local_path, "rb") as f:
|
||||
data = f.read()
|
||||
client().put_object(Bucket=bkt, Body=data, Key=key)
|
||||
return f"cos://{bkt}/{key}"
|
||||
|
||||
|
||||
def run_hourly_digest_loop(machine_id: str, stop, jsonl_path: str = "数据.jsonl",
|
||||
check_interval: int = 120):
|
||||
"""后台线程: 每小时把刚结束的那个整点小时的处理摘要上传COS, 每小时只传一次。
|
||||
main.py(本地模式) 不需要接入(一次性批处理没有"常驻小时"的概念, 见设计文档)。
|
||||
stop 为 threading.Event, set() 即退出。"""
|
||||
last_uploaded_hour = None
|
||||
while not stop.wait(check_interval):
|
||||
now = datetime.now()
|
||||
prev_hour_dt = now.replace(minute=0, second=0, microsecond=0) - timedelta(hours=1)
|
||||
prev_hour_key = prev_hour_dt.strftime("%Y%m%d%H")
|
||||
if last_uploaded_hour == prev_hour_key:
|
||||
continue
|
||||
try:
|
||||
dst = upload_hourly_digest(machine_id, jsonl_path, prev_hour_dt)
|
||||
if dst:
|
||||
logging.info(f"[小时摘要] {prev_hour_key} 已上传 -> {dst}")
|
||||
else:
|
||||
logging.info(f"[小时摘要] {prev_hour_key} 本地 数据.jsonl 尚不存在, 跳过上传")
|
||||
last_uploaded_hour = prev_hour_key
|
||||
except Exception as e:
|
||||
logging.error(f"[小时摘要] {prev_hour_key} 上传失败(下次继续重试): {e}")
|
||||
0
tool/log/__init__.py
Normal file
0
tool/log/__init__.py
Normal file
218
tool/log/log_manager.py
Normal file
218
tool/log/log_manager.py
Normal file
@@ -0,0 +1,218 @@
|
||||
# log_manager.py
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
import threading
|
||||
from collections import deque
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
|
||||
class ProgramFilter(logging.Filter):
|
||||
"""自定义日志过滤器，为日志记录添加程序名称属性"""
|
||||
|
||||
def filter(self, record: logging.LogRecord) -> bool:
|
||||
if not hasattr(record, 'program'):
|
||||
record.program = "UNKNOWN"
|
||||
return True
|
||||
|
||||
|
||||
class UnifiedLogger:
|
||||
"""统一日志管理器（线程安全单例），支持多程序日志收集"""
|
||||
|
||||
_instance: Optional['UnifiedLogger'] = None
|
||||
_lock = threading.Lock()
|
||||
|
||||
def __new__(cls) -> 'UnifiedLogger':
|
||||
with cls._lock:
|
||||
if cls._instance is None:
|
||||
cls._instance = super().__new__(cls)
|
||||
return cls._instance
|
||||
|
||||
def __init__(self) -> None:
|
||||
# 避免重复初始化
|
||||
if hasattr(self, '_initialized') and self._initialized:
|
||||
return
|
||||
|
||||
# 每视频状态使用线程本地存储：批处理多线程并发时，每个视频在各自线程内
|
||||
# 独立记录日志/计时/统计，互不串扰。（嵌套子线程如OCR并发worker不应直接
|
||||
# 调用本logger的状态方法——它们会落到worker线程的独立状态而丢失。）
|
||||
self._local = threading.local()
|
||||
|
||||
# 是否打印详细日志到控制台。默认 False：控制台只显示"视频→步骤"里程碑，
|
||||
# 详细日志仍完整存进每视频 JSON(log/<video>.json)。设 LOG_VERBOSE=1 恢复全部。
|
||||
self.verbose = os.environ.get("LOG_VERBOSE", "0") == "1"
|
||||
|
||||
# 自定义 logger（避免污染根日志器）
|
||||
self.logger = logging.getLogger("UnifiedLogger")
|
||||
self.logger.setLevel(logging.DEBUG)
|
||||
self.logger.propagate = False # 不传播到 root，避免每条日志重复打印
|
||||
self.logger.addFilter(ProgramFilter())
|
||||
|
||||
# 控制台 Handler：非 verbose 时只放行 WARNING+，把 INFO 细节挡在控制台外(仍进 JSON)
|
||||
console_handler = logging.StreamHandler()
|
||||
console_handler.setLevel(logging.DEBUG if self.verbose else logging.WARNING)
|
||||
console_formatter = logging.Formatter(
|
||||
'%(asctime)s [%(levelname)s] [%(program)s] %(message)s'
|
||||
)
|
||||
console_handler.setFormatter(console_formatter)
|
||||
self.logger.addHandler(console_handler)
|
||||
|
||||
# 第三方库降噪：jieba 的 DEBUG("Building prefix dict...")太吵
|
||||
logging.getLogger("jieba").setLevel(logging.WARNING)
|
||||
# qcloud_cos SDK 噪音大: 每次 get/put/head object 都打一条 INFO(带完整url/headers),
|
||||
# 而且它把"对象不存在"(HEAD 404, 本项目里大量用来判断分片是否已完成/锁是否存在,
|
||||
# 属于正常控制流)也当 WARNING 打一条 —— 两种都没信息量, 真正的失败走异常
|
||||
# (CosServiceError), 本项目自己的 try/except 会记录, 不受这里调级影响。
|
||||
logging.getLogger("qcloud_cos").setLevel(logging.ERROR)
|
||||
|
||||
self._initialized = True
|
||||
|
||||
def step(self, video: str, step: str, extra: str = "") -> None:
|
||||
"""打印一条简洁的"视频 → 步骤"里程碑（始终显示，不受 verbose 影响）。"""
|
||||
line = f"[{datetime.now().strftime('%H:%M:%S')}] {video} → {step}"
|
||||
if extra:
|
||||
line += f" ({extra})"
|
||||
print(line, flush=True)
|
||||
try:
|
||||
self._state()["log_buffer"].append({
|
||||
"timestamp": datetime.now().isoformat(), "level": "STEP",
|
||||
"program": "step", "message": f"{step} {extra}".strip(),
|
||||
})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
def _state(self) -> dict:
|
||||
"""获取当前线程的日志状态，惰性初始化。"""
|
||||
st = getattr(self._local, 'state', None)
|
||||
if st is None:
|
||||
st = {
|
||||
"log_buffer": deque(maxlen=10000),
|
||||
"log_file": "unified_log.json",
|
||||
"program_sequence": [],
|
||||
"program_start_times": {},
|
||||
"program_end_times": {},
|
||||
"custom_stats": {},
|
||||
}
|
||||
self._local.state = st
|
||||
return st
|
||||
|
||||
def init_logging(self, log_file: str = "unified_log.json") -> None:
|
||||
"""初始化日志系统（可重新指定日志文件路径）"""
|
||||
self._state()["log_file"] = log_file
|
||||
log_dir = os.path.dirname(log_file)
|
||||
if log_dir:
|
||||
os.makedirs(log_dir, exist_ok=True)
|
||||
|
||||
def reset(self) -> None:
|
||||
"""重置当前线程的日志状态（用于多线程场景下每个视频独立记录）"""
|
||||
st = self._state()
|
||||
st["log_buffer"].clear()
|
||||
st["program_sequence"].clear()
|
||||
st["program_start_times"].clear()
|
||||
st["program_end_times"].clear()
|
||||
st["custom_stats"].clear()
|
||||
|
||||
def set_stat(self, key: str, value: object) -> None:
|
||||
"""记录一项自定义统计，会写入日志 metadata.custom_stats"""
|
||||
self._state()["custom_stats"][key] = value
|
||||
|
||||
def _get_level_int(self, level: Union[str, int]) -> int:
|
||||
"""将日志级别转换为 logging 模块的整数常量"""
|
||||
if isinstance(level, int):
|
||||
return level
|
||||
level_map = {
|
||||
"DEBUG": logging.DEBUG,
|
||||
"INFO": logging.INFO,
|
||||
"WARNING": logging.WARNING,
|
||||
"ERROR": logging.ERROR,
|
||||
"CRITICAL": logging.CRITICAL
|
||||
}
|
||||
return level_map.get(level.upper(), logging.INFO)
|
||||
|
||||
def log(self, message: str, level: Union[str, int] = "info", program: str = "A") -> None:
|
||||
"""记录一条日志"""
|
||||
timestamp = datetime.now().isoformat()
|
||||
level_str = level if isinstance(level, str) else logging.getLevelName(level)
|
||||
log_entry = {
|
||||
"timestamp": timestamp,
|
||||
"level": level_str,
|
||||
"program": program,
|
||||
"message": message
|
||||
}
|
||||
|
||||
self._state()["log_buffer"].append(log_entry)
|
||||
|
||||
# 输出到自定义 logger
|
||||
level_int = self._get_level_int(level)
|
||||
self.logger.log(level_int, message, extra={"program": program})
|
||||
|
||||
def start_program(self, program_name: str) -> None:
|
||||
"""记录程序开始执行"""
|
||||
st = self._state()
|
||||
if program_name in st["program_start_times"]:
|
||||
self.logger.warning(f"程序 {program_name} 已开始，忽略重复 start 调用", extra={"program": "SYSTEM"})
|
||||
return
|
||||
|
||||
st["program_start_times"][program_name] = time.time()
|
||||
st["program_sequence"].append(program_name)
|
||||
|
||||
self.log(f"程序 {program_name} 开始执行", program=program_name)
|
||||
|
||||
def end_program(self, program_name: str) -> None:
|
||||
"""记录程序执行结束"""
|
||||
st = self._state()
|
||||
if program_name not in st["program_start_times"]:
|
||||
self.logger.error(f"程序 {program_name} 未调用 start_program，无法记录结束", extra={"program": "SYSTEM"})
|
||||
return
|
||||
|
||||
end_time = time.time()
|
||||
st["program_end_times"][program_name] = end_time
|
||||
duration = end_time - st["program_start_times"][program_name]
|
||||
|
||||
self.log(f"程序 {program_name} 执行完成 (耗时: {duration:.2f}s)", program=program_name)
|
||||
|
||||
def save_logs(self) -> None:
|
||||
"""将当前线程的所有日志（含元数据）保存到 JSON 文件"""
|
||||
st = self._state()
|
||||
logs_copy = list(st["log_buffer"])
|
||||
start_times_copy = dict(st["program_start_times"])
|
||||
end_times_copy = dict(st["program_end_times"])
|
||||
sequence_copy = list(st["program_sequence"])
|
||||
stats_copy = dict(st["custom_stats"])
|
||||
log_file = st["log_file"]
|
||||
|
||||
# 计算程序执行耗时
|
||||
durations = {}
|
||||
for prog, start in start_times_copy.items():
|
||||
end = end_times_copy.get(prog, time.time())
|
||||
durations[prog] = end - start
|
||||
|
||||
metadata = {
|
||||
"log_created": datetime.now().isoformat(),
|
||||
"program_sequence": sequence_copy,
|
||||
"total_entries": len(logs_copy),
|
||||
"program_durations": durations,
|
||||
"custom_stats": stats_copy
|
||||
}
|
||||
|
||||
full_log = {
|
||||
"metadata": metadata,
|
||||
"logs": logs_copy
|
||||
}
|
||||
|
||||
try:
|
||||
with open(log_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(full_log, f, ensure_ascii=False, indent=2)
|
||||
self.logger.info(f"所有日志已保存到: {log_file}", extra={"program": "SYSTEM"})
|
||||
except (OSError, IOError) as e:
|
||||
self.logger.error(f"保存日志失败: {e}", extra={"program": "SYSTEM"})
|
||||
|
||||
def get_logs(self) -> List[dict]:
|
||||
"""获取当前线程缓冲区中的所有日志副本"""
|
||||
return list(self._state()["log_buffer"])
|
||||
|
||||
|
||||
# 全局单例实例
|
||||
logger = UnifiedLogger()
|
||||
885
tool/merge.py
Normal file
885
tool/merge.py
Normal file
@@ -0,0 +1,885 @@
|
||||
import os
|
||||
import json
|
||||
import cv2
|
||||
import numpy as np
|
||||
import random
|
||||
import re
|
||||
import string
|
||||
import math
|
||||
import tempfile
|
||||
from collections import defaultdict
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from difflib import SequenceMatcher
|
||||
from PIL import Image
|
||||
from typing import Optional, List, Tuple, Dict, Any, Union
|
||||
# 假设这些外部模块已正确导入
|
||||
from tool.Ocr.tencentcloud_ocr import tencentcloud_ocr_api, img2base64
|
||||
from tool.whisper_manager import whisper_manager
|
||||
from tool.log.log_manager import logger
|
||||
from config.config_utils import config
|
||||
|
||||
|
||||
# ==================== 辅助函数 ====================
|
||||
def natural_sort_key(filename: str) -> List[Union[int, str]]:
|
||||
"""
|
||||
自然排序键函数，将文件名中的数字部分转换为整数进行排序。
|
||||
"""
|
||||
return [int(part) if part.isdigit() else part.lower()
|
||||
for part in re.split(r'(\d+)', filename)]
|
||||
|
||||
|
||||
def has_chinese(text: str) -> bool:
|
||||
"""文本是否含中文字符（CJK 统一表意文字）。"""
|
||||
return any('一' <= ch <= '鿿' for ch in (text or ""))
|
||||
|
||||
|
||||
def calculate_english_ratio(text: str) -> float:
|
||||
"""计算文本中英文字符（字母、数字、常见标点）的占比。"""
|
||||
if not text:
|
||||
return 0.0
|
||||
total_chars = len(text)
|
||||
if total_chars == 0:
|
||||
return 0.0
|
||||
|
||||
english_chars = 0
|
||||
for ch in text:
|
||||
if ch.isalpha() and ch.isascii():
|
||||
english_chars += 1
|
||||
elif ch.isdigit():
|
||||
english_chars += 1
|
||||
elif ch in ' .,!?;:\'"-()[]{}':
|
||||
english_chars += 1
|
||||
return english_chars / total_chars
|
||||
|
||||
|
||||
def check_subtitle_quality_enhanced(best_subtitle: str, strict_mode: bool = True) -> bool:
|
||||
"""
|
||||
增强版字幕质量检测。
|
||||
|
||||
:param best_subtitle: 待检测的文本
|
||||
:param strict_mode: 是否使用严格模式
|
||||
:return: 若文本质量合格则返回 True
|
||||
"""
|
||||
if not best_subtitle or not best_subtitle.strip():
|
||||
return True
|
||||
|
||||
subtitle = best_subtitle.strip()
|
||||
total_chars = len(subtitle)
|
||||
if total_chars == 0:
|
||||
return True
|
||||
|
||||
digit_count = sum(1 for ch in subtitle if ch.isdigit())
|
||||
symbol_count = sum(1 for ch in subtitle if ch in string.punctuation)
|
||||
special_ratio = (digit_count + symbol_count) / total_chars
|
||||
english_ratio = calculate_english_ratio(subtitle)
|
||||
|
||||
if strict_mode:
|
||||
digit_ratio = digit_count / total_chars
|
||||
if total_chars >= 10 and digit_ratio <= 0.9 and special_ratio <= 0.9:
|
||||
return True
|
||||
# 数字占比 ≤ 70% 且特殊字符占比 ≤ 80% 且英文占比 ≤ 90%
|
||||
return digit_ratio <= 0.7 and special_ratio <= 0.8 and english_ratio <= 0.9
|
||||
else:
|
||||
# 普通模式仅检查特殊字符占比
|
||||
return special_ratio <= 0.5
|
||||
|
||||
|
||||
def detect_main_region(image_files: List[str],
|
||||
input_dir: str,
|
||||
sample_size: int = 10,
|
||||
motion_threshold: int = 30) -> Tuple[int, int, int, int]:
|
||||
"""
|
||||
通过运动检测确定主画面区域。
|
||||
|
||||
:return: 边界框 (x, y, width, height)
|
||||
"""
|
||||
if not image_files:
|
||||
raise ValueError("图片列表为空")
|
||||
|
||||
first_img_path = os.path.join(input_dir, image_files[0])
|
||||
prev_frame = cv2.imread(first_img_path)
|
||||
if prev_frame is None:
|
||||
raise ValueError(f"无法读取第一帧: {first_img_path}")
|
||||
prev_gray = cv2.cvtColor(prev_frame, cv2.COLOR_BGR2GRAY)
|
||||
height, width = prev_gray.shape
|
||||
|
||||
# 随机采样中间帧（避开开头和结尾）
|
||||
mid_start = len(image_files) // 4
|
||||
mid_end = 3 * len(image_files) // 4
|
||||
sample_indices = random.sample(range(mid_start, mid_end),
|
||||
min(sample_size, mid_end - mid_start))
|
||||
|
||||
motion_accumulator = np.zeros((height, width), dtype=np.float32)
|
||||
|
||||
for idx in sample_indices:
|
||||
img_path = os.path.join(input_dir, image_files[idx])
|
||||
curr_frame = cv2.imread(img_path)
|
||||
if curr_frame is None:
|
||||
continue
|
||||
curr_gray = cv2.cvtColor(curr_frame, cv2.COLOR_BGR2GRAY)
|
||||
frame_diff = cv2.absdiff(curr_gray, prev_gray)
|
||||
_, thresh = cv2.threshold(frame_diff, motion_threshold, 255, cv2.THRESH_BINARY)
|
||||
motion_accumulator += thresh.astype(np.float32)
|
||||
prev_gray = curr_gray
|
||||
|
||||
motion_accumulator = cv2.normalize(motion_accumulator, None, 0, 255, cv2.NORM_MINMAX)
|
||||
_, motion_mask = cv2.threshold(motion_accumulator.astype(np.uint8), 100, 255, cv2.THRESH_BINARY)
|
||||
|
||||
contours, _ = cv2.findContours(motion_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
|
||||
if not contours:
|
||||
# 无运动区域时返回中心偏下区域
|
||||
return (0, height // 4, width, height // 2)
|
||||
|
||||
main_contour = max(contours, key=cv2.contourArea)
|
||||
x, y, w, h = cv2.boundingRect(main_contour)
|
||||
|
||||
# 合理性检查
|
||||
if w < width * 0.1 or h < height * 0.1:
|
||||
return (0, height // 4, width, height // 2)
|
||||
|
||||
# 考虑下方空间
|
||||
bottom_space = height - (y + h)
|
||||
if bottom_space > height * 0.1:
|
||||
extend = int(0.8 * bottom_space)
|
||||
h += extend
|
||||
# 调整 y 使区域居中于原运动区域下方
|
||||
y = y + h // 2
|
||||
h //= 2
|
||||
else:
|
||||
h += bottom_space
|
||||
|
||||
return (x, y, w, h)
|
||||
|
||||
|
||||
# ==================== 字幕区域检测 ====================
|
||||
def _sample_indices_uniform(total: int, sample_size: int) -> List[int]:
|
||||
"""均匀采样索引，确保帧间有间隔。"""
|
||||
print(total, sample_size)
|
||||
if sample_size <= 1:
|
||||
return [total // 2]
|
||||
step = total // (sample_size - 1)
|
||||
indices = [i * step for i in range(sample_size)]
|
||||
# 确保不超出范围
|
||||
indices = [min(i, total - 1) for i in indices]
|
||||
return indices
|
||||
|
||||
|
||||
def _ocr_frames_concurrent(image_files: List[str], input_dir: str,
|
||||
indices: List[int], workers: int) -> Dict[int, str]:
|
||||
"""并发对指定帧索引做 OCR，返回 {idx: 原始响应字符串}。
|
||||
在视频主线程内开子线程池，子线程只做网络请求、不写日志（满足日志线程隔离要求）。"""
|
||||
results: Dict[int, str] = {}
|
||||
if not indices:
|
||||
return results
|
||||
|
||||
def _one(idx):
|
||||
try:
|
||||
img_path = os.path.join(input_dir, image_files[idx])
|
||||
return idx, tencentcloud_ocr_api(img2base64(img_path=img_path))
|
||||
except Exception:
|
||||
return idx, None
|
||||
|
||||
with ThreadPoolExecutor(max_workers=max(1, min(workers, len(indices)))) as ex:
|
||||
for idx, res in ex.map(_one, indices):
|
||||
results[idx] = res
|
||||
return results
|
||||
|
||||
|
||||
def cv_locate_subtitle(image_files: List[str], input_dir: str,
|
||||
height: int, width: int, sample: int = 80):
|
||||
"""纯 CV(零 OCR)定位硬字幕带。读已抽好的 JPEG 帧(cv2.imread,与 AV1 解码无关)。
|
||||
返回 (verdict, region, info):
|
||||
'subtitle' (x,y,w,h) —— 找到下部动态字幕带
|
||||
'none' None —— 整体静止+无动态文字带 = PPT/文档/截图,可直接跳过
|
||||
'uncertain'None —— 拿不准,回退原 OCR 检测(不改变原行为)
|
||||
安全准则: 只有【整体静止且无任何动态文字带】才判 none,绝不误杀真字幕视频。"""
|
||||
n = len(image_files)
|
||||
if n < 10:
|
||||
return "uncertain", None, {"reason": "帧太少"}
|
||||
idxs = np.linspace(n * 0.1, n * 0.9, min(sample, n)).astype(int)
|
||||
grays = []
|
||||
for i in idxs:
|
||||
g = cv2.imread(os.path.join(input_dir, image_files[int(i)]), cv2.IMREAD_GRAYSCALE)
|
||||
if g is not None:
|
||||
grays.append(g)
|
||||
if len(grays) < 8:
|
||||
return "uncertain", None, {"reason": "读帧失败"}
|
||||
H, W = grays[0].shape
|
||||
x0, x1 = int(0.15 * W), int(0.85 * W) # 只看横向中间70%,避开角落logo
|
||||
|
||||
edge_acc = np.zeros(H); present = np.zeros(H)
|
||||
for g in grays:
|
||||
e = cv2.Canny(g, 80, 200)
|
||||
row = e[:, x0:x1].sum(axis=1) / 255.0
|
||||
edge_acc += row
|
||||
present += (row > (x1 - x0) * 0.02)
|
||||
edge_profile = edge_acc / len(grays)
|
||||
presence = present / len(grays)
|
||||
diff_acc = np.zeros(H)
|
||||
for k in range(1, len(grays)):
|
||||
diff_acc += cv2.absdiff(grays[k], grays[k - 1])[:, x0:x1].mean(axis=1)
|
||||
temporal = diff_acc / max(1, len(grays) - 1)
|
||||
|
||||
TEMP_HI = 8.0 # 带变化≥此 = 字幕在翻动(动态)
|
||||
TEMP_LO = 4.0 # 文字带变化<此 = 静止
|
||||
STATIC_MED = 3.0 # 全画面 temporal 中位数<此 = 整体静止(PPT/文档)
|
||||
PRES = 0.4
|
||||
med = float(np.median(temporal))
|
||||
|
||||
cand = (presence > PRES) & (edge_profile > edge_profile.max() * 0.25)
|
||||
rows = np.where(cand)[0]
|
||||
if len(rows) == 0:
|
||||
if med < STATIC_MED:
|
||||
return "none", None, {"reason": "无文字带且画面静止", "med": round(med, 1)}
|
||||
return "uncertain", None, {"reason": "无稳定文字带"}
|
||||
|
||||
bands = []
|
||||
s = p = rows[0]
|
||||
for r in rows[1:]:
|
||||
if r - p > H * 0.03:
|
||||
bands.append((s, p)); s = r
|
||||
p = r
|
||||
bands.append((s, p))
|
||||
MIN_H = int(H * 0.012)
|
||||
bands = [b for b in bands if (b[1] - b[0]) >= MIN_H]
|
||||
|
||||
def bchg(b):
|
||||
return float(temporal[b[0]:b[1] + 1].mean())
|
||||
|
||||
dyn = [b for b in bands if bchg(b) >= TEMP_HI] if bands else []
|
||||
if not dyn:
|
||||
# 无动态字幕带: 整体静止 或 所有文字带都静止 → PPT跳过; 否则交OCR
|
||||
if med < STATIC_MED:
|
||||
return "none", None, {"reason": "无动态字幕带且画面静止(PPT)", "med": round(med, 1)}
|
||||
if bands and max(bchg(b) for b in bands) < TEMP_LO:
|
||||
return "none", None, {"reason": "所有文字带静止", "max_chg": round(max(bchg(b) for b in bands), 1)}
|
||||
return "uncertain", None, {"reason": "拿不准(交OCR)", "med": round(med, 1)}
|
||||
|
||||
dyn.sort(key=lambda b: presence[b[0]:b[1] + 1].mean() * (0.4 + (b[0] + b[1]) / 2 / H), reverse=True)
|
||||
ys, ye = dyn[0]
|
||||
band_h = (ye - ys) / H; cy = (ys + ye) / 2 / H
|
||||
# 若抽帧已裁到下半屏(crop_bottom_ratio),帧内"偏上"不再代表标题,放宽位置判定。
|
||||
crop_r = config.get_splitframe_config().get('crop_bottom_ratio')
|
||||
try:
|
||||
crop_r = float(crop_r) if crop_r is not None else None
|
||||
except (TypeError, ValueError):
|
||||
crop_r = None
|
||||
min_cy = 0.15 if (crop_r and 0.0 < crop_r < 1.0) else 0.45
|
||||
# 有动态带=【不是PPT】。带太宽/偏上只代表不敢直接给区域 → 回退OCR,绝不判none。
|
||||
if band_h > 0.35:
|
||||
return "uncertain", None, {"reason": "动态带太宽,交OCR", "band_h": round(band_h, 2)}
|
||||
if cy < min_cy:
|
||||
return "uncertain", None, {"reason": "带偏上,交OCR", "cy": round(cy, 2)}
|
||||
pad = int(H * 0.012)
|
||||
y = max(0, ys - pad); h = min(H, ye + pad) - y
|
||||
return "subtitle", (0, int(y), int(W), int(h)), {"cy": round(cy, 2), "chg": round(bchg((ys, ye)), 1)}
|
||||
|
||||
|
||||
def detect_subtitle_region(image_files: List[str],
|
||||
input_dir: str,
|
||||
sample_size: int = 15,
|
||||
keep_num: int = 9) -> Tuple[Optional[Tuple[int, int, int, int]],
|
||||
List[int],
|
||||
List[str],
|
||||
List[Dict]]:
|
||||
"""
|
||||
使用 OCR 检测字幕区域。
|
||||
|
||||
:return: (subtitle_region, sample_indices, watermark_logs, candidate_regions)
|
||||
subtitle_region 为 (x, y, width, height)；若无字幕则返回 None
|
||||
"""
|
||||
_sub_cfg = config.get_subtitle_config()
|
||||
require_chinese = _sub_cfg.get('require_chinese', True) # 只要中文字幕
|
||||
bands_thr = int(_sub_cfg.get('redundant_bands', 12)) # 冗余阈值(垂直位置簇数)，越大越宽松
|
||||
|
||||
# 读取第一张图片获取尺寸
|
||||
first_img_path = os.path.join(input_dir, image_files[0])
|
||||
img = cv2.imread(first_img_path)
|
||||
if img is None:
|
||||
raise ValueError(f"无法读取图片: {first_img_path}")
|
||||
height, width = img.shape[:2]
|
||||
|
||||
# === CV 快速闸门(零 OCR,~1.4s)===
|
||||
# 'none' → 静态/PPT,直接跳过(省掉对这类视频的几千次 OCR);
|
||||
# 'subtitle' 且开启 cv_use_region → 直接用 CV 区域(跳过 OCR 检测);
|
||||
# 否则 → 继续走下面的 OCR 检测(行为不变)。
|
||||
if _sub_cfg.get('cv_detect', True):
|
||||
try:
|
||||
cv_verdict, cv_region, cv_info = cv_locate_subtitle(image_files, input_dir, height, width)
|
||||
except Exception as _e:
|
||||
cv_verdict, cv_region, cv_info = "uncertain", None, {"reason": f"CV异常:{_e}"}
|
||||
if cv_verdict == "none":
|
||||
logger.log(f"CV定位: 无动态字幕(静态/PPT),跳过 [{cv_info}]", level="warning", program="merge")
|
||||
return None, [], [], []
|
||||
if cv_verdict == "subtitle" and _sub_cfg.get('cv_use_region', False):
|
||||
logger.log(f"CV定位: 字幕区域 {cv_region} (零OCR) [{cv_info}]", program="merge")
|
||||
x, y, w, h = cv_region
|
||||
cand = [{"x_min": x, "y_min": y, "x_max": x + w, "y_max": y + h, "width": w, "height": h}]
|
||||
return cv_region, [], [], cand
|
||||
logger.log(f"CV定位: {cv_verdict} [{cv_info}] → 走OCR检测", program="merge")
|
||||
|
||||
# 均匀采样帧（扩大到 10%-90% 区间，提高命中字幕帧的概率）
|
||||
mid_start = len(image_files) // 10
|
||||
mid_end = 9 * len(image_files) // 10
|
||||
sample_range = mid_end - mid_start
|
||||
sample_indices = [mid_start + idx for idx in _sample_indices_uniform(sample_range, sample_size)]
|
||||
|
||||
# 存储候选区域和水印信息
|
||||
candidate_regions = [] # 每个元素为 dict: {x_min, y_min, x_max, y_max, width, height}
|
||||
text_positions = defaultdict(list) # {text: [{'x':..., 'y':..., 'idx':...}]}
|
||||
watermark_logs = []
|
||||
|
||||
# 并发预取采样帧的 OCR（原为逐帧串行，~15帧×2s 全在等网络）
|
||||
ocr_workers = config.get_subtitle_config().get('ocr_workers', 10)
|
||||
ocr_cache = _ocr_frames_concurrent(image_files, input_dir, sample_indices, ocr_workers)
|
||||
|
||||
for idx in sample_indices:
|
||||
ocr_result = ocr_cache.get(idx)
|
||||
if not ocr_result:
|
||||
continue
|
||||
ocr_data = json.loads(ocr_result)
|
||||
if "TextDetections" not in ocr_data:
|
||||
continue
|
||||
|
||||
for detection in ocr_data["TextDetections"]:
|
||||
if "Polygon" not in detection:
|
||||
continue
|
||||
polygon = detection["Polygon"]
|
||||
xs = [point["X"] for point in polygon]
|
||||
ys = [point["Y"] for point in polygon]
|
||||
x_min, x_max = min(xs), max(xs)
|
||||
y_min, y_max = min(ys), max(ys)
|
||||
w = x_max - x_min
|
||||
h = y_max - y_min
|
||||
center_x = (x_min + x_max) // 2
|
||||
center_y = (y_min + y_max) // 2
|
||||
|
||||
# 过滤左右边缘区域的文字（可能是水印或logo）
|
||||
if center_x > width * 0.85 or center_x < width * 0.15:
|
||||
continue
|
||||
if detection.get('Confidence', 0) <= 75:
|
||||
continue
|
||||
|
||||
detected_text = detection.get("DetectedText", "")
|
||||
if not check_subtitle_quality_enhanced(detected_text) and detection.get('Confidence', 0) <= 90:
|
||||
continue
|
||||
if len(detected_text) < 5:
|
||||
continue
|
||||
# 只要中文字幕：没有中文字符的(纯英文/数字)不当作候选字幕区域
|
||||
if require_chinese and not has_chinese(detected_text):
|
||||
continue
|
||||
|
||||
# 水印检测：文本相似度>0.7且位置相近
|
||||
is_watermark = False
|
||||
for existing_text, pos_list in text_positions.items():
|
||||
similarity = SequenceMatcher(None, detected_text, existing_text).ratio()
|
||||
if similarity > 0.7:
|
||||
# 检查位置是否相近
|
||||
for pos in pos_list:
|
||||
if pos["idx"] != idx and abs(pos["y"] - center_y) < h:
|
||||
is_watermark = True
|
||||
watermark_logs.append(
|
||||
f"第{idx}帧可能的水印: {detected_text} (位置: {center_x}, {center_y})")
|
||||
logger.log(f"检测到可能的水印: {detected_text} (位置: {center_x}, {center_y})",
|
||||
program="merge")
|
||||
# 标记原文本为水印
|
||||
pos["is_watermark"] = True
|
||||
break
|
||||
if is_watermark:
|
||||
break
|
||||
|
||||
if is_watermark:
|
||||
continue
|
||||
|
||||
# 记录文本位置
|
||||
text_positions[detected_text].append({
|
||||
"text": detected_text,
|
||||
"x": center_x,
|
||||
"y": center_y,
|
||||
"is_watermark": False,
|
||||
"idx": idx
|
||||
})
|
||||
|
||||
# 存储候选区域
|
||||
candidate_regions.append({
|
||||
"x_min": x_min,
|
||||
"y_min": y_min,
|
||||
"x_max": x_max,
|
||||
"y_max": y_max,
|
||||
"width": w,
|
||||
"height": h
|
||||
})
|
||||
|
||||
# 如果没有候选区域，检查首尾帧是否有可能存在字幕
|
||||
if not candidate_regions:
|
||||
# 检查前 keep_num 张和后 keep_num 张是否有文字（并发）
|
||||
keep_num = 9
|
||||
n = len(image_files)
|
||||
check_indices = sorted(set(
|
||||
list(range(min(keep_num, n))) + list(range(max(0, n - keep_num), n))
|
||||
))
|
||||
fb_cache = _ocr_frames_concurrent(image_files, input_dir, check_indices, ocr_workers)
|
||||
has_text = False
|
||||
for idx in check_indices:
|
||||
ocr_result = fb_cache.get(idx)
|
||||
if ocr_result:
|
||||
ocr_data = json.loads(ocr_result)
|
||||
if "TextDetections" in ocr_data and ocr_data["TextDetections"]:
|
||||
has_text = True
|
||||
break
|
||||
|
||||
if not has_text:
|
||||
logger.log("保留的前后9张图片均未检测到任何文字，认为该视频没有字幕", level="warning", program="merge")
|
||||
return None, sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
# 无候选区域但有文字：回退到主区域检测
|
||||
main_region = detect_main_region(image_files, input_dir)
|
||||
x, y, w, h = main_region
|
||||
# 只取主区域下半部分
|
||||
return (0, y + h // 2, width, h // 2), sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
# 逆向清理：只清理画面上半部分（y < height*0.5）的水印候选区域
|
||||
# 下半部分即便被误判为水印也保留，避免把字幕字幕一并删掉
|
||||
confirmed_watermark_ys = {
|
||||
pos["y"]
|
||||
for pos_list in text_positions.values()
|
||||
for pos in pos_list
|
||||
if pos.get("is_watermark")
|
||||
}
|
||||
top_watermark_ys = {wy for wy in confirmed_watermark_ys if wy < height * 0.5}
|
||||
if top_watermark_ys:
|
||||
before = len(candidate_regions)
|
||||
candidate_regions = [
|
||||
r for r in candidate_regions
|
||||
if not any(
|
||||
abs((r["y_min"] + r["y_max"]) // 2 - wy) < 80
|
||||
for wy in top_watermark_ys
|
||||
)
|
||||
]
|
||||
removed = before - len(candidate_regions)
|
||||
if removed:
|
||||
logger.log(f"逆向移除 {removed} 个上半区水印候选区域", program="merge")
|
||||
|
||||
# 过滤过小的区域（高度<15）
|
||||
candidate_regions = [r for r in candidate_regions if r["height"] >= 15]
|
||||
|
||||
# 水印移除/高度过滤后可能把候选全部清空 → 判无字幕(否则下面 min() 会因空序列崩成"拼接失败")
|
||||
if not candidate_regions:
|
||||
logger.log("候选字幕区域被过滤后为空(高度过小/全是水印)，判无字幕", level="warning", program="merge")
|
||||
return None, sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
# 冗余判定：用"不同垂直位置簇"的数量，而非候选框总数。
|
||||
# 正常字幕会在多帧的同一位置重复出现，框数随采样帧数增长但集中在少数垂直位置；
|
||||
# 纯文本/水印过多的视频才会把文字散布到很多不同的垂直位置。
|
||||
region_cys = sorted((r["y_min"] + r["y_max"]) // 2 for r in candidate_regions)
|
||||
distinct_bands = 1 if region_cys else 0
|
||||
for _i in range(1, len(region_cys)):
|
||||
if region_cys[_i] - region_cys[_i - 1] > 80:
|
||||
distinct_bands += 1
|
||||
if distinct_bands >= bands_thr:
|
||||
logger.log(f"冗余文字过多（文字分散在{distinct_bands}个垂直位置，共{len(candidate_regions)}个候选区域），视频将被过滤",
|
||||
level="warning", program="merge")
|
||||
return None, sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
# 合并所有候选区域
|
||||
all_y_min = min(r["y_min"] for r in candidate_regions)
|
||||
all_y_max = max(r["y_max"] for r in candidate_regions)
|
||||
merged_h = all_y_max - all_y_min
|
||||
# 扩展边界
|
||||
expanded_y_min = max(0, int(all_y_min - 65))
|
||||
expanded_y_max = min(height, int(all_y_max + 65))
|
||||
final_h = expanded_y_max - expanded_y_min
|
||||
|
||||
# 如果合并区域过高（>60% 画面高度），用聚类找最大候选簇，避免退回全帧
|
||||
if final_h >= height * 0.6 and len(candidate_regions) >= 4:
|
||||
logger.log("合并区域过高，聚类寻找最大字幕区域簇", program="merge")
|
||||
best_cluster = []
|
||||
for region in candidate_regions:
|
||||
cy = (region["y_min"] + region["y_max"]) // 2
|
||||
cluster = [r for r in candidate_regions
|
||||
if abs((r["y_min"] + r["y_max"]) // 2 - cy) < 200]
|
||||
# 优先更大的簇；同等大小时选中心 y 更大的（靠近底部的字幕）
|
||||
if len(cluster) > len(best_cluster) or (
|
||||
len(cluster) == len(best_cluster) and best_cluster and
|
||||
sum((r["y_min"] + r["y_max"]) // 2 for r in cluster) >
|
||||
sum((r["y_min"] + r["y_max"]) // 2 for r in best_cluster)):
|
||||
best_cluster = cluster
|
||||
|
||||
if best_cluster:
|
||||
c_y_min = min(r["y_min"] for r in best_cluster)
|
||||
c_y_max = max(r["y_max"] for r in best_cluster)
|
||||
exp_y_min = max(0, c_y_min - 65)
|
||||
exp_y_max = min(height, c_y_max + 65)
|
||||
logger.log(f"聚类选定字幕区域: y={exp_y_min}-{exp_y_max}", program="merge")
|
||||
return (0, exp_y_min, width, exp_y_max - exp_y_min), sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
main_region = detect_main_region(image_files, input_dir)
|
||||
x, y, w, h = main_region
|
||||
return (0, y, width, h), sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
return (0, expanded_y_min, width, final_h), sample_indices, watermark_logs, candidate_regions
|
||||
|
||||
|
||||
# ==================== 拼图配置计算 ====================
|
||||
def calculate_optimal_tile_configuration(subtitle_region: Tuple[int, int, int, int],
|
||||
tile_width: int,
|
||||
tile_height: int,
|
||||
first_img_path: str,
|
||||
logger,
|
||||
max_base64_size: int = 10 * 1024 * 1024) -> Tuple[int, int]:
|
||||
"""
|
||||
计算最优的拼图配置（横向和纵向数量）。
|
||||
|
||||
:return: (tile_width, tile_height)
|
||||
"""
|
||||
x, y, w, h = subtitle_region
|
||||
BORDER_SIZE = 40
|
||||
tile_width_px = w + 2 * BORDER_SIZE
|
||||
tile_height_px = h + 2 * BORDER_SIZE
|
||||
|
||||
# 估算单个图块的大小（基于第一张图片）
|
||||
img = cv2.imread(first_img_path)
|
||||
if img is None:
|
||||
raise ValueError(f"无法读取图片: {first_img_path}")
|
||||
img_h, img_w = img.shape[:2]
|
||||
img_size = os.path.getsize(first_img_path)
|
||||
single_tile_size = (tile_width_px * tile_height_px) / (img_w * img_h) * img_size * 3 # RGB估算
|
||||
single_tile_base64 = single_tile_size * 4 // 3 # base64 增加 33%
|
||||
|
||||
# 每张合成图最多拼几块：硬上限(可配置) 与 base64 10MB 限制 取小。
|
||||
# 加大上限 → OCR 请求数下降(每张装更多帧)，但单次 OCR 图更大、且文字条更密，
|
||||
# 过大有 OCR 漏识别/超分辨率上限的风险，调大后务必抽查 SRT 质量。
|
||||
hard_cap = config.get_subtitle_config().get('max_tiles_per_composite', 40)
|
||||
try:
|
||||
hard_cap = int(hard_cap)
|
||||
except (TypeError, ValueError):
|
||||
hard_cap = 40
|
||||
max_tiles = min(hard_cap, int(max_base64_size // single_tile_base64))
|
||||
max_tiles = max(1, max_tiles)
|
||||
# 横向块数上限随总块数放宽(块多时仍想接近 0.5 的宽高比，避免拼成超细长条)
|
||||
w_cap = max(7, int(round((max_tiles * 0.5) ** 0.5)) + 1)
|
||||
logger.log(f"单个图块 base64 大小: {single_tile_base64 / 1024:.1f} KB", program="merge")
|
||||
logger.log(f"最大允许图块数量: {max_tiles} (硬上限 {hard_cap})", program="merge")
|
||||
|
||||
# 寻找最佳宽高比（接近 0.5，即横向多于纵向）
|
||||
best_config = (tile_width, tile_height) # 默认
|
||||
best_ratio_diff = float('inf')
|
||||
for w_tiles in range(1, min(max_tiles, w_cap) + 1):
|
||||
h_tiles = int(max_tiles / w_tiles)
|
||||
aspect = w_tiles / h_tiles
|
||||
diff = abs(aspect - 0.5)
|
||||
if diff < best_ratio_diff:
|
||||
best_ratio_diff = diff
|
||||
best_config = (w_tiles, h_tiles)
|
||||
|
||||
logger.log(f"推荐拼图配置: {best_config[0]} x {best_config[1]}", program="merge")
|
||||
return best_config
|
||||
|
||||
|
||||
# ==================== 音频提取（可复用） ====================
|
||||
def extract_audio_text(video_path: str, logger) -> Dict[str, str]:
|
||||
"""
|
||||
从视频中提取开头和结尾的音频文字。
|
||||
|
||||
:return: {'first_3s': str, 'last_3s': str}
|
||||
"""
|
||||
audio_text = {}
|
||||
if not video_path or not os.path.exists(video_path):
|
||||
return audio_text
|
||||
|
||||
logger.log("开始提取视频中的音频文字...", program="merge")
|
||||
# 从配置读取 whisper 模型大小（默认 small）
|
||||
model_size = config.get_audio_config().get('model_size', 'small')
|
||||
whisper_manager.load_model(model_size)
|
||||
|
||||
temp_audio_path = os.path.join(tempfile.gettempdir(),
|
||||
f"temp_audio_{os.path.basename(video_path)}.wav")
|
||||
try:
|
||||
total_duration = whisper_manager.extract_audio_from_video(video_path, temp_audio_path, logger)
|
||||
if total_duration is None:
|
||||
logger.log("音频提取失败，跳过音频文字识别", program="merge")
|
||||
return audio_text
|
||||
|
||||
if total_duration >= 5:
|
||||
first_5s_text = whisper_manager.extract_audio_text_with_whisper(
|
||||
temp_audio_path, 0, min(5, total_duration), logger
|
||||
)
|
||||
audio_text["first_3s"] = first_5s_text
|
||||
logger.log(f"前五秒音频文字: {first_5s_text}", program="merge")
|
||||
|
||||
if total_duration >= 10:
|
||||
last_5s_start = int(total_duration - 5)
|
||||
last_5s_text = whisper_manager.extract_audio_text_with_whisper(
|
||||
temp_audio_path, last_5s_start, total_duration, logger
|
||||
)
|
||||
audio_text["last_3s"] = last_5s_text
|
||||
logger.log(f"后五秒音频文字: {last_5s_text}", program="merge")
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temp_audio_path)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return audio_text
|
||||
|
||||
|
||||
# ==================== 主处理函数 ====================
|
||||
def process_images(input_dir, output_dir, logger, tile_width=3, tile_height=6, keep_num=9, interval_seconds=None,
|
||||
video_path=None):
|
||||
"""
|
||||
处理图片拼接并生成元数据，自动计算最优拼图配置
|
||||
|
||||
参数:
|
||||
input_dir: 输入图片目录
|
||||
output_dir: 输出目录
|
||||
keep_num: 保留的首尾图片数量
|
||||
"""
|
||||
# 创建输出目录
|
||||
logger.start_program("merge")
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
composite_dir = os.path.join(output_dir, "composite")
|
||||
os.makedirs(composite_dir, exist_ok=True)
|
||||
|
||||
# 获取所有图片文件并排序
|
||||
image_files = [f for f in os.listdir(input_dir) if f.lower().endswith(('.png', '.jpg', '.jpeg'))]
|
||||
image_files.sort(key=natural_sort_key)
|
||||
total_images = len(image_files)
|
||||
duration = total_images / 3
|
||||
# 确定字幕区域（通过OCR检测）
|
||||
subtitle_region, sample_indices, watermark, candidate_regions = detect_subtitle_region(image_files, input_dir)
|
||||
|
||||
if subtitle_region is None:
|
||||
logger.end_program("merge")
|
||||
return duration, None, None, None, None, None, sample_indices, watermark, candidate_regions
|
||||
logger.log(f"通过OCR检测确定字幕区域: {subtitle_region}", program="merge")
|
||||
first_img_path = os.path.join(input_dir, image_files[0])
|
||||
|
||||
# 提取音频文字（如果提供了视频路径）
|
||||
audio_text = {}
|
||||
if video_path and os.path.exists(video_path):
|
||||
logger.log("开始提取视频中的音频文字...", program="merge")
|
||||
model_size = config.get_audio_config().get('model_size', 'small')
|
||||
whisper_manager.load_model(model_size)
|
||||
# 创建临时音频文件
|
||||
temp_audio_path = os.path.join(tempfile.gettempdir(), f"temp_audio_{os.path.basename(video_path)}.wav")
|
||||
total_duration = whisper_manager.extract_audio_from_video(video_path, temp_audio_path, logger)
|
||||
# 从视频提取音频
|
||||
if total_duration is not None:
|
||||
|
||||
# 提取前五秒文字
|
||||
if total_duration >= 5:
|
||||
first_5s_text = whisper_manager.extract_audio_text_with_whisper(temp_audio_path, 0,
|
||||
min(5, total_duration), logger)
|
||||
audio_text["first_3s"] = first_5s_text
|
||||
logger.log(f"前五秒音频文字: {first_5s_text}", program="merge")
|
||||
|
||||
# 提取后五秒文字
|
||||
if total_duration >= 10:
|
||||
last_5s_start = int(total_duration - 5)
|
||||
last_5s_text = whisper_manager.extract_audio_text_with_whisper(temp_audio_path, last_5s_start,
|
||||
total_duration, logger)
|
||||
audio_text["last_3s"] = last_5s_text
|
||||
logger.log(f"后五秒音频文字: {last_5s_text}", program="merge")
|
||||
|
||||
# 清理临时音频文件
|
||||
try:
|
||||
os.unlink(temp_audio_path)
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
logger.log("音频提取失败，跳过音频文字识别", program="merge")
|
||||
# 边框设置
|
||||
BORDER_SIZE = 40
|
||||
|
||||
# 保留前keep_num张和最后keep_num张图片
|
||||
kept_first = image_files[:keep_num]
|
||||
kept_last = image_files[-keep_num:] if len(image_files) > 2 * keep_num else []
|
||||
|
||||
# 需要处理的图片（中间部分）
|
||||
process_files = image_files[keep_num:-keep_num] if len(image_files) > 2 * keep_num else image_files[keep_num:]
|
||||
|
||||
# 归一化图块像素预算：高分辨率帧或过高的字幕区域会让单个图块的 base64 过大，
|
||||
# 撑到 10MB 上限后每张合成图能塞的块数骤减（从~32 掉到 5~8），导致合成图（=OCR请求）
|
||||
# 数量激增 3~6 倍。按基准像素量（约 1080p 单字幕条）等比缩小裁剪后的字幕条，使每张
|
||||
# 合成图的块数稳定在~32，OCR 成本与源分辨率/区域高度无关；文字分辨率仍足够 OCR。
|
||||
x0, y0, w0, h0 = subtitle_region
|
||||
no_composite = config.get_subtitle_config().get('no_composite', False)
|
||||
if no_composite:
|
||||
# 不拼接：每个字幕条单独成图(tile 1×1)、不降采样(单条 base64 很小，全分辨率最准)。
|
||||
# 下游 OCR 映射/网格搜索/去重完全复用，只是每张图=1 条、OCR 次数=帧数。
|
||||
tile_scale = 1.0
|
||||
w_scaled, h_scaled = w0, h0
|
||||
tile_width, tile_height = 1, 1
|
||||
logger.log("不拼接模式：每个字幕条单独 OCR(tile 1×1，全分辨率)", program="merge")
|
||||
else:
|
||||
# 归一化图块像素预算：高分辨率帧或过高的字幕区域会让单个图块的 base64 过大，
|
||||
# 撑到 10MB 上限后每张合成图能塞的块数骤减（从~32 掉到 5~8），导致合成图（=OCR请求）
|
||||
# 数量激增 3~6 倍。按基准像素量（约 1080p 单字幕条）等比缩小裁剪后的字幕条，使每张
|
||||
# 合成图的块数稳定在~32，OCR 成本与源分辨率/区域高度无关；文字分辨率仍足够 OCR。
|
||||
TARGET_TILE_AREA = 1920 * 240
|
||||
tile_scale = min(1.0, math.sqrt(TARGET_TILE_AREA / float(max(1, w0 * h0))))
|
||||
w_scaled = max(1, int(round(w0 * tile_scale)))
|
||||
h_scaled = max(1, int(round(h0 * tile_scale)))
|
||||
if tile_scale < 1.0:
|
||||
logger.log(f"字幕条降采样: {w0}x{h0} → {w_scaled}x{h_scaled} (scale={tile_scale:.2f})", program="merge")
|
||||
tile_width, tile_height = calculate_optimal_tile_configuration(
|
||||
(x0, y0, w_scaled, h_scaled), tile_width, tile_height, first_img_path, logger)
|
||||
|
||||
# 初始化元数据
|
||||
metadata = {
|
||||
"tile_width": tile_width,
|
||||
"tile_height": tile_height,
|
||||
"subtitle_region": subtitle_region,
|
||||
"border_size": BORDER_SIZE,
|
||||
"interval_seconds": interval_seconds,
|
||||
"audio_text": audio_text,
|
||||
"kept_first": [],
|
||||
"kept_last": [],
|
||||
"composites": []
|
||||
}
|
||||
|
||||
# 复制保留的图片到composite文件夹
|
||||
kept_first_new_names = []
|
||||
for i, img in enumerate(kept_first):
|
||||
src = os.path.join(input_dir, img)
|
||||
dst_filename = f"001_{i + 1:02d}_original_{img}"
|
||||
dst = os.path.join(composite_dir, dst_filename)
|
||||
|
||||
if not os.path.exists(dst):
|
||||
with Image.open(src) as im:
|
||||
bordered_img = Image.new('RGB',
|
||||
(im.width + 2 * BORDER_SIZE, im.height + 2 * BORDER_SIZE),
|
||||
(255, 255, 255))
|
||||
bordered_img.paste(im, (BORDER_SIZE, BORDER_SIZE))
|
||||
|
||||
if img.lower().endswith('.png'):
|
||||
bordered_img.save(dst, format='PNG')
|
||||
else:
|
||||
bordered_img.save(dst, format='JPEG', quality=95)
|
||||
|
||||
kept_first_new_names.append(dst_filename)
|
||||
|
||||
metadata["kept_first"] = kept_first_new_names
|
||||
|
||||
# 处理中间图片进行拼接
|
||||
if process_files:
|
||||
x, y, w, h = subtitle_region # 原始帧上的裁剪坐标
|
||||
ws, hs = w_scaled, h_scaled # 降采样后的图块尺寸（拼接/坐标用此）
|
||||
|
||||
# 计算图块和合成图尺寸（基于降采样后的尺寸）
|
||||
tile_img_width = ws + 2 * BORDER_SIZE
|
||||
tile_img_height = hs + 2 * BORDER_SIZE
|
||||
composite_width = tile_width * tile_img_width
|
||||
composite_height = tile_height * tile_img_height
|
||||
|
||||
# 分批：每批 = 一张合成图
|
||||
tiles_per_composite = tile_width * tile_height
|
||||
batches = []
|
||||
for i in range(0, len(process_files), tiles_per_composite):
|
||||
batches.append((len(batches) + 1, process_files[i:i + tiles_per_composite]))
|
||||
|
||||
def _build_composite(batch):
|
||||
"""处理一张合成图：读源帧→裁剪→贴块→写盘。返回 (序号, composite_data)。
|
||||
热点：逐帧读 1080p JPEG + 裁剪 + 写盘，原为单线程；这里并行化（PIL 编解码
|
||||
在 C 层释放 GIL，线程池能真正并行，按核数提速）。"""
|
||||
composite_num, batch_files = batch
|
||||
composite_img = Image.new('RGB', (composite_width, composite_height), (255, 255, 255))
|
||||
composite_filename = f"002_{composite_num:04d}_composite.jpg"
|
||||
composite_data = {
|
||||
"composite_id": f"composite_{composite_num:04d}",
|
||||
"file_name": composite_filename,
|
||||
"tiles": []
|
||||
}
|
||||
for pos in range(tiles_per_composite):
|
||||
row = pos // tile_width
|
||||
col = pos % tile_width
|
||||
x_pos = col * tile_img_width
|
||||
y_pos = row * tile_img_height
|
||||
|
||||
if pos < len(batch_files):
|
||||
img_name = batch_files[pos]
|
||||
img_path = os.path.join(input_dir, img_name)
|
||||
with Image.open(img_path) as img:
|
||||
subtitle_img = img.crop((x, y, x + w, y + h))
|
||||
if tile_scale < 1.0:
|
||||
subtitle_img = subtitle_img.resize((ws, hs))
|
||||
# 合成图背景已是白色，直接把字幕条贴到带边框的偏移位置
|
||||
composite_img.paste(subtitle_img, (x_pos + BORDER_SIZE, y_pos + BORDER_SIZE))
|
||||
composite_data["tiles"].append({
|
||||
"position": pos,
|
||||
"x": x_pos + BORDER_SIZE,
|
||||
"y": y_pos + BORDER_SIZE,
|
||||
"width": ws,
|
||||
"height": hs,
|
||||
"source_image": img_name,
|
||||
"is_blank": False
|
||||
})
|
||||
else:
|
||||
# 空白块：合成图背景本就是白色，无需粘贴
|
||||
composite_data["tiles"].append({
|
||||
"position": pos,
|
||||
"x": x_pos + BORDER_SIZE,
|
||||
"y": y_pos + BORDER_SIZE,
|
||||
"width": ws,
|
||||
"height": hs,
|
||||
"source_image": None,
|
||||
"is_blank": True
|
||||
})
|
||||
|
||||
output_path = os.path.join(composite_dir, composite_filename)
|
||||
composite_img.save(output_path, format='JPEG', quality=95)
|
||||
return composite_num, composite_data
|
||||
|
||||
# 并行处理所有合成图，结果按序号排序回填（顺序不能乱，元数据要对得上时间轴）
|
||||
merge_workers = config.get_subtitle_config().get('merge_workers') or (os.cpu_count() or 8)
|
||||
merge_workers = max(1, min(int(merge_workers), len(batches)))
|
||||
results = []
|
||||
with ThreadPoolExecutor(max_workers=merge_workers) as executor:
|
||||
for r in executor.map(_build_composite, batches):
|
||||
results.append(r)
|
||||
results.sort(key=lambda t: t[0])
|
||||
metadata["composites"] = [cd for _, cd in results]
|
||||
|
||||
# 处理最后保留的图片
|
||||
kept_last_new_names = []
|
||||
for i, img in enumerate(kept_last):
|
||||
src = os.path.join(input_dir, img)
|
||||
dst_filename = f"003_{i + 1:02d}_original_{img}"
|
||||
dst = os.path.join(composite_dir, dst_filename)
|
||||
|
||||
if not os.path.exists(dst):
|
||||
with Image.open(src) as im:
|
||||
bordered_img = Image.new('RGB',
|
||||
(im.width + 2 * BORDER_SIZE, im.height + 2 * BORDER_SIZE),
|
||||
(255, 255, 255))
|
||||
bordered_img.paste(im, (BORDER_SIZE, BORDER_SIZE))
|
||||
|
||||
if img.lower().endswith('.png'):
|
||||
bordered_img.save(dst, format='PNG')
|
||||
else:
|
||||
bordered_img.save(dst, format='JPEG', quality=95)
|
||||
|
||||
kept_last_new_names.append(dst_filename)
|
||||
|
||||
metadata["kept_last"] = kept_last_new_names
|
||||
|
||||
# 保存元数据（保存到 composite_dir，避免被清理时删除）
|
||||
metadata_path = os.path.join(composite_dir, "metadata.json")
|
||||
with open(metadata_path, 'w') as f:
|
||||
json.dump(metadata, f, indent=2)
|
||||
|
||||
logger.log(f"处理完成，拼图配置: {tile_width}x{tile_height}", program="merge")
|
||||
logger.log(f"共生成 {len(metadata['composites'])} 张合成图", program="merge")
|
||||
logger.log(f"元数据已保存到: {metadata_path}", program="merge")
|
||||
logger.end_program("merge")
|
||||
return duration, subtitle_region, tile_width, tile_height, len(
|
||||
metadata['composites']), audio_text, sample_indices, watermark, candidate_regions
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 示例用法
|
||||
input_directory = "input_images"
|
||||
output_directory = "output"
|
||||
from log.log_manager import logger # 假设已有全局 logger
|
||||
|
||||
result = process_images(input_directory, output_directory, logger)
|
||||
471
tool/process.py
Normal file
471
tool/process.py
Normal file
@@ -0,0 +1,471 @@
|
||||
from config.config_utils import config, get_ffprobe
|
||||
import argparse
|
||||
import hashlib
|
||||
import logging
|
||||
import subprocess
|
||||
import shutil
|
||||
import sys
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
from tool.log.log_manager import UnifiedLogger
|
||||
from tool.VideoProcessing.multithread_algo import split_frame
|
||||
|
||||
|
||||
def derive_video_name(cos_path: str) -> str:
|
||||
"""把 COS 完整路径变成本地工作目录/文件名用的唯一标识:
|
||||
<原文件名(不含扩展名)>_<完整路径MD5前8位>。
|
||||
|
||||
不能只用文件名 —— 源数据里大量不同父目录复用同样的数字文件名(如
|
||||
cos://bkt/0021/17688/00046.mp4 和 cos://bkt/0021/17690/00046.mp4), 纯文件名当本地
|
||||
标识会导致两个不同视频被并发处理到时互相踩踏同一份本地文件(下载/抽帧/拼接的中间文件
|
||||
全挤在同一个 ./split/00046 目录), 引发 "Directory not empty" 崩溃和一连串连锁失败
|
||||
(实测占某台机器全部 error 记录的 100%)。
|
||||
|
||||
保留原文件名前缀是为了日志/目录名还能看出是哪个视频, 不至于变成纯哈希、丧失可读性；
|
||||
main.py 里 process_video_task() 必须调用同一个函数(而不是各自按文件名重算), 两处
|
||||
对同一个 cos_path 才会算出同一个 video_name。"""
|
||||
filename = os.path.basename(cos_path.strip())
|
||||
stem = os.path.splitext(filename)[0]
|
||||
digest = hashlib.md5(cos_path.strip().encode("utf-8")).hexdigest()[:8]
|
||||
return f"{stem}_{digest}"
|
||||
|
||||
|
||||
def _probe_duration(video_path):
|
||||
"""用 ffprobe 取视频时长(秒)；失败返回 None。
|
||||
先取流时长, 再取容器(format)时长——有些封装(尤其AV1/部分mp4)流里没duration、只在format里。"""
|
||||
# ① 流时长
|
||||
try:
|
||||
r = subprocess.run([
|
||||
get_ffprobe(), '-v', 'error', '-select_streams', 'v:0',
|
||||
'-show_entries', 'stream=duration', '-of', 'json', video_path
|
||||
], capture_output=True, text=True)
|
||||
d = float(json.loads(r.stdout)['streams'][0].get('duration', 0) or 0)
|
||||
if d > 0:
|
||||
return d
|
||||
except Exception:
|
||||
pass
|
||||
# ② 容器时长(兜底)
|
||||
try:
|
||||
r = subprocess.run([
|
||||
get_ffprobe(), '-v', 'error',
|
||||
'-show_entries', 'format=duration', '-of', 'json', video_path
|
||||
], capture_output=True, text=True)
|
||||
d = float(json.loads(r.stdout).get('format', {}).get('duration', 0) or 0)
|
||||
if d > 0:
|
||||
return d
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
|
||||
def _append_long_video(cos_path, list_path):
|
||||
"""把被"视频过长"过滤掉的视频 COS 路径追加到一份 txt，供以后(GPU机器/放大上限)补跑。
|
||||
进程池下多个 worker 进程会并发写同一文件：用 O_APPEND 单行追加(POSIX 下小写原子)，
|
||||
失败也只记日志、绝不影响主流程。"""
|
||||
if not cos_path or not list_path:
|
||||
return
|
||||
try:
|
||||
with open(list_path, 'a', encoding='utf-8') as f:
|
||||
f.write(cos_path.rstrip('\n') + '\n')
|
||||
except Exception:
|
||||
pass
|
||||
from tool.merge import process_images
|
||||
from tool.getsrt import processsrt, OcrIncompleteError
|
||||
|
||||
# 配置日志
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format='%(asctime)s - %(levelname)s - %(message)s',
|
||||
handlers=[
|
||||
logging.FileHandler('video_processing.log'),
|
||||
logging.StreamHandler(sys.stdout)
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def delete_local_folder(folder_path):
|
||||
"""
|
||||
删除本地文件夹，但保留子文件夹composite及其内容
|
||||
:param folder_path: 文件夹路径
|
||||
:return:
|
||||
"""
|
||||
try:
|
||||
if not os.path.exists(folder_path):
|
||||
logging.warning(f"文件夹不存在，无需删除: {folder_path}")
|
||||
return True
|
||||
|
||||
# 定义要保留的子文件夹路径
|
||||
composite_path = os.path.join(folder_path, 'composite')
|
||||
|
||||
# 如果composite子文件夹存在，先临时保存它
|
||||
temp_dir = None
|
||||
if os.path.exists(composite_path):
|
||||
# 创建一个临时目录来保存composite文件夹
|
||||
temp_dir = os.path.join(os.path.dirname(folder_path), f"temp_composite_{os.getpid()}")
|
||||
shutil.move(composite_path, temp_dir)
|
||||
|
||||
# 删除整个文件夹
|
||||
shutil.rmtree(folder_path)
|
||||
|
||||
# 如果之前有composite文件夹，将其移回原位
|
||||
if temp_dir and os.path.exists(temp_dir):
|
||||
# rmtree 删除了父目录本身，需要先重建
|
||||
os.makedirs(folder_path, exist_ok=True)
|
||||
shutil.move(temp_dir, composite_path)
|
||||
logging.info(f"已删除本地文件夹内容，但保留了composite子文件夹: {folder_path}")
|
||||
else:
|
||||
logging.info(f"已删除本地文件夹: {folder_path}")
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
logging.error(f"删除文件夹失败: {e}")
|
||||
# 尝试恢复临时文件夹以防出错
|
||||
if temp_dir and os.path.exists(temp_dir):
|
||||
try:
|
||||
os.makedirs(folder_path, exist_ok=True)
|
||||
shutil.move(temp_dir, os.path.join(folder_path, 'composite'))
|
||||
except Exception as restore_error:
|
||||
logging.error(f"恢复composite文件夹失败: {restore_error}")
|
||||
return False
|
||||
|
||||
|
||||
def download_from_cos(cos_target, cos_source, coscli_path, cos_config_path, local_dest_path):
|
||||
"""
|
||||
从COS下载文件到 local_dest_path。
|
||||
:param cos_target: 目录
|
||||
:param cos_source: 完整 COS 路径
|
||||
:param coscli_path: cos exe文件地址
|
||||
:param cos_config_path: cos 配置文件
|
||||
:param local_dest_path: 下载到的本地目标路径 —— 必须是调用方(process_single_video)按
|
||||
derive_video_name() 算出的 collision-safe 路径, 不能在这里自己按 os.path.basename(cos_source)
|
||||
重新拼一个 './video/<文件名>' —— 之前就是这么写的, 跟调用方检查是否存在用的 local_video_path
|
||||
(带路径MD5后缀)对不上, 下载明明成功了、下一步却判"文件不存在", 100%必现。
|
||||
:return:
|
||||
"""
|
||||
try:
|
||||
# cos_source 是完整 COS 路径时直接使用，否则拼接目录
|
||||
full_cos_path = cos_source if cos_source.startswith("cos://") else f"{cos_target}{cos_source}"
|
||||
cmd = [f"{coscli_path}", "-c", f"{cos_config_path}", "cp",
|
||||
full_cos_path, local_dest_path]
|
||||
# 执行命令（带超时：网络卡住时不会永久阻塞 worker，超时即判下载失败）
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, timeout=1800)
|
||||
|
||||
if result.returncode == 0:
|
||||
logging.info(f"成功下载到: {local_dest_path}")
|
||||
return cos_source
|
||||
else:
|
||||
logging.error(f"下载失败: {result.stderr}")
|
||||
return False
|
||||
|
||||
except subprocess.TimeoutExpired:
|
||||
logging.error(f"下载超时(>1800s)，跳过: {cos_source}")
|
||||
return False
|
||||
except Exception as e:
|
||||
logging.error(f"下载过程中出错: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def process_single_video(cos_source, cos_target):
|
||||
"""
|
||||
单个视频处理流程
|
||||
:param cos_source: 单个视频名称
|
||||
:param cos_target: 上传位置
|
||||
:return: True 处理成功, False 处理失败
|
||||
"""
|
||||
# 1. 读取参数
|
||||
split_frame_config = config.get_splitframe_config()
|
||||
download_flag = split_frame_config.get('download')
|
||||
coscli_path = split_frame_config.get('coscli_path')
|
||||
cos_config_path = split_frame_config.get('cosconfig_path')
|
||||
filename = os.path.basename(cos_source)
|
||||
root_path = "./video"
|
||||
video_name = derive_video_name(cos_source)
|
||||
local_video_path = os.path.join(root_path, video_name + os.path.splitext(filename)[1])
|
||||
|
||||
# 使用线程安全的日志器，避免多线程下重置单例导致状态混乱
|
||||
logger = UnifiedLogger()
|
||||
logger.reset()
|
||||
logger.init_logging(f"log/{video_name}.json")
|
||||
logger.start_program("main")
|
||||
|
||||
try:
|
||||
return _process_video_pipeline(
|
||||
cos_source, cos_target, video_name, local_video_path,
|
||||
split_frame_config, download_flag, coscli_path, cos_config_path, logger
|
||||
)
|
||||
except OcrIncompleteError as e:
|
||||
# 不丢帧: OCR没全成功 → 不产出带洞SRT, 把视频记到"OCR失败待重跑"列表, 判失败(以后用另一jsonl补跑)
|
||||
try:
|
||||
fail_list = config.get_subtitle_config().get('ocr_fail_list', './OCR失败待重跑.txt')
|
||||
_append_long_video(cos_source, fail_list)
|
||||
except Exception:
|
||||
pass
|
||||
logger.log(f"视频{video_name} OCR未完成(不丢帧→不产出带洞SRT, 已记入待重跑): {e}",
|
||||
level="error", program="main")
|
||||
_save_error_metadata(
|
||||
cos_source, video_name, error_reason=f"OcrIncomplete: {e}",
|
||||
download_flag=download_flag, interval_seconds=split_frame_config.get('interval_seconds', 0.33)
|
||||
)
|
||||
logger.end_program("main")
|
||||
logger.save_logs()
|
||||
return False
|
||||
except Exception as e:
|
||||
logger.log(f"视频{video_name}处理流程异常: {type(e).__name__}: {e}", level="error", program="main")
|
||||
_save_error_metadata(
|
||||
cos_source, video_name, error_reason=f"{type(e).__name__}: {e}",
|
||||
download_flag=download_flag, interval_seconds=split_frame_config.get('interval_seconds', 0.33)
|
||||
)
|
||||
logger.end_program("main")
|
||||
logger.save_logs()
|
||||
return False
|
||||
|
||||
|
||||
def _process_video_pipeline(cos_source, cos_target, video_name, local_video_path,
|
||||
split_frame_config, download_flag, coscli_path, cos_config_path, logger):
|
||||
"""
|
||||
视频处理内部流水线，将各阶段错误统一记录并抛出
|
||||
"""
|
||||
# 2. 进行数据下载
|
||||
logger.step(video_name, "开始")
|
||||
if download_flag:
|
||||
logger.log("从COS下载视频...", program="main")
|
||||
if not os.path.exists(local_video_path):
|
||||
if not download_from_cos(cos_target, cos_source, coscli_path, cos_config_path, local_video_path):
|
||||
raise RuntimeError(f"视频{video_name}下载失败")
|
||||
logger.step(video_name, "下载完成")
|
||||
|
||||
# 检查文件是否存在
|
||||
if not os.path.exists(local_video_path):
|
||||
raise FileNotFoundError(f"视频{video_name}不存在: {local_video_path}")
|
||||
|
||||
# 判断字幕文件是否存在
|
||||
name = os.path.basename(local_video_path).split(".")[0]
|
||||
if os.path.exists(f"./split/{name}/composite/{name}.srt"):
|
||||
return True
|
||||
|
||||
# 视频时长过滤（抽帧前 ffprobe 取时长，省掉抽帧/拼接/OCR；也避免超长视频撑爆内存/卡死worker）
|
||||
_sub_cfg = config.get_subtitle_config()
|
||||
min_dur = _sub_cfg.get('min_duration_seconds', 300)
|
||||
max_dur = _sub_cfg.get('max_duration_seconds', 5400) # 默认1.5小时上限
|
||||
vdur = _probe_duration(local_video_path)
|
||||
# 在【删视频前】把时长存进日志stat: ① ffprobe能处理AV1(cv2不行) ② delete_video删源视频后main.py就读不到了
|
||||
if vdur is not None and vdur > 0:
|
||||
logger.set_stat("video_duration", round(vdur, 2))
|
||||
if vdur is not None and 0 < vdur < min_dur:
|
||||
logger.step(video_name, "已过滤", f"视频过短 {vdur:.0f}s < {min_dur}s")
|
||||
raise ValueError(f"视频{video_name}被过滤: 视频过短({vdur:.0f}秒, 阈值{min_dur}秒)")
|
||||
if vdur is not None and max_dur and vdur > max_dur:
|
||||
logger.step(video_name, "已过滤", f"视频过长 {vdur:.0f}s > {max_dur}s")
|
||||
# 把超长视频的 COS 路径单独留一份，供以后在 GPU 机器/放大上限后补跑
|
||||
# (当前 run 仍记 数据.jsonl 跳过它，不占 worker)
|
||||
long_list = _sub_cfg.get('long_video_list', './超长待重跑.txt')
|
||||
_append_long_video(cos_source, long_list)
|
||||
raise ValueError(f"视频{video_name}被过滤: 视频过长({vdur:.0f}秒, 阈值{max_dur}秒)")
|
||||
|
||||
# 3. 进行视频抽帧
|
||||
interval_seconds = split_frame_config.get('interval_seconds', 0.33)
|
||||
try:
|
||||
_t = time.time()
|
||||
duration, planned_frames, actual_frames, backend = split_frame(local_video_path, interval_seconds)
|
||||
logger.set_stat("frame_count", actual_frames)
|
||||
logger.log("抽帧成功", program="main")
|
||||
logger.step(video_name, "抽帧完成",
|
||||
f"{actual_frames}帧, {backend}, {time.time()-_t:.0f}s, 视频{duration:.0f}s")
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"视频{video_name}抽帧处理失败: {e}") from e
|
||||
|
||||
# 抽帧成功但一帧没出: 视频没被 min/max_duration_seconds 过滤掉、真去抽了、但结果是空——
|
||||
# 不是"没字幕"的正常情况, 是抽帧本身出了问题(比如时长探测失败导致期望帧数算成0, 见
|
||||
# split_frame_ffmpeg 的时长兜底), 应该报错让人查, 而不是让空帧列表流进 process_images()
|
||||
# 炸出一个看不懂的 "list index out of range"。
|
||||
if actual_frames == 0:
|
||||
raise RuntimeError(f"视频{video_name}抽帧结果为0帧(时长{duration:.0f}s), 抽帧本身有问题, 非正常无字幕")
|
||||
|
||||
try:
|
||||
_t = time.time()
|
||||
result = process_images(
|
||||
f"./split/{video_name}", f"./split/{video_name}", logger,
|
||||
interval_seconds=interval_seconds, video_path=local_video_path)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"视频{video_name}图片拼接失败: {e}") from e
|
||||
|
||||
duration, subtitle_region, tile_width, tile_height, compositenum, audio_text, sample_indices, watermark, candidate_regions = result
|
||||
if subtitle_region is None:
|
||||
logger.step(video_name, "已过滤", "未检测到字幕区域")
|
||||
raise ValueError(f"视频{video_name}被过滤: 未检测到字幕区域")
|
||||
logger.step(video_name, "拼接完成", f"{compositenum}张合成图, {time.time()-_t:.0f}s")
|
||||
|
||||
# 4. 删除本地抽帧文件（保留 composite 子文件夹）
|
||||
# delete_folder = os.path.join("./split", video_name)
|
||||
# if not delete_local_folder(delete_folder):
|
||||
# logger.log("本地文件夹删除失败", level="warning", program="main")
|
||||
|
||||
delete_flag = split_frame_config.get('delete_video')
|
||||
if delete_flag:
|
||||
try:
|
||||
if os.path.exists(local_video_path):
|
||||
os.remove(local_video_path)
|
||||
logger.log(f"已删除本地视频文件: {local_video_path}", program="main")
|
||||
except OSError as e:
|
||||
logger.log(f"删除视频文件失败: {e}", level="warning", program="main")
|
||||
|
||||
# 5. 转字幕
|
||||
composite_path = os.path.join("./split", video_name)
|
||||
os.makedirs(composite_path, exist_ok=True)
|
||||
if not os.path.exists(composite_path):
|
||||
raise FileNotFoundError(f"视频{video_name}的 composite 文件夹不存在: {composite_path}")
|
||||
|
||||
try:
|
||||
_t = time.time()
|
||||
score, conf, srt_path, contents, best_grids, audio_frame, h = processsrt(
|
||||
composite_path, "ocr_result.jsonl", f"{video_name}.ocr.srt", interval_seconds, logger)
|
||||
_getsrt_elapsed = time.time() - _t
|
||||
except Exception as e:
|
||||
raise RuntimeError(f"视频{video_name}字幕生成失败: {e}") from e
|
||||
|
||||
# 7. 记录元数据
|
||||
if score is None:
|
||||
filter_reason = srt_path if isinstance(srt_path, str) else "未知原因"
|
||||
logger.step(video_name, "已过滤", str(filter_reason))
|
||||
logger.log(f"视频{video_name}被过滤，原因：{filter_reason}", level="warning", program="main")
|
||||
_save_filter_metadata(
|
||||
cos_source, video_name, filter_reason=filter_reason,
|
||||
download_flag=download_flag, interval_seconds=interval_seconds,
|
||||
duration=duration, subtitle_region=subtitle_region,
|
||||
tile_width=tile_width, tile_height=tile_height, compositenum=compositenum,
|
||||
audio_text=audio_text, sample_indices=sample_indices,
|
||||
watermark=watermark, candidate_regions=candidate_regions,
|
||||
contents=contents, best_grids=best_grids, audio_frame=audio_frame,
|
||||
h=h, score=score, conf=conf
|
||||
)
|
||||
logger.end_program("main")
|
||||
logger.save_logs()
|
||||
return False
|
||||
|
||||
_save_success_metadata(
|
||||
cos_source, video_name, srt_path=srt_path,
|
||||
download_flag=download_flag, interval_seconds=interval_seconds,
|
||||
duration=duration, subtitle_region=subtitle_region,
|
||||
tile_width=tile_width, tile_height=tile_height, compositenum=compositenum,
|
||||
audio_text=audio_text, sample_indices=sample_indices,
|
||||
watermark=watermark, candidate_regions=candidate_regions,
|
||||
contents=contents, best_grids=best_grids, audio_frame=audio_frame,
|
||||
h=h, score=score, conf=conf
|
||||
)
|
||||
logger.step(video_name, "字幕完成",
|
||||
f"质量分{score:.0f}, {_getsrt_elapsed:.0f}s, 等待上传")
|
||||
logger.log(f"视频{video_name}处理完成", program="main")
|
||||
logger.end_program("main")
|
||||
logger.save_logs()
|
||||
return True
|
||||
|
||||
|
||||
def _build_metadata_common(download_flag, interval_seconds, duration, subtitle_region,
|
||||
tile_width, tile_height, compositenum, audio_text,
|
||||
sample_indices, watermark, candidate_regions,
|
||||
contents, best_grids, audio_frame, h):
|
||||
"""构建元数据的公共部分，避免代码重复"""
|
||||
best_grid_data = best_grids.get("composite") if best_grids else None
|
||||
return {
|
||||
"download_phase": {
|
||||
"download_flag": download_flag
|
||||
},
|
||||
"frame_extract": {
|
||||
"interval": interval_seconds,
|
||||
"duration": duration,
|
||||
"success": True
|
||||
},
|
||||
"subtitle_localize": {
|
||||
"sample_indices": sample_indices,
|
||||
"watermark": watermark,
|
||||
"candidate_regions": candidate_regions,
|
||||
"subtitle_position": {
|
||||
"dilation": "65",
|
||||
"final_region": subtitle_region
|
||||
},
|
||||
"audio_extract": audio_text
|
||||
},
|
||||
"composite_build": {
|
||||
"BORDER_SIZE": 40,
|
||||
"tile_width": tile_width,
|
||||
"tile_height": tile_height,
|
||||
"composite_num": compositenum,
|
||||
"success": True
|
||||
},
|
||||
"ocr_process": {
|
||||
"ocr_result": contents,
|
||||
"best_grid": {
|
||||
"composite": {
|
||||
"grid_position": best_grid_data[0] if best_grid_data else None,
|
||||
"grid_width": best_grid_data[1] if best_grid_data else None,
|
||||
"grid_height": best_grid_data[2] if best_grid_data else None,
|
||||
"grid_top_bound": best_grid_data[3] if best_grid_data else None,
|
||||
"grid_left_bound": best_grid_data[4] if best_grid_data else None
|
||||
}
|
||||
}
|
||||
},
|
||||
"frame_processing": {
|
||||
"audio_frame": audio_frame,
|
||||
"average_height": h,
|
||||
"merge_params": {
|
||||
"confidence_bottom_line": 75,
|
||||
"merge_srt_similarity_bottom_line": 0.75,
|
||||
"audio_similarity_bottom_line": 0.8
|
||||
}
|
||||
},
|
||||
"custom_score": None,
|
||||
"ocr_confidence": None
|
||||
}
|
||||
|
||||
|
||||
def _save_success_metadata(cos_source, video_name, srt_path, **kwargs):
|
||||
"""保存成功的元数据到 info 目录"""
|
||||
os.makedirs("./info", exist_ok=True)
|
||||
score = kwargs.pop("score")
|
||||
conf = kwargs.pop("conf")
|
||||
common = _build_metadata_common(**kwargs)
|
||||
common["custom_score"] = score
|
||||
common["ocr_confidence"] = conf
|
||||
common["srt_output"] = srt_path
|
||||
|
||||
metadata = {"main_process": {"input_video": cos_source}}
|
||||
metadata["main_process"].update(common)
|
||||
|
||||
metadata_js = os.path.join("./info", f"{video_name}.json")
|
||||
with open(metadata_js, 'w', encoding='utf-8') as f:
|
||||
json.dump(metadata, f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _save_filter_metadata(cos_source, video_name, filter_reason, **kwargs):
|
||||
"""保存被过滤视频的元数据到 info_error 目录"""
|
||||
os.makedirs("./info_error", exist_ok=True)
|
||||
score = kwargs.pop("score")
|
||||
conf = kwargs.pop("conf")
|
||||
common = _build_metadata_common(**kwargs)
|
||||
common["custom_score"] = score
|
||||
common["ocr_confidence"] = conf
|
||||
common["error"] = filter_reason
|
||||
|
||||
metadata = {"main_process": {"input_video": cos_source}}
|
||||
metadata["main_process"].update(common)
|
||||
|
||||
metadata_js = os.path.join("./info_error", f"{video_name}.json")
|
||||
with open(metadata_js, 'w', encoding='utf-8') as f:
|
||||
json.dump(metadata, f, ensure_ascii=False, indent=2)
|
||||
|
||||
|
||||
def _save_error_metadata(cos_source, video_name, error_reason, download_flag, interval_seconds):
|
||||
"""保存处理异常视频的元数据到 info_error 目录"""
|
||||
os.makedirs("./info_error", exist_ok=True)
|
||||
metadata = {
|
||||
"main_process": {
|
||||
"input_video": cos_source,
|
||||
"download_phase": {"download_flag": download_flag},
|
||||
"frame_extract": {"interval": interval_seconds},
|
||||
"error": error_reason
|
||||
}
|
||||
}
|
||||
metadata_js = os.path.join("./info_error", f"{video_name}.json")
|
||||
with open(metadata_js, 'w', encoding='utf-8') as f:
|
||||
json.dump(metadata, f, ensure_ascii=False, indent=2)
|
||||
227
tool/whisper_manager.py
Normal file
227
tool/whisper_manager.py
Normal file
@@ -0,0 +1,227 @@
|
||||
import os
|
||||
import hashlib
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
from collections import OrderedDict
|
||||
from typing import Optional
|
||||
|
||||
from pydub import AudioSegment
|
||||
|
||||
from config.config_utils import get_ffmpeg, get_ffprobe
|
||||
|
||||
# 让 pydub 也用项目自带的 ffmpeg/ffprobe(显式指定，不依赖 PATH)
|
||||
AudioSegment.converter = get_ffmpeg()
|
||||
AudioSegment.ffmpeg = get_ffmpeg()
|
||||
AudioSegment.ffprobe = get_ffprobe()
|
||||
|
||||
_MAX_CACHE_SIZE = 512
|
||||
|
||||
|
||||
class WhisperManager:
|
||||
_instance = None
|
||||
_lock = threading.Lock()
|
||||
|
||||
def __new__(cls):
|
||||
if cls._instance is None:
|
||||
with cls._lock:
|
||||
if cls._instance is None:
|
||||
cls._instance = super().__new__(cls)
|
||||
cls._instance._initialized = False
|
||||
return cls._instance
|
||||
|
||||
def __init__(self):
|
||||
if not self._initialized:
|
||||
self.model = None
|
||||
self.model_name = None
|
||||
self._load_lock = threading.Lock() # 仅保护模型加载，推理不加锁
|
||||
self._cache = OrderedDict()
|
||||
self._cache_lock = threading.Lock()
|
||||
self._initialized = True
|
||||
|
||||
@staticmethod
|
||||
def _cuda_available() -> bool:
|
||||
"""nvidia-smi 可用即认为有可用 GPU。"""
|
||||
try:
|
||||
r = subprocess.run(["nvidia-smi"], capture_output=True, timeout=10)
|
||||
return r.returncode == 0
|
||||
except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
|
||||
return False
|
||||
|
||||
@staticmethod
|
||||
def _gpu_count() -> int:
|
||||
"""数 GPU 张数（nvidia-smi -L 行数）。"""
|
||||
try:
|
||||
r = subprocess.run(["nvidia-smi", "-L"], capture_output=True, text=True, timeout=10)
|
||||
if r.returncode == 0:
|
||||
return sum(1 for ln in r.stdout.splitlines() if ln.strip().startswith("GPU "))
|
||||
except (FileNotFoundError, OSError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
return 0
|
||||
|
||||
def _resolve_device(self):
|
||||
"""决定 whisper 跑在 cpu 还是 gpu。读 config.audio.device：auto/cuda/gpu/cpu。
|
||||
device_index=auto/last 时用【最后一张卡】，与抽帧(用前 N-1 张)错开。
|
||||
返回 (device, compute_type, device_index)。"""
|
||||
try:
|
||||
from config.config_utils import config
|
||||
acfg = config.get_audio_config() or {}
|
||||
except Exception:
|
||||
acfg = {}
|
||||
dev = str(acfg.get("device", "auto")).lower()
|
||||
idx_cfg = acfg.get("device_index", "auto")
|
||||
ctype = str(acfg.get("compute_type", "float16"))
|
||||
if dev in ("cuda", "gpu") or (dev == "auto" and self._cuda_available()):
|
||||
if str(idx_cfg).lower() in ("auto", "last"):
|
||||
idx = max(0, self._gpu_count() - 1) # 用最后一张
|
||||
else:
|
||||
try:
|
||||
idx = int(idx_cfg)
|
||||
except (TypeError, ValueError):
|
||||
idx = 0
|
||||
return "cuda", ctype, idx
|
||||
return "cpu", "int8", 0
|
||||
|
||||
def load_model(self, model_name: str = "base"):
|
||||
"""加载 faster-whisper 模型：有 GPU 用 cuda(float16)，否则 CPU(int8)；GPU 失败自动回退 CPU。"""
|
||||
with self._load_lock:
|
||||
if self.model is None or self.model_name != model_name:
|
||||
from faster_whisper import WhisperModel
|
||||
device, ctype, idx = self._resolve_device()
|
||||
# CPU 推理限制内部线程数：防止进程池下每个 worker 的 whisper 各抓一堆核 → 线程爆炸。
|
||||
# whisper 只识别首尾各5秒,2 线程足够;可用 config.audio.cpu_threads 调整。
|
||||
try:
|
||||
from config.config_utils import config as _cfg
|
||||
cpu_threads = int((_cfg.get_audio_config() or {}).get("cpu_threads", 2) or 2)
|
||||
except Exception:
|
||||
cpu_threads = 2
|
||||
cpu_threads = max(1, cpu_threads)
|
||||
try:
|
||||
print(f"[WhisperManager] 加载 faster-whisper '{model_name}' (device={device}, {ctype})...")
|
||||
if device == "cuda":
|
||||
self.model = WhisperModel(model_name, device="cuda",
|
||||
device_index=idx, compute_type=ctype)
|
||||
else:
|
||||
self.model = WhisperModel(model_name, device="cpu",
|
||||
compute_type="int8", cpu_threads=cpu_threads)
|
||||
except Exception as e:
|
||||
print(f"[WhisperManager] GPU 加载失败({type(e).__name__}: {e})，回退 CPU(int8, cpu_threads={cpu_threads})")
|
||||
self.model = WhisperModel(model_name, device="cpu",
|
||||
compute_type="int8", cpu_threads=cpu_threads)
|
||||
self.model_name = model_name
|
||||
print(f"[WhisperManager] 模型 '{model_name}' 加载完成。")
|
||||
return self.model
|
||||
|
||||
def extract_audio_from_video(
|
||||
self,
|
||||
video_path: str,
|
||||
output_audio_path: str,
|
||||
logger: Optional[object] = None
|
||||
) -> Optional[float]:
|
||||
if not os.path.exists(video_path):
|
||||
if logger:
|
||||
logger.log(f"视频文件不存在: {video_path}", level="error")
|
||||
return None
|
||||
|
||||
cmd = [
|
||||
get_ffmpeg(), "-loglevel", "error",
|
||||
"-i", video_path,
|
||||
"-vn", "-acodec", "pcm_s16le",
|
||||
"-ar", "16000", "-ac", "1",
|
||||
"-y", output_audio_path
|
||||
]
|
||||
try:
|
||||
subprocess.run(cmd, check=True, capture_output=True, text=True)
|
||||
except subprocess.CalledProcessError as e:
|
||||
if logger:
|
||||
logger.log(f"ffmpeg 提取音频失败: {e.stderr}", level="error")
|
||||
return None
|
||||
|
||||
try:
|
||||
audio = AudioSegment.from_file(output_audio_path)
|
||||
duration = len(audio) / 1000.0
|
||||
if logger:
|
||||
logger.log(f"音频提取成功，时长: {duration:.2f} 秒")
|
||||
return duration
|
||||
except Exception as e:
|
||||
if logger:
|
||||
logger.log(f"获取音频时长失败: {e}", level="error")
|
||||
return None
|
||||
|
||||
def extract_audio_text_with_whisper(
|
||||
self,
|
||||
audio_path: str,
|
||||
start_time: float,
|
||||
end_time: float,
|
||||
logger: Optional[object] = None,
|
||||
use_cache: bool = True
|
||||
) -> str:
|
||||
if self.model is None:
|
||||
raise RuntimeError("Whisper 模型未加载，请先调用 load_model()")
|
||||
|
||||
if not os.path.exists(audio_path):
|
||||
if logger:
|
||||
logger.log(f"音频文件不存在: {audio_path}", level="error")
|
||||
return ""
|
||||
|
||||
cache_key = None
|
||||
if use_cache:
|
||||
mtime = os.path.getmtime(audio_path)
|
||||
cache_key = hashlib.md5(
|
||||
f"{audio_path}_{mtime}_{start_time}_{end_time}".encode()
|
||||
).hexdigest()
|
||||
with self._cache_lock:
|
||||
if cache_key in self._cache:
|
||||
self._cache.move_to_end(cache_key)
|
||||
if logger:
|
||||
logger.log("使用缓存识别结果")
|
||||
return self._cache[cache_key]
|
||||
|
||||
try:
|
||||
full_audio = AudioSegment.from_file(audio_path)
|
||||
start_ms = int(start_time * 1000)
|
||||
end_ms = int(end_time * 1000) + 500
|
||||
segment = full_audio[start_ms:end_ms]
|
||||
|
||||
if len(segment) == 0:
|
||||
if logger:
|
||||
logger.log("音频片段为空，跳过识别")
|
||||
return ""
|
||||
|
||||
segment = segment.set_frame_rate(16000).set_channels(1).set_sample_width(2)
|
||||
|
||||
with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp:
|
||||
temp_path = tmp.name
|
||||
segment.export(temp_path, format="wav")
|
||||
|
||||
try:
|
||||
# faster-whisper CPU 模式线程安全，无需加锁，多线程并发推理
|
||||
segments, _ = self.model.transcribe(
|
||||
temp_path, language="zh", beam_size=5
|
||||
)
|
||||
text = "".join(seg.text for seg in segments).strip()
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temp_path)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
if use_cache and cache_key:
|
||||
with self._cache_lock:
|
||||
self._cache[cache_key] = text
|
||||
self._cache.move_to_end(cache_key)
|
||||
while len(self._cache) > _MAX_CACHE_SIZE:
|
||||
self._cache.popitem(last=False)
|
||||
|
||||
if logger:
|
||||
logger.log(f"音频片段识别结果: {text}")
|
||||
|
||||
return text
|
||||
|
||||
except Exception as e:
|
||||
if logger:
|
||||
logger.log(f"音频识别失败: {e}", level="error")
|
||||
return ""
|
||||
|
||||
|
||||
whisper_manager = WhisperManager()
|
||||
36
tools/_dl.py
Normal file
36
tools/_dl.py
Normal file
@@ -0,0 +1,36 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
import json, os, sys
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
from config.config_utils import config
|
||||
|
||||
cos = config.get_cos_config()
|
||||
client = CosS3Client(CosConfig(Region=cos["region"], SecretId=cos["secret_id"],
|
||||
SecretKey=cos["secret_key"]))
|
||||
bucket = cos["bucket"]
|
||||
|
||||
rows = []
|
||||
for line in open("数据.jsonl", encoding="utf-8"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if r.get("status") == "success" and r.get("srt_cos_path"):
|
||||
rows.append(r)
|
||||
recent = rows[-15:]
|
||||
|
||||
os.makedirs("srt_check", exist_ok=True)
|
||||
ok = 0
|
||||
for r in recent:
|
||||
p = r["srt_cos_path"]
|
||||
key = p.split("://", 1)[1].split("/", 1)[1] # 去掉 cos://bucket/
|
||||
name = os.path.basename(key)
|
||||
try:
|
||||
client.download_file(Bucket=bucket, Key=key, DestFilePath=os.path.join("srt_check", name))
|
||||
ok += 1
|
||||
except Exception as e:
|
||||
print("下载失败", name, e)
|
||||
print(f"下载成功 {ok}/{len(recent)} → srt_check/")
|
||||
100
tools/analyze_ocr_result.py
Normal file
100
tools/analyze_ocr_result.py
Normal file
@@ -0,0 +1,100 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
分析一个 ocr_result.jsonl，判断"冗余文字过多"是真冗余还是误杀。
|
||||
复现过滤逻辑：看文字的垂直位置分布(distinct_bands)、每帧文字框数、有没有稳定字幕行。
|
||||
|
||||
用法: python temp/analyze_ocr_result.py "ocr_result (1).jsonl"
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
PATH = sys.argv[1] if len(sys.argv) > 1 else "ocr_result.jsonl"
|
||||
|
||||
|
||||
def parse(path):
|
||||
"""解析 'pic_path: X' + json 交替格式，返回 [(frame, detections), ...]。"""
|
||||
pairs = []
|
||||
cur_frame = None
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.rstrip("\n")
|
||||
if line.startswith("pic_path:"):
|
||||
cur_frame = line.split("pic_path:", 1)[1].strip()
|
||||
elif line.strip():
|
||||
try:
|
||||
data = json.loads(line)
|
||||
pairs.append((cur_frame, data.get("TextDetections", [])))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
return pairs
|
||||
|
||||
|
||||
def center_y(det):
|
||||
poly = det.get("Polygon", [])
|
||||
ys = [p.get("Y", 0) for p in poly]
|
||||
return sum(ys) / len(ys) if ys else 0
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
pairs = parse(PATH)
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {PATH}")
|
||||
return
|
||||
if not pairs:
|
||||
print("没解析到数据(格式不对?)")
|
||||
return
|
||||
|
||||
n_frames = len(pairs)
|
||||
boxes_per_frame = [len(d) for _, d in pairs]
|
||||
total_boxes = sum(boxes_per_frame)
|
||||
frames_with_text = sum(1 for b in boxes_per_frame if b)
|
||||
|
||||
# 所有文字框的中心 Y
|
||||
all_cy = []
|
||||
for _, dets in pairs:
|
||||
for det in dets:
|
||||
all_cy.append(center_y(det))
|
||||
|
||||
print(f"文件: {PATH}")
|
||||
print("=" * 60)
|
||||
print(f"帧/图数: {n_frames}")
|
||||
print(f"有文字的: {frames_with_text}")
|
||||
print(f"文字框总数: {total_boxes}")
|
||||
print(f"平均每帧框数: {total_boxes / n_frames:.1f}")
|
||||
print(f"最多一帧框数: {max(boxes_per_frame) if boxes_per_frame else 0}")
|
||||
|
||||
# distinct vertical bands(复现 merge.py 的冗余判定：中心Y间隔>80算一个新垂直簇)
|
||||
cys = sorted(all_cy)
|
||||
bands = 1 if cys else 0
|
||||
for i in range(1, len(cys)):
|
||||
if cys[i] - cys[i - 1] > 80:
|
||||
bands += 1
|
||||
print(f"不同垂直位置簇: {bands} (>=8 会被判冗余)")
|
||||
|
||||
# Y 分布直方图(每 100px 一档)，看文字集中在哪还是散布全屏
|
||||
hist = defaultdict(int)
|
||||
for y in all_cy:
|
||||
hist[int(y // 100) * 100] += 1
|
||||
print("\n文字垂直分布(Y每100px一档，看文字集中度):")
|
||||
if hist:
|
||||
mx = max(hist.values())
|
||||
for y in sorted(hist):
|
||||
bar = "#" * int(hist[y] / mx * 40)
|
||||
print(f" y={y:>5}-{y+99:<5} {hist[y]:>6} {bar}")
|
||||
|
||||
# 判定
|
||||
print("\n" + "=" * 60)
|
||||
if bands >= 8:
|
||||
print(f"❌ 真冗余：文字散布在 {bands} 个不同垂直位置(满屏文字/弹幕/文档类)。")
|
||||
elif total_boxes / n_frames > 10:
|
||||
print(f"⚠️ 偏冗余：平均每帧 {total_boxes/n_frames:.0f} 个文字框，画面文字很多。")
|
||||
else:
|
||||
topy = max(hist, key=hist.get) if hist else 0
|
||||
print(f"✅ 可能误杀：文字主要集中在 y={topy}~{topy+99}，像是有固定字幕行的。")
|
||||
print(" 若确实有正常字幕，可放宽 merge.py 的 distinct_bands 阈值。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
463
tools/bench_ocr_endpoints.py
Normal file
463
tools/bench_ocr_endpoints.py
Normal file
@@ -0,0 +1,463 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
压测每个 OCR 端点(IP)的实际 QPS，并把结果写成权重文件 config/ocr_weights.json，
|
||||
供运行时按 QPS 加权分发请求(见 tool/Ocr/tencentcloud_ocr.py 的 OcrDispatcher)。
|
||||
|
||||
做法：对每个端点用固定并发、固定请求数发同一张样例图，
|
||||
QPS = 成功请求数 / 墙钟耗时；同时报告平均时延和错误率。
|
||||
|
||||
本脚本完全自包含：单文件，不读 config、不 import 项目任何模块，拷过去就能跑。
|
||||
|
||||
用法:
|
||||
python bench_ocr_endpoints.py [样例图片] [--requests N] [--concurrency C] [--no-write]
|
||||
python bench_ocr_endpoints.py [样例图片] --url http://IP:PORT [--url http://IP2:PORT ...]
|
||||
|
||||
样例图片不给时，自动在 ./composite、./split 下找一张 .jpg。
|
||||
端点：默认内置 Polaris 地址；要测别的 IP 用 --url(可多次)。
|
||||
--requests 每个端点发的总请求数(默认 60)
|
||||
--concurrency 每个端点的并发线程数(默认 10)
|
||||
--no-write 只打印，不写 ocr_weights.json
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import glob
|
||||
import itertools
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
import requests
|
||||
|
||||
OCR_PATH = "/youtu/ocrapi/hpgeneralocr_v3"
|
||||
OCR_PORT = 18081
|
||||
|
||||
# 内置的多台 OCR 服务 IP（自动负载均衡测试用）
|
||||
DEFAULT_IPS = [
|
||||
"30.154.78.83"
|
||||
]
|
||||
DEFAULT_ENDPOINTS = [f"http://{ip}:{OCR_PORT}" for ip in DEFAULT_IPS]
|
||||
WEIGHTS_FILE = "./ocr_weights.json"
|
||||
|
||||
|
||||
def get_endpoints(cli_urls):
|
||||
"""命令行 --url 优先，否则用内置默认端点。"""
|
||||
if cli_urls:
|
||||
return [u.rstrip("/") for u in cli_urls]
|
||||
return list(DEFAULT_ENDPOINTS)
|
||||
|
||||
|
||||
def find_sample_image(given):
|
||||
if given:
|
||||
if os.path.exists(given):
|
||||
return given
|
||||
print(f"指定的图片不存在: {given}")
|
||||
return None
|
||||
for root in ("./composite", "./split"):
|
||||
hits = glob.glob(os.path.join(root, "**", "*.jpg"), recursive=True)
|
||||
if hits:
|
||||
return hits[0]
|
||||
return None
|
||||
|
||||
|
||||
def img2base64(img_path):
|
||||
with open(img_path, "rb") as f:
|
||||
return base64.b64encode(f.read()).decode()
|
||||
|
||||
|
||||
def ocr_request_once(base_url, image_b64, timeout=60):
|
||||
"""向指定端点发一次 OCR 请求，返回原始响应文本；失败抛异常。"""
|
||||
url = base_url.rstrip("/") + OCR_PATH
|
||||
app_id = f"{int(time.time() * 1000)}{random.randint(100000, 999999)}"
|
||||
payload = {
|
||||
"app_id": app_id,
|
||||
"image": image_b64,
|
||||
"ocr_template": "OCR",
|
||||
"options": {"pdf_page_index": 0, "detect_text": True},
|
||||
}
|
||||
resp = requests.post(url, headers={"Content-Type": "application/json"},
|
||||
json=payload, timeout=timeout)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
|
||||
|
||||
def _ok(raw):
|
||||
"""判断一次响应是否算成功：有内容且不是 error。"""
|
||||
if not raw:
|
||||
return False
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return False
|
||||
if "TextDetections" in data:
|
||||
return True
|
||||
return data.get("error_code", 0) == 0
|
||||
|
||||
|
||||
def bench_endpoint(base_url, image_b64, requests_n, concurrency):
|
||||
"""对单个端点压测，返回统计 dict（含一个失败原因采样 sample_err）。"""
|
||||
latencies = []
|
||||
ok = 0
|
||||
err = 0
|
||||
sample_err = [None] # 采样第一个失败原因
|
||||
|
||||
def one_call():
|
||||
t0 = time.time()
|
||||
try:
|
||||
raw = ocr_request_once(base_url, image_b64, timeout=60)
|
||||
dt = time.time() - t0
|
||||
if _ok(raw):
|
||||
return (True, dt)
|
||||
# HTTP 200 但内容是错误：采样响应体片段
|
||||
if sample_err[0] is None:
|
||||
sample_err[0] = f"响应非成功: {raw[:200]}"
|
||||
return (False, dt)
|
||||
except Exception as e:
|
||||
if sample_err[0] is None:
|
||||
sample_err[0] = f"{type(e).__name__}: {e}"
|
||||
return (False, time.time() - t0)
|
||||
|
||||
t_start = time.time()
|
||||
with ThreadPoolExecutor(max_workers=concurrency) as ex:
|
||||
futures = [ex.submit(one_call) for _ in range(requests_n)]
|
||||
for fut in as_completed(futures):
|
||||
success, dt = fut.result()
|
||||
latencies.append(dt)
|
||||
if success:
|
||||
ok += 1
|
||||
else:
|
||||
err += 1
|
||||
wall = time.time() - t_start
|
||||
|
||||
return {
|
||||
"qps": ok / wall if wall > 0 else 0.0,
|
||||
"ok": ok,
|
||||
"err": err,
|
||||
"wall": wall,
|
||||
"avg_latency": sum(latencies) / len(latencies) if latencies else 0.0,
|
||||
"err_rate": err / requests_n if requests_n else 0.0,
|
||||
"sample_err": sample_err[0],
|
||||
}
|
||||
|
||||
|
||||
def soak_endpoint(base_url, image_pool, concurrency, report_interval=5):
|
||||
"""对单个端点持续满速压测：固定 concurrency 个线程不停发请求，
|
||||
每次从 image_pool 随机抽一张图(避免被服务端缓存)，每 report_interval 秒打印滚动 QPS，
|
||||
直到 Ctrl+C。image_pool 可以是单张或几千张。"""
|
||||
stop = threading.Event()
|
||||
lock = threading.Lock()
|
||||
st = {"ok": 0, "err": 0, "lat": 0.0, "n": 0}
|
||||
sample_err = [None]
|
||||
|
||||
def worker():
|
||||
while not stop.is_set():
|
||||
img = random.choice(image_pool) # 每次随机抽一张
|
||||
t0 = time.time()
|
||||
try:
|
||||
raw = ocr_request_once(base_url, img, timeout=60)
|
||||
success = _ok(raw)
|
||||
if not success and sample_err[0] is None:
|
||||
sample_err[0] = f"响应非成功: {raw[:160]}"
|
||||
except Exception as e:
|
||||
success = False
|
||||
if sample_err[0] is None:
|
||||
sample_err[0] = f"{type(e).__name__}: {e}"
|
||||
dt = time.time() - t0
|
||||
with lock:
|
||||
st["n"] += 1
|
||||
st["lat"] += dt
|
||||
if success:
|
||||
st["ok"] += 1
|
||||
else:
|
||||
st["err"] += 1
|
||||
|
||||
threads = [threading.Thread(target=worker, daemon=True) for _ in range(concurrency)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
|
||||
print(f"持续满速压测 {base_url} 并发={concurrency} 每{report_interval}s 报一次 (Ctrl+C 停止)")
|
||||
print(f"{'用时s':>6} {'累计OK':>9} {'瞬时QPS':>9} {'平均QPS':>9} {'时延ms':>8} {'错误率':>7}")
|
||||
print("-" * 56)
|
||||
t_start = time.time()
|
||||
last_ok, last_t = 0, t_start
|
||||
try:
|
||||
while True:
|
||||
time.sleep(report_interval)
|
||||
now = time.time()
|
||||
with lock:
|
||||
ok, err, n, lat = st["ok"], st["err"], st["n"], st["lat"]
|
||||
inst = (ok - last_ok) / (now - last_t) if now > last_t else 0
|
||||
avg = ok / (now - t_start) if now > t_start else 0
|
||||
avg_lat = (lat / n * 1000) if n else 0
|
||||
er = err / n * 100 if n else 0
|
||||
print(f"{now-t_start:6.0f} {ok:>9} {inst:>9.1f} {avg:>9.1f} {avg_lat:>8.0f} {er:>6.0f}%", flush=True)
|
||||
last_ok, last_t = ok, now
|
||||
except KeyboardInterrupt:
|
||||
stop.set()
|
||||
el = time.time() - t_start
|
||||
with lock:
|
||||
ok, n = st["ok"], st["n"]
|
||||
print("-" * 56)
|
||||
print(f"停止。{el:.0f}s 内成功 {ok} / 共发 {n}，平均 {ok/el if el else 0:.1f} QPS")
|
||||
if sample_err[0]:
|
||||
print(f"失败采样: {sample_err[0]}")
|
||||
|
||||
|
||||
def health_check(endpoints, image_b64):
|
||||
"""给每个端点发一次请求，返回 (存活列表, 死亡列表)。"""
|
||||
alive, dead = [], []
|
||||
|
||||
def _check(ep):
|
||||
try:
|
||||
return ep, _ok(ocr_request_once(ep, image_b64, timeout=15))
|
||||
except Exception:
|
||||
return ep, False
|
||||
|
||||
with ThreadPoolExecutor(max_workers=min(32, len(endpoints))) as ex:
|
||||
for ep, ok in ex.map(_check, endpoints):
|
||||
(alive if ok else dead).append(ep)
|
||||
return alive, dead
|
||||
|
||||
|
||||
def bench_aggregate(endpoints, image_b64, requests_n, concurrency):
|
||||
"""把请求轮询分发到所有端点（模拟运行时自动负载均衡），测合并 QPS。"""
|
||||
rr = itertools.cycle(endpoints)
|
||||
rr_lock = threading.Lock()
|
||||
latencies = []
|
||||
ok = 0
|
||||
err = 0
|
||||
stat_lock = threading.Lock()
|
||||
|
||||
def next_ep():
|
||||
with rr_lock:
|
||||
return next(rr)
|
||||
|
||||
def one_call(_):
|
||||
ep = next_ep()
|
||||
t0 = time.time()
|
||||
try:
|
||||
raw = ocr_request_once(ep, image_b64, timeout=60)
|
||||
success = _ok(raw)
|
||||
except Exception:
|
||||
success = False
|
||||
dt = time.time() - t0
|
||||
with stat_lock:
|
||||
latencies.append(dt)
|
||||
return success
|
||||
|
||||
t_start = time.time()
|
||||
with ThreadPoolExecutor(max_workers=concurrency) as ex:
|
||||
for success in ex.map(one_call, range(requests_n)):
|
||||
if success:
|
||||
ok += 1
|
||||
else:
|
||||
err += 1
|
||||
wall = time.time() - t_start
|
||||
|
||||
return {
|
||||
"qps": ok / wall if wall > 0 else 0.0,
|
||||
"ok": ok,
|
||||
"err": err,
|
||||
"wall": wall,
|
||||
"avg_latency": sum(latencies) / len(latencies) if latencies else 0.0,
|
||||
"err_rate": err / requests_n if requests_n else 0.0,
|
||||
}
|
||||
|
||||
|
||||
def run_aggregate(endpoints, image_b64, requests_n, concurrency):
|
||||
"""先健康检查筛掉死端点，再在存活端点上做聚合负载测试。"""
|
||||
print(f"健康检查 {len(endpoints)} 个端点 ...", flush=True)
|
||||
alive, dead = health_check(endpoints, image_b64)
|
||||
print(f" 存活: {len(alive)} 死亡: {len(dead)}")
|
||||
if dead:
|
||||
for ep in dead:
|
||||
print(f" ✗ {ep}")
|
||||
if not alive:
|
||||
print("没有存活端点，无法测试。")
|
||||
return
|
||||
print("=" * 64)
|
||||
print(f"聚合负载测试: {len(alive)} 个存活端点轮询分发，"
|
||||
f"总请求 {requests_n}，总并发 {concurrency}")
|
||||
s = bench_aggregate(alive, image_b64, requests_n, concurrency)
|
||||
print("-" * 64)
|
||||
print(f" 合并 QPS = {s['qps']:.2f} 成功={s['ok']} 失败={s['err']}"
|
||||
f" 平均时延={s['avg_latency']*1000:.0f}ms 错误率={s['err_rate']*100:.0f}%")
|
||||
if alive:
|
||||
print(f" 单端点平均 ≈ {s['qps']/len(alive):.2f} QPS"
|
||||
f" （{len(alive)} 台合计 {s['qps']:.0f} QPS）")
|
||||
print("=" * 64)
|
||||
|
||||
|
||||
def sweep_endpoint(base_url, image_b64, start, max_c, per_worker):
|
||||
"""自动加压找极限 QPS：并发从 start 逐级翻倍到 max_c，
|
||||
QPS 不再随并发上升(或错误率升高)即判定到极限，停止。"""
|
||||
print(f"自动加压 {base_url} (并发 {start}→{max_c} 逐级翻倍)")
|
||||
print(f"{'并发':>6} {'QPS':>9} {'时延ms':>9} {'错误率':>8}")
|
||||
print("-" * 36)
|
||||
best_qps, best_c = 0.0, 0
|
||||
prev_qps = 0.0
|
||||
c = start
|
||||
while c <= max_c:
|
||||
reqs = max(c * per_worker, 30) # 每个并发至少发 per_worker 次，样本够稳
|
||||
s = bench_endpoint(base_url, image_b64, reqs, c)
|
||||
print(f"{c:>6} {s['qps']:>9.2f} {s['avg_latency']*1000:>9.0f} {s['err_rate']*100:>7.0f}%")
|
||||
if s["qps"] > best_qps:
|
||||
best_qps, best_c = s["qps"], c
|
||||
if s["err_rate"] > 0.1:
|
||||
print(" → 错误率升高，已超过服务承载，停止。")
|
||||
break
|
||||
if prev_qps > 0 and s["qps"] < prev_qps * 1.08:
|
||||
print(" → QPS 不再随并发上升，已到极限，停止。")
|
||||
break
|
||||
prev_qps = s["qps"]
|
||||
c *= 2
|
||||
print("-" * 36)
|
||||
print(f"极限 QPS ≈ {best_qps:.2f} (最佳并发 {best_c})")
|
||||
return best_qps, best_c
|
||||
|
||||
|
||||
def gen_random_images(n, size=(1280, 200)):
|
||||
"""生成 n 张带随机文字的图片(让 OCR 真干活)，返回 base64 列表。需要 Pillow。"""
|
||||
try:
|
||||
import io
|
||||
import string
|
||||
from PIL import Image, ImageDraw
|
||||
except ImportError:
|
||||
print("生成随机图需要 Pillow(pip install pillow)；改用 --pool 指向真实图目录")
|
||||
raise
|
||||
chars = string.ascii_letters + string.digits + " ,.!?-"
|
||||
pool = []
|
||||
print(f"生成 {n} 张随机文字图...", flush=True)
|
||||
for i in range(n):
|
||||
bg = random.randint(200, 255)
|
||||
img = Image.new("RGB", size, (bg, bg, bg))
|
||||
d = ImageDraw.Draw(img)
|
||||
for row in range(random.randint(1, 3)):
|
||||
text = "".join(random.choice(chars) for _ in range(random.randint(10, 40)))
|
||||
d.text((20, 20 + row * 50), text, fill=(0, 0, 0))
|
||||
buf = io.BytesIO()
|
||||
img.save(buf, format="JPEG", quality=85)
|
||||
pool.append(base64.b64encode(buf.getvalue()).decode())
|
||||
if (i + 1) % 1000 == 0:
|
||||
print(f" 已生成 {i+1}/{n}", flush=True)
|
||||
return pool
|
||||
|
||||
|
||||
def build_image_pool(args):
|
||||
"""构建测试图池(base64 列表)：
|
||||
--pool DIR 用真实图目录(最贴近实际负载) > --gen N 随机生成 > 单张样例图。"""
|
||||
if getattr(args, "pool", None):
|
||||
files = (glob.glob(os.path.join(args.pool, "**", "*.jpg"), recursive=True) +
|
||||
glob.glob(os.path.join(args.pool, "**", "*.png"), recursive=True))
|
||||
if not files:
|
||||
print(f"目录 {args.pool} 下没有图片")
|
||||
return []
|
||||
random.shuffle(files)
|
||||
cap = args.gen if args.gen else 5000
|
||||
files = files[:cap]
|
||||
print(f"从 {args.pool} 取 {len(files)} 张真实图作为图池")
|
||||
return [img2base64(f) for f in files]
|
||||
if getattr(args, "gen", 0):
|
||||
return gen_random_images(args.gen)
|
||||
sample = find_sample_image(args.image)
|
||||
if not sample:
|
||||
return []
|
||||
print(f"单张样例图: {sample}")
|
||||
return [img2base64(sample)]
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("image", nargs="?", default=None)
|
||||
ap.add_argument("--url", action="append", default=None, help="OCR 端点(可多次)")
|
||||
ap.add_argument("--requests", type=int, default=60)
|
||||
ap.add_argument("--concurrency", type=int, default=10)
|
||||
ap.add_argument("--aggregate", action="store_true",
|
||||
help="聚合负载测试:请求轮询分发到所有存活端点，测合并总 QPS")
|
||||
ap.add_argument("--sweep", action="store_true",
|
||||
help="自动加压找极限 QPS(并发逐级翻倍直到 QPS 不再上升)")
|
||||
ap.add_argument("--soak", action="store_true",
|
||||
help="单端点持续满速压测(固定并发不停发，实时滚动QPS，Ctrl+C 停)")
|
||||
ap.add_argument("--interval", type=int, default=5, help="soak 模式打印间隔秒")
|
||||
ap.add_argument("--gen", type=int, default=0,
|
||||
help="随机生成 N 张带文字的图作为图池(每次请求随机抽，避免被缓存)")
|
||||
ap.add_argument("--pool", default=None,
|
||||
help="用该目录下的真实图作为图池(最贴近实际负载，如 ./split/某视频)")
|
||||
ap.add_argument("--start-concurrency", type=int, default=10, help="sweep 起始并发")
|
||||
ap.add_argument("--max-concurrency", type=int, default=320, help="sweep 最大并发")
|
||||
ap.add_argument("--per-worker", type=int, default=6, help="sweep 每档每并发发多少次请求")
|
||||
ap.add_argument("--no-write", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
endpoints = get_endpoints(args.url)
|
||||
if not endpoints:
|
||||
print("没有可用的 OCR 端点")
|
||||
return
|
||||
|
||||
# 构建图池：--pool 真实图目录 / --gen 随机生成 / 默认单张样例
|
||||
image_pool = build_image_pool(args)
|
||||
if not image_pool:
|
||||
print("没有可用图片：给一张 jpg 路径、或用 --gen N / --pool 目录。")
|
||||
return
|
||||
print(f"图池大小: {len(image_pool)} 张 | 端点数: {len(endpoints)} 并发: {args.concurrency}")
|
||||
print("=" * 64)
|
||||
|
||||
image_b64 = image_pool[0] # 非 soak 模式用单张(测纯吞吐)
|
||||
|
||||
# ---- soak 模式：单端点持续满速压测，每次随机抽图，直到 Ctrl+C ----
|
||||
if args.soak:
|
||||
soak_endpoint(endpoints[0], image_pool, args.concurrency, args.interval)
|
||||
return
|
||||
|
||||
# ---- aggregate 模式：所有端点轮询分发，测合并总 QPS ----
|
||||
if args.aggregate:
|
||||
run_aggregate(endpoints, image_b64, args.requests, args.concurrency)
|
||||
return
|
||||
|
||||
# ---- sweep 模式：自动加压找极限，不写权重文件 ----
|
||||
if args.sweep:
|
||||
for i, ep in enumerate(endpoints, 1):
|
||||
print(f"\n[{i}/{len(endpoints)}]", end=" ")
|
||||
sweep_endpoint(ep, image_b64, args.start_concurrency,
|
||||
args.max_concurrency, args.per_worker)
|
||||
return
|
||||
|
||||
results = {}
|
||||
for i, ep in enumerate(endpoints, 1):
|
||||
print(f"[{i}/{len(endpoints)}] 压测 {ep} ...", flush=True)
|
||||
stat = bench_endpoint(ep, image_b64, args.requests, args.concurrency)
|
||||
results[ep] = stat
|
||||
print(f" QPS={stat['qps']:.2f} 成功={stat['ok']} 失败={stat['err']}"
|
||||
f" 平均时延={stat['avg_latency']*1000:.0f}ms"
|
||||
f" 错误率={stat['err_rate']*100:.0f}%")
|
||||
if stat["err"] and stat.get("sample_err"):
|
||||
print(f" 失败原因(采样): {stat['sample_err']}")
|
||||
|
||||
print("=" * 64)
|
||||
print("汇总(按 QPS 降序):")
|
||||
for ep, s in sorted(results.items(), key=lambda kv: kv[1]["qps"], reverse=True):
|
||||
print(f" {ep:48s} QPS={s['qps']:6.2f} 时延={s['avg_latency']*1000:5.0f}ms"
|
||||
f" 错误率={s['err_rate']*100:.0f}%")
|
||||
|
||||
total_qps = sum(s["qps"] for s in results.values())
|
||||
print(f"\n合计可用 QPS ≈ {total_qps:.2f}")
|
||||
|
||||
# 写权重文件：QPS 即权重；错误率过高(>50%)的端点权重清零(不参与分发)
|
||||
weights = {ep: (round(s["qps"], 3) if s["err_rate"] <= 0.5 else 0.0)
|
||||
for ep, s in results.items()}
|
||||
|
||||
if args.no_write:
|
||||
print("\n--no-write：不写权重文件。")
|
||||
return
|
||||
|
||||
os.makedirs(os.path.dirname(WEIGHTS_FILE), exist_ok=True)
|
||||
with open(WEIGHTS_FILE, "w", encoding="utf-8") as f:
|
||||
json.dump(weights, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n已写入权重文件: {WEIGHTS_FILE}")
|
||||
print("运行时会按此权重把 OCR 请求加权分发到各端点（重启进程或新批次生效）。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
117
tools/check_srt.py
Normal file
117
tools/check_srt.py
Normal file
@@ -0,0 +1,117 @@
|
||||
"""
|
||||
使用 cos-python-sdk-v5 全量扫描 0004/data/ 下的文件，
|
||||
找出已有 SRT 文件的目录，将对应的 MP4 COS 路径写入 txt 文件。
|
||||
|
||||
安装依赖：pip install cos-python-sdk-v5
|
||||
"""
|
||||
import os
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
from config.config_utils import config as app_config
|
||||
|
||||
SCAN_PREFIX = "0004/data/"
|
||||
OUTPUT_TXT = "./text/0004_has_srt.txt"
|
||||
DIR_RE = re.compile(r'.+/data/[^/]+/[^/]+/$')
|
||||
|
||||
|
||||
def build_client() -> tuple[CosS3Client, str]:
|
||||
cos_cfg = app_config.get_cos_config()
|
||||
secret_id = cos_cfg.get("secret_id")
|
||||
secret_key = cos_cfg.get("secret_key")
|
||||
region = cos_cfg.get("region", "ap-chongqing")
|
||||
bucket = cos_cfg.get("bucket", "asr-dig-data-1253181912")
|
||||
|
||||
if not secret_id or not secret_key or "填入" in secret_id:
|
||||
raise ValueError("请先在 config/config.json 的 tencentcloud.cos 中填入 secret_id 和 secret_key")
|
||||
|
||||
cfg = CosConfig(Region=region, SecretId=secret_id, SecretKey=secret_key)
|
||||
client = CosS3Client(cfg)
|
||||
return client, bucket
|
||||
|
||||
|
||||
def list_all_objects(client: CosS3Client, bucket: str, prefix: str) -> list[str]:
|
||||
"""分页获取 prefix 下全部 Key"""
|
||||
keys = []
|
||||
marker = ""
|
||||
page = 0
|
||||
|
||||
while True:
|
||||
page += 1
|
||||
kwargs = dict(Bucket=bucket, Prefix=prefix, MaxKeys=1000)
|
||||
if marker:
|
||||
kwargs["Marker"] = marker
|
||||
|
||||
resp = client.list_objects(**kwargs)
|
||||
contents = resp.get("Contents", [])
|
||||
keys.extend(item["Key"] for item in contents)
|
||||
|
||||
print(f"\r 第 {page} 页，已获取 {len(keys)} 个文件...", end="", flush=True)
|
||||
|
||||
if resp.get("IsTruncated") == "true":
|
||||
marker = resp.get("NextMarker", "")
|
||||
else:
|
||||
break
|
||||
|
||||
print()
|
||||
return keys
|
||||
|
||||
|
||||
def main():
|
||||
client, bucket = build_client()
|
||||
bucket_prefix = f"cos://{bucket}"
|
||||
|
||||
print(f"正在全量扫描 {bucket_prefix}/{SCAN_PREFIX} ...")
|
||||
keys = list_all_objects(client, bucket, SCAN_PREFIX)
|
||||
print(f"共找到 {len(keys)} 个文件")
|
||||
|
||||
if not keys:
|
||||
print("未找到任何文件，请检查密钥和路径是否正确")
|
||||
return
|
||||
|
||||
# 按父目录分组
|
||||
dir_files: dict[str, list[str]] = defaultdict(list)
|
||||
for key in keys:
|
||||
idx = key.rfind("/")
|
||||
dir_path = key[:idx + 1] # e.g. "0004/data/BV1xx/BV1xx/"
|
||||
filename = key[idx + 1:]
|
||||
if filename:
|
||||
dir_files[dir_path].append(filename)
|
||||
|
||||
# 找有 SRT 的目录，收集 MP4 路径
|
||||
mp4_paths = []
|
||||
has_srt_cnt = 0
|
||||
no_srt_cnt = 0
|
||||
|
||||
for dir_path, filenames in sorted(dir_files.items()):
|
||||
# 只处理 data/{id}/{id}/ 结构
|
||||
if not DIR_RE.match(dir_path):
|
||||
continue
|
||||
|
||||
has_srt = any(f.lower().endswith(".srt") for f in filenames)
|
||||
mp4_files = [f for f in filenames if f.lower().endswith(".mp4")]
|
||||
|
||||
if has_srt:
|
||||
has_srt_cnt += 1
|
||||
for mp4 in mp4_files:
|
||||
mp4_paths.append(f"{bucket_prefix}/{dir_path}{mp4}")
|
||||
else:
|
||||
no_srt_cnt += 1
|
||||
|
||||
print(f"有 SRT 的目录: {has_srt_cnt} 个 → {len(mp4_paths)} 个 MP4")
|
||||
print(f"无 SRT 的目录: {no_srt_cnt} 个")
|
||||
|
||||
if not mp4_paths:
|
||||
print("没有找到含 SRT 的目录")
|
||||
return
|
||||
|
||||
os.makedirs("./text", exist_ok=True)
|
||||
with open(OUTPUT_TXT, "w", encoding="utf-8") as f:
|
||||
for path in mp4_paths:
|
||||
f.write(path + "\n")
|
||||
|
||||
print(f"已写入: {OUTPUT_TXT} ({len(mp4_paths)} 条)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
157
tools/compare_jsonl.py
Normal file
157
tools/compare_jsonl.py
Normal file
@@ -0,0 +1,157 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
并排对比两个(或多个)机器的 数据.jsonl，定位谁慢、慢在哪个阶段。
|
||||
|
||||
用法:
|
||||
python temp/compare_jsonl.py 2号机器数据.jsonl 3号机器数据.jsonl
|
||||
python temp/compare_jsonl.py a.jsonl b.jsonl c.jsonl ...
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
def parse_dt(s):
|
||||
try:
|
||||
return datetime.fromisoformat(s)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def analyze(path):
|
||||
"""读一个 jsonl，返回汇总指标。"""
|
||||
s = {
|
||||
"n": 0, "ok": 0, "err": 0, "no_sub": 0,
|
||||
"dur": 0.0, "elapsed": 0.0,
|
||||
"stage": {"split_frame": 0.0, "merge": 0.0, "getsrt": 0.0},
|
||||
"ocr": 0, "n_ocr": 0,
|
||||
"starts": [], "ends": [],
|
||||
}
|
||||
try:
|
||||
f = open(path, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
return None
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
s["n"] += 1
|
||||
st = r.get("status")
|
||||
if st == "success":
|
||||
s["ok"] += 1
|
||||
elif st == "no_subtitle":
|
||||
s["no_sub"] += 1
|
||||
else:
|
||||
s["err"] += 1
|
||||
try:
|
||||
s["dur"] += float(r.get("duration") or 0)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
el = r.get("elapsed_seconds")
|
||||
if el:
|
||||
try:
|
||||
s["elapsed"] += float(el)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
sd = r.get("stage_durations") or {}
|
||||
for k in s["stage"]:
|
||||
try:
|
||||
s["stage"][k] += float(sd.get(k) or 0)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
ic = r.get("image_counts") or {}
|
||||
toc = ic.get("total_ocr_count")
|
||||
if toc:
|
||||
try:
|
||||
s["ocr"] += int(toc)
|
||||
s["n_ocr"] += 1
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
ds = parse_dt(r.get("process_start"))
|
||||
de = parse_dt(r.get("process_end"))
|
||||
if ds:
|
||||
s["starts"].append(ds)
|
||||
if de:
|
||||
s["ends"].append(de)
|
||||
return s
|
||||
|
||||
|
||||
def derive(s):
|
||||
"""从汇总算出对比指标。"""
|
||||
d = {}
|
||||
n = max(1, s["n"])
|
||||
d["记录数"] = s["n"]
|
||||
d["成功率%"] = round(s["ok"] / n * 100, 1)
|
||||
d["视频总时长h"] = round(s["dur"] / 3600, 1)
|
||||
# 墙钟跨度 + 实时倍率
|
||||
span = 0.0
|
||||
if s["starts"] and s["ends"]:
|
||||
span = (max(s["ends"]) - min(s["starts"])).total_seconds()
|
||||
d["墙钟跨度h"] = round(span / 3600, 2)
|
||||
d["实时倍率x"] = round(s["dur"] / span, 2) if span > 0 else 0
|
||||
d["视频/天"] = round(s["n"] / span * 86400) if span > 0 else 0
|
||||
d["有效并发"] = round(s["elapsed"] / span, 1) if span > 0 else 0
|
||||
# 平均每视频各阶段秒数
|
||||
d["抽帧s/视频"] = round(s["stage"]["split_frame"] / n, 1)
|
||||
d["拼接s/视频"] = round(s["stage"]["merge"] / n, 1)
|
||||
d["getsrt s/视频"] = round(s["stage"]["getsrt"] / n, 1)
|
||||
total_stage = sum(s["stage"].values()) or 1
|
||||
d["抽帧占比%"] = round(s["stage"]["split_frame"] / total_stage * 100)
|
||||
d["拼接占比%"] = round(s["stage"]["merge"] / total_stage * 100)
|
||||
d["getsrt占比%"] = round(s["stage"]["getsrt"] / total_stage * 100)
|
||||
d["平均OCR次/视频"] = round(s["ocr"] / s["n_ocr"]) if s["n_ocr"] else 0
|
||||
d["平均耗时s/视频"] = round(s["elapsed"] / n, 1)
|
||||
return d
|
||||
|
||||
|
||||
def main():
|
||||
paths = sys.argv[1:]
|
||||
if len(paths) < 2:
|
||||
print("用法: python temp/compare_jsonl.py a.jsonl b.jsonl [c.jsonl ...]")
|
||||
return
|
||||
|
||||
cols, names = [], []
|
||||
for p in paths:
|
||||
s = analyze(p)
|
||||
if s is None:
|
||||
print(f"找不到文件: {p}")
|
||||
return
|
||||
cols.append(derive(s))
|
||||
names.append(p.replace("数据.jsonl", "").replace(".jsonl", "") or p)
|
||||
|
||||
keys = list(cols[0].keys())
|
||||
w = max(14, max(len(n) for n in names) + 2)
|
||||
print("=" * (18 + w * len(cols)))
|
||||
print(f"{'指标':<16}" + "".join(f"{n:>{w}}" for n in names))
|
||||
print("-" * (18 + w * len(cols)))
|
||||
for k in keys:
|
||||
print(f"{k:<16}" + "".join(f"{str(c[k]):>{w}}" for c in cols))
|
||||
print("=" * (18 + w * len(cols)))
|
||||
|
||||
# 自动点评(两台时)
|
||||
if len(cols) == 2:
|
||||
a, b = cols
|
||||
slow_i = 0 if a["实时倍率x"] < b["实时倍率x"] else 1
|
||||
fast_i = 1 - slow_i
|
||||
print(f"\n【分析】{names[slow_i]} 更慢(实时倍率 "
|
||||
f"{cols[slow_i]['实时倍率x']}x vs {cols[fast_i]['实时倍率x']}x)")
|
||||
# 找慢在哪个阶段
|
||||
for stage, key in [("抽帧", "抽帧s/视频"), ("拼接", "拼接s/视频"), ("getsrt(OCR)", "getsrt s/视频")]:
|
||||
sv, fv = cols[slow_i][key], cols[fast_i][key]
|
||||
if fv and sv > fv * 1.3:
|
||||
print(f" ▸ {stage} 慢: {sv}s vs {fv}s (慢 {sv/fv:.1f}x)")
|
||||
if cols[slow_i]["有效并发"] < cols[fast_i]["有效并发"] * 0.8:
|
||||
print(f" ▸ 并发低: {cols[slow_i]['有效并发']} vs {cols[fast_i]['有效并发']}"
|
||||
f"(CPU/内存/OCR 排队?)")
|
||||
if cols[slow_i]["平均OCR次/视频"] > cols[fast_i]["平均OCR次/视频"] * 1.3:
|
||||
print(f" ▸ OCR 次数多: {cols[slow_i]['平均OCR次/视频']} vs "
|
||||
f"{cols[fast_i]['平均OCR次/视频']}(视频更长/composite更多)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
99
tools/daily_stats.py
Normal file
99
tools/daily_stats.py
Normal file
@@ -0,0 +1,99 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
按天统计 数据.jsonl：每天处理了多少条视频、处理的视频总时长是多少。
|
||||
日期取 process_start（没有则回退 timestamp）的年月日。
|
||||
|
||||
用法（在项目根目录运行）:
|
||||
python temp/daily_stats.py [数据.jsonl路径]
|
||||
默认: python temp/daily_stats.py 数据.jsonl
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
|
||||
JSONL_PATH = sys.argv[1] if len(sys.argv) > 1 else "数据.jsonl"
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
f = open(JSONL_PATH, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {JSONL_PATH}")
|
||||
return
|
||||
|
||||
# 每天: count/success/no_subtitle/error 计数；dur=全部视频时长；dur_ok=成功视频时长
|
||||
days = defaultdict(lambda: {"count": 0, "success": 0, "no_subtitle": 0,
|
||||
"error": 0, "dur": 0.0, "dur_ok": 0.0})
|
||||
no_date = 0
|
||||
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
ts = rec.get("process_start") or rec.get("timestamp") or ""
|
||||
day = ts[:10] if len(ts) >= 10 else None # YYYY-MM-DD
|
||||
if not day:
|
||||
no_date += 1
|
||||
continue
|
||||
|
||||
d = days[day]
|
||||
d["count"] += 1
|
||||
|
||||
status = rec.get("status")
|
||||
if status == "success":
|
||||
d["success"] += 1
|
||||
elif status == "no_subtitle":
|
||||
d["no_subtitle"] += 1
|
||||
else:
|
||||
d["error"] += 1
|
||||
|
||||
dur = rec.get("duration") or 0
|
||||
try:
|
||||
dur = float(dur)
|
||||
d["dur"] += dur
|
||||
if status == "success":
|
||||
d["dur_ok"] += dur # 只累计成功出字幕的视频时长
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
if not days:
|
||||
print("没有可统计的数据")
|
||||
return
|
||||
|
||||
print(f"文件: {JSONL_PATH}")
|
||||
print("=" * 84)
|
||||
print(f"{'日期':<12}{'视频数':>7}{'成功':>7}{'无字幕':>7}{'错误':>7}"
|
||||
f"{'总时长(h)':>12}{'成功时长(h)':>14}")
|
||||
print("-" * 84)
|
||||
|
||||
tot = {"count": 0, "success": 0, "no_subtitle": 0, "error": 0, "dur": 0.0, "dur_ok": 0.0}
|
||||
for day in sorted(days):
|
||||
d = days[day]
|
||||
print(f"{day:<12}{d['count']:>7}{d['success']:>7}{d['no_subtitle']:>7}"
|
||||
f"{d['error']:>7}{d['dur']/3600:>12.1f}{d['dur_ok']/3600:>14.1f}")
|
||||
for k in tot:
|
||||
tot[k] += d[k]
|
||||
|
||||
print("-" * 84)
|
||||
print(f"{'合计':<12}{tot['count']:>7}{tot['success']:>7}{tot['no_subtitle']:>7}"
|
||||
f"{tot['error']:>7}{tot['dur']/3600:>12.1f}{tot['dur_ok']/3600:>14.1f}")
|
||||
if no_date:
|
||||
print(f"\n（{no_date} 条无日期字段，已跳过）")
|
||||
|
||||
n_days = len(days)
|
||||
if n_days:
|
||||
print(f"\n共 {n_days} 天 | 日均处理 {tot['count']/n_days:.0f} 条 / "
|
||||
f"{tot['dur']/3600/n_days:.1f} h | 日均成功 {tot['success']/n_days:.0f} 条 / "
|
||||
f"{tot['dur_ok']/3600/n_days:.1f} h")
|
||||
print(f"成功率 {tot['success']/tot['count']*100:.1f}% | "
|
||||
f"成功时长占比 {tot['dur_ok']/tot['dur']*100:.1f}%" if tot['dur'] else "")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
96
tools/download_srt.py
Normal file
96
tools/download_srt.py
Normal file
@@ -0,0 +1,96 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
从 数据.jsonl 中读取成功记录，将 MP4 和 SRT 文件下载到本地 ./成功/ 目录。
|
||||
用法：python download_srt.py [数据.jsonl路径] [本地输出目录]
|
||||
默认：python download_srt.py 数据.jsonl ./成功
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from config.config_utils import config
|
||||
|
||||
JSONL_PATH = sys.argv[1] if len(sys.argv) > 1 else "数据.jsonl"
|
||||
OUTPUT_DIR = sys.argv[2] if len(sys.argv) > 2 else "./成功"
|
||||
|
||||
|
||||
def download(coscli_path, cos_config_path, cos_src, local_path):
|
||||
"""下载单个文件，返回是否成功"""
|
||||
if os.path.exists(local_path):
|
||||
return "skip"
|
||||
cmd = [coscli_path, "-c", cos_config_path, "cp", cos_src, local_path]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
return "ok" if result.returncode == 0 else result.stderr.strip()
|
||||
|
||||
|
||||
def main():
|
||||
if not os.path.exists(JSONL_PATH):
|
||||
print(f"找不到文件: {JSONL_PATH}")
|
||||
return
|
||||
|
||||
cfg = config.get_splitframe_config()
|
||||
coscli_path = cfg.get('coscli_path', './coscli-linux-amd64')
|
||||
cos_config_path = cfg.get('cosconfig_path', '.cos.yaml')
|
||||
|
||||
# 读取所有成功记录
|
||||
records = []
|
||||
with open(JSONL_PATH, 'r', encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
if rec.get('status') == 'success' and rec.get('srt_cos_path'):
|
||||
records.append(rec)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
records = records[:50]
|
||||
total = len(records)
|
||||
print(f"共找到 {total} 条成功记录，开始下载...")
|
||||
os.makedirs(OUTPUT_DIR, exist_ok=True)
|
||||
|
||||
success = skip = failed = 0
|
||||
|
||||
for i, rec in enumerate(records, 1):
|
||||
video_cos = rec.get('video_cos_path').replace(".mp4", ".mp3")
|
||||
srt_cos = rec.get('srt_cos_path')
|
||||
|
||||
video_name = os.path.splitext(os.path.basename(video_cos))[0] if video_cos else None
|
||||
|
||||
print(f"[{i}/{total}] {video_name}")
|
||||
|
||||
# 下载 MP4
|
||||
if video_cos:
|
||||
local_mp4 = os.path.join(OUTPUT_DIR, os.path.basename(video_cos))
|
||||
ret = download(coscli_path, cos_config_path, video_cos, local_mp4)
|
||||
if ret == "ok":
|
||||
success += 1
|
||||
print(f" ✓ 视频: {os.path.basename(video_cos)}")
|
||||
elif ret == "skip":
|
||||
skip += 1
|
||||
print(f" - 视频已存在，跳过")
|
||||
else:
|
||||
failed += 1
|
||||
print(f" ✗ 视频下载失败: {ret}")
|
||||
|
||||
# 下载 SRT
|
||||
if srt_cos:
|
||||
local_srt = os.path.join(OUTPUT_DIR, os.path.basename(srt_cos))
|
||||
ret = download(coscli_path, cos_config_path, srt_cos, local_srt)
|
||||
if ret == "ok":
|
||||
success += 1
|
||||
print(f" ✓ 字幕: {os.path.basename(srt_cos)}")
|
||||
elif ret == "skip":
|
||||
skip += 1
|
||||
print(f" - 字幕已存在，跳过")
|
||||
else:
|
||||
failed += 1
|
||||
print(f" ✗ 字幕下载失败: {ret}")
|
||||
|
||||
print(f"\n完成 — 下载: {success} 失败: {failed} 跳过(已存在): {skip}")
|
||||
print(f"文件保存在: {os.path.abspath(OUTPUT_DIR)}")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
69
tools/extract_errors.py
Normal file
69
tools/extract_errors.py
Normal file
@@ -0,0 +1,69 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
从 数据.jsonl 提取【最终状态是错误/无字幕】的视频 COS 路径，写到 txt(可当任务列表重跑测试)。
|
||||
按每个视频的【最后一条记录】判定最终状态(忽略中间重试)。
|
||||
|
||||
用法:
|
||||
python temp/extract_errors.py 2号机器数据.jsonl error_videos.txt
|
||||
python temp/extract_errors.py 2号机器数据.jsonl out.txt error,no_subtitle # 也要无字幕的
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
JSONL = sys.argv[1] if len(sys.argv) > 1 else "数据.jsonl"
|
||||
OUT = sys.argv[2] if len(sys.argv) > 2 else "error_videos.txt"
|
||||
STATUSES = sys.argv[3].split(",") if len(sys.argv) > 3 else ["error"]
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
f = open(JSONL, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {JSONL}")
|
||||
return
|
||||
|
||||
latest = {} # video_cos_path -> (status, error_reason)
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
vid = r.get("video_cos_path")
|
||||
if not vid:
|
||||
continue
|
||||
latest[vid] = (r.get("status"), r.get("error_reason", ""))
|
||||
|
||||
# 最终状态计数
|
||||
status_count = Counter(s for s, _ in latest.values())
|
||||
# 选中的(最终状态在 STATUSES 里)
|
||||
selected = [(v, er) for v, (s, er) in latest.items() if s in STATUSES]
|
||||
|
||||
# 写 txt(每行一个 COS 路径)
|
||||
with open(OUT, "w", encoding="utf-8") as w:
|
||||
for v, _ in selected:
|
||||
w.write(v + "\n")
|
||||
|
||||
print(f"文件: {JSONL}")
|
||||
print(f"去重后视频数: {len(latest)}")
|
||||
print(f"最终状态分布: {dict(status_count)}")
|
||||
print(f"提取状态 {STATUSES}: {len(selected)} 个 → 写入 {OUT}")
|
||||
|
||||
# 错误原因 Top（帮你看主要错在哪）
|
||||
reasons = Counter()
|
||||
for _, er in selected:
|
||||
if er:
|
||||
key = er.split(":")[0][:60] if ":" in er else er[:60]
|
||||
reasons[key] += 1
|
||||
if reasons:
|
||||
print("\n错误原因 Top:")
|
||||
for k, c in reasons.most_common(10):
|
||||
print(f" {c:>5} {k}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
316
tools/fleet_monitor.py
Normal file
316
tools/fleet_monitor.py
Normal file
@@ -0,0 +1,316 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""全队健康度监控: 汇总所有机器最新一小时的处理摘要, 跑异常检测, 产出报告。
|
||||
配合 run_worker.py 里的 job_cos.run_hourly_digest_loop() 后台线程(每台机器每小时自动上传
|
||||
一份小摘要到 COS)使用 —— 本脚本负责把这些小摘要拉回来汇总判断。
|
||||
|
||||
用法(项目根目录运行, 建议 crontab 每小时跑一次，比如 5分/10分 触发一次都行，脚本自己会
|
||||
算准要看哪个整点):
|
||||
python tools/fleet_monitor.py # 单小时报告 + 异常检测(原有行为)
|
||||
python tools/fleet_monitor.py --history 24 # 全队历史趋势(最近24个小时, 不跑异常检测)
|
||||
python tools/fleet_monitor.py --history # 不带数字 = 默认 24 小时
|
||||
|
||||
设计文档: docs/superpowers/specs/2026-07-08-fleet-monitoring-design.md
|
||||
不接外部通知渠道(企业微信/钉钉/飞书webhook 是后续可插拔扩展点, 本次只落地本地+COS报告文件)。
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
import unicodedata
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from tool import job_cos # noqa: E402
|
||||
|
||||
# ---- 异常检测阈值(先写死成常量, 要调直接改这里) ----
|
||||
OFFLINE_HOURS = 2 # 机器掉线: 最新摘要距目标小时 >= 这么多小时
|
||||
SUCCESS_RATE_MIN = 0.5 # 单小时成功率骤降阈值
|
||||
SUCCESS_RATE_MIN_SAMPLE = 10 # 样本太少(成功+错误 < 这个数)不判断成功率, 避免噪音
|
||||
MACHINE_DROP_RATIO = 0.3 # 单机吞吐骤降: 本小时产出 < 自己24h均值的这个比例
|
||||
FLEET_DROP_RATIO = 0.5 # 全队吞吐骤降: 全队本小时产出 < 全队24h均值的这个比例
|
||||
BASELINE_HOURS = 24 # 基线取多少小时的历史均值(不含目标小时本身)
|
||||
KNOWN_MACHINES_LOOKBACK = 48 # "这台机器算不算存在"往回看多少小时
|
||||
|
||||
REPORT_DIR = "./monitor_reports"
|
||||
|
||||
DEFAULT_HISTORY_HOURS = 24 # --history 不带数字时默认看多少小时
|
||||
|
||||
|
||||
def _dw(s):
|
||||
"""字符串显示宽度(中文/全角算2), 用于历史趋势表格对齐。"""
|
||||
return sum(2 if unicodedata.east_asian_width(c) in ("W", "F") else 1 for c in str(s))
|
||||
|
||||
|
||||
def _padl(s, w):
|
||||
"""按显示宽度左对齐(文字在前, 空格补在后面) —— 跟 hourly_stats.py 的同名函数保持一致约定。"""
|
||||
s = str(s)
|
||||
return s + " " * max(0, w - _dw(s))
|
||||
|
||||
|
||||
def _padr(s, w):
|
||||
"""按显示宽度右对齐(空格补在前面, 文字贴右边)。"""
|
||||
return " " * max(0, w - _dw(s)) + str(s)
|
||||
|
||||
|
||||
def target_hour(now=None) -> datetime:
|
||||
"""确定要汇总的目标整点小时: 最新一个已经过完的整点; 如果当前分钟数<5(刚跨过整点没多久,
|
||||
部分机器可能还没上传完), 再往回退一个小时留缓冲。"""
|
||||
now = now or datetime.now()
|
||||
completed = now.replace(minute=0, second=0, microsecond=0) - timedelta(hours=1)
|
||||
if now.minute < 5:
|
||||
completed -= timedelta(hours=1)
|
||||
return completed
|
||||
|
||||
|
||||
def fetch_range_digests(end_hour_dt: datetime, hours_back: int) -> dict:
|
||||
"""拉取 [end_hour_dt - hours_back, end_hour_dt] 闭区间(按小时)所有机器的摘要,
|
||||
返回 {hour_key(YYYYMMDDHH): {machine_id: digest_dict} | None}。
|
||||
区分两种"没数据": 该小时【列COS目录本身失败】(比如COS暂时不可达)记为 None——这是需要
|
||||
main() 判断是否整体报错退出的信号; 该小时【列成功但目录是空的】(单纯没人上报)记为 {}——
|
||||
这是正常情况, 不代表故障。单台机器的摘要【下载】失败只跳过该机器、打印警告, 不影响这个
|
||||
小时其它机器的数据, 也不算列目录失败。"""
|
||||
result = {}
|
||||
for i in range(hours_back + 1):
|
||||
hdt = end_hour_dt - timedelta(hours=i)
|
||||
hkey = hdt.strftime("%Y%m%d%H")
|
||||
try:
|
||||
files = job_cos.list_hourly_digest_files_for_hour(hdt)
|
||||
except Exception as e:
|
||||
print(f"[WARN] 列出 {hkey} 的摘要文件失败: {e}")
|
||||
result[hkey] = None
|
||||
continue
|
||||
machines = {}
|
||||
for machine_id, key in files:
|
||||
try:
|
||||
machines[machine_id] = job_cos.download_hourly_digest(key)
|
||||
except Exception as e:
|
||||
print(f"[WARN] 下载 {machine_id} 在 {hkey} 的摘要失败, 跳过: {e}")
|
||||
result[hkey] = machines
|
||||
return result
|
||||
|
||||
|
||||
def compute_baseline(range_digests: dict, target_hdt: datetime) -> dict:
|
||||
"""算每台机器过去 BASELINE_HOURS 小时(不含目标小时本身)的 success_hours 均值。
|
||||
只用真实拉到的小时数据平均(缺的小时/列目录失败的小时都不计入, 不当0算, 避免拖低均值),
|
||||
没有任何数据的机器不出现在结果里。"""
|
||||
sums = {}
|
||||
counts = {}
|
||||
for i in range(1, BASELINE_HOURS + 1):
|
||||
hkey = (target_hdt - timedelta(hours=i)).strftime("%Y%m%d%H")
|
||||
for machine_id, digest in (range_digests.get(hkey) or {}).items():
|
||||
sums[machine_id] = sums.get(machine_id, 0.0) + digest.get("success_hours", 0.0)
|
||||
counts[machine_id] = counts.get(machine_id, 0) + 1
|
||||
return {m: sums[m] / counts[m] for m in sums}
|
||||
|
||||
|
||||
def known_machines_and_latest_hour(range_digests: dict) -> dict:
|
||||
"""扫全部拉到的小时, 返回 {machine_id: 该机器出现过的最新 hour_key}。列目录失败(None)的
|
||||
小时直接跳过, 不当"没有任何机器"处理(避免把偶发的单小时COS故障误判成全队一起掉线)。"""
|
||||
latest = {}
|
||||
for hkey, machines in range_digests.items():
|
||||
if machines is None:
|
||||
continue
|
||||
for machine_id in machines:
|
||||
if machine_id not in latest or hkey > latest[machine_id]:
|
||||
latest[machine_id] = hkey
|
||||
return latest
|
||||
|
||||
|
||||
def aggregate_history(range_digests: dict, hours_back: int, end_hdt: datetime) -> list:
|
||||
"""把 fetch_range_digests() 拉到的多小时摘要,按小时汇总成全队合计一行,按时间正序(从早到晚)
|
||||
返回 [(hour_key, machine_count, count, success, no_subtitle, error, success_hours, ok), ...]。
|
||||
ok=False 表示这个小时列COS目录本身失败(range_digests[hkey] is None), 其余字段填0/0.0;
|
||||
ok=True 且 machine_count=0 是"列目录成功但没人上报"的正常空窗口(比如还没到那个整点),
|
||||
跟 ok=False(拉取本身失败)是两码事,不能混在一起看成同一种"没数据"。"""
|
||||
rows = []
|
||||
for i in range(hours_back, -1, -1):
|
||||
hdt = end_hdt - timedelta(hours=i)
|
||||
hkey = hdt.strftime("%Y%m%d%H")
|
||||
machines = range_digests.get(hkey)
|
||||
if machines is None:
|
||||
rows.append((hkey, 0, 0, 0, 0, 0, 0.0, False))
|
||||
continue
|
||||
count = sum(d.get("count", 0) for d in machines.values())
|
||||
success = sum(d.get("success", 0) for d in machines.values())
|
||||
no_sub = sum(d.get("no_subtitle", 0) for d in machines.values())
|
||||
error = sum(d.get("error", 0) for d in machines.values())
|
||||
succ_hours = sum(d.get("success_hours", 0.0) for d in machines.values())
|
||||
rows.append((hkey, len(machines), count, success, no_sub, error, succ_hours, True))
|
||||
return rows
|
||||
|
||||
|
||||
def print_history(hours_back: int):
|
||||
"""打印全队历史多个小时的汇总趋势(拉COS各机器已上传的小时摘要, 不依赖本机 数据.jsonl,
|
||||
只要各机器在跑带小时摘要上传功能的新版 run_worker.py 就能看到)。跟 tools/hourly_stats.py
|
||||
的单机版本是同一个"按process_start所在小时分桶计数"的逻辑, 只是数据源从本机 数据.jsonl
|
||||
换成了COS上各机器上传的小时摘要, 且是全队加总, 不是单机。"""
|
||||
thdt = target_hour()
|
||||
print(f"目标小时: {thdt.strftime('%Y-%m-%d %H:00')} (往前拉 {hours_back} 小时, 共 {hours_back+1} 个小时)")
|
||||
range_digests = fetch_range_digests(thdt, hours_back)
|
||||
rows = aggregate_history(range_digests, hours_back, thdt)
|
||||
|
||||
# 每列宽度都比内容本身多留至少2个显示宽度的间隔, 避免"小时"这种定长字符串(如
|
||||
# "2026-07-09 13:00"共16字符)紧贴着下一列数字、看起来像连在一起。
|
||||
W = {"小时": 18, "机器数": 8, "视频数": 8, "成功": 6, "无字幕": 8, "错误": 6, "产出小时数": 12}
|
||||
header = (_padl("小时", W["小时"]) + _padr("机器数", W["机器数"]) + _padr("视频数", W["视频数"]) +
|
||||
_padr("成功", W["成功"]) + _padr("无字幕", W["无字幕"]) + _padr("错误", W["错误"]) +
|
||||
_padr("产出小时数", W["产出小时数"]))
|
||||
width = sum(W.values())
|
||||
print("=" * width)
|
||||
print(header)
|
||||
print("-" * width)
|
||||
|
||||
tot = {"count": 0, "success": 0, "no_subtitle": 0, "error": 0, "succ_hours": 0.0}
|
||||
fail_hours = []
|
||||
for hkey, mcount, count, success, no_sub, error, succ_hours, ok in rows:
|
||||
disp_hour = f"{hkey[:4]}-{hkey[4:6]}-{hkey[6:8]} {hkey[8:10]}:00"
|
||||
if not ok:
|
||||
print(_padl(disp_hour, W["小时"]) + _padr("(拉取失败)", width - W["小时"]))
|
||||
fail_hours.append(hkey)
|
||||
continue
|
||||
print(_padl(disp_hour, W["小时"]) + _padr(mcount, W["机器数"]) + _padr(count, W["视频数"]) +
|
||||
_padr(success, W["成功"]) + _padr(no_sub, W["无字幕"]) + _padr(error, W["错误"]) +
|
||||
_padr(f"{succ_hours:.1f}", W["产出小时数"]))
|
||||
tot["count"] += count
|
||||
tot["success"] += success
|
||||
tot["no_subtitle"] += no_sub
|
||||
tot["error"] += error
|
||||
tot["succ_hours"] += succ_hours
|
||||
|
||||
print("-" * width)
|
||||
nh = len(rows) - len(fail_hours)
|
||||
if nh > 0:
|
||||
print(_padl("合计/时均", W["小时"]) + _padr(tot["count"], W["视频数"] + W["机器数"]) +
|
||||
_padr(tot["success"], W["成功"]) + _padr(tot["no_subtitle"], W["无字幕"]) +
|
||||
_padr(tot["error"], W["错误"]) + _padr(f"{tot['succ_hours']:.1f}", W["产出小时数"]))
|
||||
print(f"\n共 {nh} 个有效小时 | 时均处理 {tot['count']/nh:.0f} 条 | "
|
||||
f"时均产出 {tot['succ_hours']/nh:.1f} 小时视频内容")
|
||||
if fail_hours:
|
||||
print(f"\n({len(fail_hours)} 个小时COS拉取失败, 已跳过: {', '.join(fail_hours)})")
|
||||
|
||||
|
||||
def detect_anomalies(target_hdt: datetime, target_digests: dict,
|
||||
baseline: dict, latest_hour: dict) -> list:
|
||||
"""跑4条异常检测规则, 返回 [(level, message), ...]，level ∈ {"严重","警告"}。"""
|
||||
anomalies = []
|
||||
|
||||
# ① 机器掉线: 最新一次上报距目标小时 >= OFFLINE_HOURS
|
||||
for machine_id, last_hkey in latest_hour.items():
|
||||
last_hdt = datetime.strptime(last_hkey, "%Y%m%d%H")
|
||||
gap_hours = (target_hdt - last_hdt).total_seconds() / 3600
|
||||
if gap_hours >= OFFLINE_HOURS:
|
||||
anomalies.append(("严重", f"机器 {machine_id} 已连续{gap_hours:.0f}小时无上报数据，可能停机"))
|
||||
|
||||
# ② 单小时成功率骤降
|
||||
for machine_id, digest in target_digests.items():
|
||||
succ, err = digest.get("success", 0), digest.get("error", 0)
|
||||
sample = succ + err
|
||||
if sample >= SUCCESS_RATE_MIN_SAMPLE:
|
||||
rate = succ / sample
|
||||
if rate < SUCCESS_RATE_MIN:
|
||||
anomalies.append(("警告", f"机器 {machine_id} 本小时成功率仅{rate*100:.0f}%({succ}/{sample})"))
|
||||
|
||||
# ③ 单机吞吐骤降
|
||||
for machine_id, digest in target_digests.items():
|
||||
base = baseline.get(machine_id)
|
||||
if base and base > 0:
|
||||
cur = digest.get("success_hours", 0.0)
|
||||
if cur < base * MACHINE_DROP_RATIO:
|
||||
anomalies.append(("警告", f"机器 {machine_id} 本小时产出{cur:.1f}小时，远低于24h均值{base:.1f}小时"))
|
||||
|
||||
# ④ 全队吞吐骤降
|
||||
fleet_cur = sum(d.get("success_hours", 0.0) for d in target_digests.values())
|
||||
fleet_base = sum(baseline.values())
|
||||
if fleet_base > 0 and fleet_cur < fleet_base * FLEET_DROP_RATIO:
|
||||
anomalies.append(("严重", f"全队本小时总产出{fleet_cur:.1f}小时，远低于24h均值{fleet_base:.1f}小时，可能是全局性问题(比如OCR端点故障)"))
|
||||
|
||||
return anomalies
|
||||
|
||||
|
||||
def render_report(target_hdt: datetime, target_digests: dict,
|
||||
baseline: dict, anomalies: list) -> str:
|
||||
"""生成 Markdown 报告文本。"""
|
||||
lines = [f"# 全队监控报告 {target_hdt.strftime('%Y-%m-%d %H:00')}", ""]
|
||||
lines.append("## 异常告警")
|
||||
if anomalies:
|
||||
for level, msg in anomalies:
|
||||
icon = "🔴" if level == "严重" else "🟡"
|
||||
lines.append(f"- {icon} [{level}] {msg}")
|
||||
else:
|
||||
lines.append("✅ 本小时一切正常")
|
||||
lines.append("")
|
||||
lines.append("## 本小时明细")
|
||||
lines.append("| 机器 | 视频数 | 成功 | 无字幕 | 错误 | 产出小时数 |")
|
||||
lines.append("|---|---|---|---|---|---|")
|
||||
for machine_id in sorted(target_digests):
|
||||
d = target_digests[machine_id]
|
||||
lines.append(f"| {machine_id} | {d.get('count',0)} | {d.get('success',0)} | "
|
||||
f"{d.get('no_subtitle',0)} | {d.get('error',0)} | {d.get('success_hours',0):.1f} |")
|
||||
lines.append("")
|
||||
lines.append("## 过去24小时全队趋势")
|
||||
fleet_cur = sum(d.get("success_hours", 0.0) for d in target_digests.values())
|
||||
fleet_base = sum(baseline.values())
|
||||
lines.append(f"全队本小时产出小时数总和：{fleet_cur:.1f}，24小时均值：{fleet_base:.1f}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def _parse_history_arg(argv):
|
||||
"""解析 --history [N] —— 带上就返回要看多少小时(不带数字用默认值), 没带这个参数返回 None
|
||||
(跑原来的单小时报告模式)。"""
|
||||
if "--history" not in argv:
|
||||
return None
|
||||
i = argv.index("--history")
|
||||
if i + 1 < len(argv):
|
||||
try:
|
||||
return int(argv[i + 1])
|
||||
except ValueError:
|
||||
pass
|
||||
return DEFAULT_HISTORY_HOURS
|
||||
|
||||
|
||||
def main():
|
||||
hours_back = _parse_history_arg(sys.argv[1:])
|
||||
if hours_back is not None:
|
||||
print_history(hours_back)
|
||||
return
|
||||
|
||||
thdt = target_hour()
|
||||
print(f"目标小时: {thdt.strftime('%Y-%m-%d %H:00')}")
|
||||
|
||||
range_digests = fetch_range_digests(thdt, KNOWN_MACHINES_LOOKBACK)
|
||||
target_hkey = thdt.strftime("%Y%m%d%H")
|
||||
target_slot = range_digests.get(target_hkey)
|
||||
if target_slot is None:
|
||||
# 目标小时本身列目录都失败了(COS大概率整体不可达) -> 按设计文档"错误处理"要求直接报错退出,
|
||||
# 不生成一份看起来正常、实际数据全空的假报告掩盖问题。
|
||||
print(f"[FAIL] 目标小时 {target_hkey} 的摘要目录列取失败(COS可能不可达), 中止, 不生成报告")
|
||||
sys.exit(1)
|
||||
target_digests = target_slot
|
||||
if not target_digests:
|
||||
print(f"[WARN] 目标小时 {target_hkey} 没有拉到任何机器的摘要, 可能还没到点(正常空目录, 不是故障)")
|
||||
|
||||
baseline = compute_baseline(range_digests, thdt)
|
||||
latest_hour = known_machines_and_latest_hour(range_digests)
|
||||
anomalies = detect_anomalies(thdt, target_digests, baseline, latest_hour)
|
||||
|
||||
report = render_report(thdt, target_digests, baseline, anomalies)
|
||||
|
||||
os.makedirs(REPORT_DIR, exist_ok=True)
|
||||
local_path = os.path.join(REPORT_DIR, f"{thdt.strftime('%Y%m%d_%H')}.md")
|
||||
with open(local_path, "w", encoding="utf-8") as f:
|
||||
f.write(report)
|
||||
print(f"[OK] 报告已写入 {local_path}")
|
||||
|
||||
try:
|
||||
dst = job_cos.upload_monitor_report(local_path, thdt)
|
||||
if dst:
|
||||
print(f"[OK] 报告已上传 -> {dst}")
|
||||
except Exception as e:
|
||||
print(f"[WARN] 报告上传COS失败(本地文件已保留): {e}")
|
||||
|
||||
if anomalies:
|
||||
print(f"\n共检测到 {len(anomalies)} 条异常，详见报告。")
|
||||
else:
|
||||
print("\n本小时一切正常。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
47
tools/gen_test_srt.py
Normal file
47
tools/gen_test_srt.py
Normal file
@@ -0,0 +1,47 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""用真实OCR数据生成一份【已过滤水印】的测试SRT。跳过读图(monkeypatch)。
|
||||
用法: python temp/gen_test_srt.py <composite目录> <输出srt>
|
||||
"""
|
||||
import sys, os, json
|
||||
sys.path.insert(0, os.path.abspath("."))
|
||||
import tool.getsrt as G
|
||||
from tool.log.log_manager import UnifiedLogger
|
||||
|
||||
G.find_best_ocr_match = lambda *a, **k: (None, 0) # 跳过首尾音频匹配的读图
|
||||
|
||||
D = sys.argv[1] if len(sys.argv) > 1 else "test_srt/BV1BamdYLExJ"
|
||||
OUT = sys.argv[2] if len(sys.argv) > 2 else "test_srt/BV1BamdYLExJ.filtered.srt"
|
||||
META = os.path.join(D, "metadata.json")
|
||||
logger = UnifiedLogger(); logger.reset(); logger.init_logging("rerun_log.json"); logger.start_program("rerun")
|
||||
|
||||
pairs = []
|
||||
lines = open(os.path.join(D, "ocr_result.jsonl"), encoding="utf-8").readlines()
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
if i + 1 < len(lines) and lines[i].startswith("pic_path: "):
|
||||
fn = os.path.basename(lines[i].strip()[len("pic_path: "):])
|
||||
try:
|
||||
pairs.append([os.path.join(D, fn), json.loads(lines[i + 1])]); i += 2; continue
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
i += 1
|
||||
|
||||
# 坐标水印过滤(就是接进 getsrt 的那个)
|
||||
G._filter_watermark_by_position(pairs, logger)
|
||||
|
||||
best = G.analyze_subtitle_blocks(pairs, META, logger)
|
||||
h = G.analyze_h(pairs, META)
|
||||
results = []
|
||||
for idx in range(len(pairs)):
|
||||
try:
|
||||
text, conf, flag, sim = G.extract_highest_subtitle(idx, pairs, best, META, h, logger)
|
||||
except Exception:
|
||||
text = ""
|
||||
if text and not G.check_subtitle_quality_enhanced(text) and conf <= 90:
|
||||
text = ""
|
||||
results.append((idx, text or ""))
|
||||
# 静态文字过滤(频率, 兜底)
|
||||
results = G._filter_static_text(results, logger)
|
||||
G.generate_srt_from_results(results, OUT, interval_seconds=0.33)
|
||||
n = sum(1 for l in open(OUT, encoding="utf-8") if "-->" in l)
|
||||
print("生成: %s 共 %d 条字幕" % (OUT, n))
|
||||
174
tools/hourly_stats.py
Normal file
174
tools/hourly_stats.py
Normal file
@@ -0,0 +1,174 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
按【小时】统计 数据.jsonl：每小时处理多少条视频、视频总时长、成功情况、处理倍速。
|
||||
小时取 process_start（没有则回退 timestamp）的 YYYY-MM-DD HH。
|
||||
|
||||
"处理倍速" = 该小时内处理掉的【视频内容总时长】 / 1 墙钟小时。
|
||||
例: 一个墙钟小时里处理了 40 小时的视频内容 → 倍速 40x → 等效有效并发≈40。
|
||||
这个数最能反映真实吞吐/并发利用率(对照 max-workers 看是否吃满)。
|
||||
|
||||
用法(项目根目录运行):
|
||||
python temp/hourly_stats.py [数据.jsonl路径] [--last N]
|
||||
--last N : 只看最近 N 个有数据的小时(默认全部)
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
import unicodedata
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta
|
||||
|
||||
|
||||
def _parse(ts):
|
||||
try:
|
||||
return datetime.fromisoformat(ts)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _dw(s):
|
||||
"""字符串显示宽度(中文/全角算2)。"""
|
||||
return sum(2 if unicodedata.east_asian_width(c) in ("W", "F") else 1 for c in str(s))
|
||||
|
||||
|
||||
def _padr(s, w):
|
||||
"""按显示宽度右对齐。"""
|
||||
s = str(s)
|
||||
return " " * max(0, w - _dw(s)) + s
|
||||
|
||||
|
||||
def _padl(s, w):
|
||||
"""按显示宽度左对齐。"""
|
||||
s = str(s)
|
||||
return s + " " * max(0, w - _dw(s))
|
||||
|
||||
|
||||
# 列宽(显示宽度)
|
||||
COLW = {"小时": 16, "视频数": 8, "成功": 6, "无字幕": 8, "错误": 6,
|
||||
"总时长h": 10, "成功h": 9, "处理倍速": 10, "有效并发": 10}
|
||||
|
||||
|
||||
def _row(hour, count, success, no_sub, error, dur_h, ok_h, speed, eff):
|
||||
return (_padl(hour, COLW["小时"]) + _padr(count, COLW["视频数"]) +
|
||||
_padr(success, COLW["成功"]) + _padr(no_sub, COLW["无字幕"]) +
|
||||
_padr(error, COLW["错误"]) + _padr(dur_h, COLW["总时长h"]) +
|
||||
_padr(ok_h, COLW["成功h"]) + _padr(speed, COLW["处理倍速"]) +
|
||||
_padr(eff, COLW["有效并发"]))
|
||||
|
||||
args = [a for a in sys.argv[1:]]
|
||||
last_n = None
|
||||
if "--last" in args:
|
||||
i = args.index("--last")
|
||||
try:
|
||||
last_n = int(args[i + 1])
|
||||
del args[i:i + 2]
|
||||
except (IndexError, ValueError):
|
||||
last_n = None
|
||||
JSONL_PATH = args[0] if args else "数据.jsonl"
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
f = open(JSONL_PATH, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {JSONL_PATH}")
|
||||
return
|
||||
|
||||
hours = defaultdict(lambda: {"count": 0, "success": 0, "no_subtitle": 0,
|
||||
"error": 0, "dur": 0.0, "dur_ok": 0.0, "elapsed": 0.0})
|
||||
busy = defaultdict(float) # 每个墙钟小时内, 所有视频实际占用的总秒数(精确并发用)
|
||||
no_date = 0
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
ts = rec.get("process_start") or rec.get("timestamp") or ""
|
||||
hour = ts[:13] if len(ts) >= 13 else None # YYYY-MM-DD HH(按开始时间归桶)
|
||||
if not hour:
|
||||
no_date += 1
|
||||
continue
|
||||
|
||||
h = hours[hour]
|
||||
h["count"] += 1
|
||||
status = rec.get("status")
|
||||
if status == "success":
|
||||
h["success"] += 1
|
||||
elif status == "no_subtitle":
|
||||
h["no_subtitle"] += 1
|
||||
else:
|
||||
h["error"] += 1
|
||||
|
||||
try:
|
||||
dur = float(rec.get("duration") or 0)
|
||||
h["dur"] += dur
|
||||
if status == "success":
|
||||
h["dur_ok"] += dur
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
try:
|
||||
h["elapsed"] += float(rec.get("elapsed_seconds") or 0)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
# 精确并发: 把 [process_start, process_end] 的时长按实际跨越的每个小时切开累加,
|
||||
# 避免"长视频整段算进开始那个小时"导致失真。
|
||||
st = _parse(rec.get("process_start"))
|
||||
en = _parse(rec.get("process_end"))
|
||||
if st and en and en > st:
|
||||
cur = st
|
||||
while cur < en:
|
||||
hk = cur.strftime("%Y-%m-%dT%H")
|
||||
nxt = cur.replace(minute=0, second=0, microsecond=0) + timedelta(hours=1)
|
||||
seg_end = min(en, nxt)
|
||||
busy[hk] += (seg_end - cur).total_seconds()
|
||||
cur = seg_end
|
||||
|
||||
if not hours:
|
||||
print("没有可统计的数据")
|
||||
return
|
||||
|
||||
keys = sorted(hours)
|
||||
if last_n:
|
||||
keys = keys[-last_n:]
|
||||
|
||||
width = sum(COLW.values())
|
||||
print(f"文件: {JSONL_PATH}")
|
||||
print("=" * width)
|
||||
print(_row("小时", "视频数", "成功", "无字幕", "错误", "总时长h", "成功h", "处理倍速", "有效并发"))
|
||||
print("-" * width)
|
||||
|
||||
tot = {"count": 0, "success": 0, "no_subtitle": 0, "error": 0,
|
||||
"dur": 0.0, "dur_ok": 0.0, "elapsed": 0.0}
|
||||
busy_sum = 0.0
|
||||
for hour in keys:
|
||||
h = hours[hour]
|
||||
speed = h["dur"] / 3600 # 处理倍速: 视频内容小时数 / 1墙钟小时
|
||||
eff = busy[hour] / 3600 # 有效并发: 该小时所有视频实际占用秒数 / 3600(=平均并发)
|
||||
busy_sum += busy[hour]
|
||||
print(_row(hour, h["count"], h["success"], h["no_subtitle"], h["error"],
|
||||
f"{h['dur']/3600:.1f}", f"{h['dur_ok']/3600:.1f}",
|
||||
f"{speed:.1f}x", f"{eff:.1f}"))
|
||||
for k in tot:
|
||||
tot[k] += h[k]
|
||||
|
||||
print("-" * width)
|
||||
nh = len(keys)
|
||||
print(_row("合计/均值", tot["count"], tot["success"], tot["no_subtitle"], tot["error"],
|
||||
f"{tot['dur']/3600:.1f}", f"{tot['dur_ok']/3600:.1f}",
|
||||
f"{tot['dur']/3600/nh:.1f}x", f"{busy_sum/3600/nh:.1f}"))
|
||||
if no_date:
|
||||
print(f"\n（{no_date} 条无时间字段，已跳过）")
|
||||
print(f"\n共 {nh} 个小时 | 时均处理 {tot['count']/nh:.0f} 条 | "
|
||||
f"时均成功 {tot['success']/nh:.0f} 条")
|
||||
if tot["count"]:
|
||||
print(f"成功率 {tot['success']/tot['count']*100:.1f}%")
|
||||
print("提示: '有效并发'明显小于你设的 max-workers → 有串行瓶颈(GIL/锁/等待),不是并发不够。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
142
tools/patch_config.py
Normal file
142
tools/patch_config.py
Normal file
@@ -0,0 +1,142 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""在【项目根目录】运行: 把本次更新需要的 config.json 配置键就地改好。
|
||||
只动下面列出的键, 绝不碰你的密钥/COS路径/其它已调过的值。幂等, 可重复运行。
|
||||
|
||||
用法: python3 tools/patch_config.py
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
P = "config/config.json"
|
||||
if not os.path.exists(P):
|
||||
print("✗ 找不到 config/config.json, 请 cd 到项目根目录再运行")
|
||||
sys.exit(1)
|
||||
|
||||
d = json.load(open(P, encoding="utf-8"))
|
||||
sf = d.setdefault("splitframe", {})
|
||||
sub = d.setdefault("subtitle", {})
|
||||
ocr = d.setdefault("tencentcloud", {}).setdefault("ocr", {})
|
||||
dp = d.setdefault("dispatch", {})
|
||||
|
||||
changes = []
|
||||
|
||||
# -1) COS 任务分发(常驻 worker 认领任务 + 每日 数据.jsonl 备份)。缺失才加, 不动已有值。
|
||||
# jobs_root=取任务地址; daily_backup_path=数据.jsonl 备份位置; machine_id 留空=用主机名/环境变量。
|
||||
_bkt = (d.get("tencentcloud", {}).get("cos", {}) or {}).get("bucket", "asr-dig-data-1253181912")
|
||||
_dispatch_defaults = [
|
||||
("enabled", True),
|
||||
("source", "cos"), # cos=从COS认领 / local=只读本地 ./text2(启动脚本据此选模式)
|
||||
("run_long_video", False), # true=本机专跑超长队列(long_list_backup_path)
|
||||
("auto_fallback_long_video", True), # false=主队列空闲时也不兼跑长视频队列,只等主队列
|
||||
("global_dedup_on_startup", True), # false=启动时不拉全队去重清单,只按本机 数据.jsonl 去重
|
||||
("machine_id", ""),
|
||||
("jobs_root", "cos://%s/_jobs" % _bkt),
|
||||
("shard_size", 2000),
|
||||
("keep_alive", True),
|
||||
("idle_poll_seconds", 60),
|
||||
("claim_shuffle", True),
|
||||
("claim_match", ""),
|
||||
("heartbeat_seconds", 300),
|
||||
("daily_backup_path", "cos://%s/_jobs/daily" % _bkt),
|
||||
("long_list_backup_path", "cos://%s/_jobs/long_video" % _bkt),
|
||||
("monitor_root", "cos://%s/_jobs/monitor" % _bkt),
|
||||
("long_list_clear_after_upload", True),
|
||||
("long_list_min_batch", 1000), # 领取长视频队列时凑够这么多条任务才开始处理一批
|
||||
("daily_backup_hour", 10),
|
||||
("stop_flag_key", "_jobs/STOP"),
|
||||
]
|
||||
for k, v in _dispatch_defaults:
|
||||
if k not in dp:
|
||||
dp[k] = v
|
||||
changes.append("dispatch.%s -> %s (新增, COS任务分发)" % (k, v))
|
||||
|
||||
# 0) OCR 重试封顶 + 单次超时 (防"端点全超时时一张图卡~1小时"拖死worker)
|
||||
if "max_attempts" not in ocr:
|
||||
ocr["max_attempts"] = 4
|
||||
changes.append("ocr.max_attempts -> 4 (新增, 重试封顶)")
|
||||
# 单次OCR超时: 调到30s(大图负载下最慢~12s, 12s砍太狠; 有max_attempts封顶不会卡死)
|
||||
if ocr.get("timeout", 0) < 30:
|
||||
_old = ocr.get("timeout")
|
||||
ocr["timeout"] = 30
|
||||
changes.append("ocr.timeout %s -> 30 (大图留足余量)" % _old)
|
||||
if "dispatch_log_every" not in ocr:
|
||||
ocr["dispatch_log_every"] = 1000
|
||||
changes.append("ocr.dispatch_log_every -> 1000 (新增, 每1000次派发落盘计数, 0=关)")
|
||||
|
||||
# 0b) 处理完删源视频(本来就在COS上, 本地是冗余副本)。防 ./video/ 堆积撑爆磁盘。
|
||||
if sf.get("delete_video") is not True:
|
||||
sf["delete_video"] = True
|
||||
changes.append("delete_video -> True (处理完删源视频, 防磁盘爆)")
|
||||
|
||||
# 1) 抽帧 ffmpeg 线程数: 统一收敛到 "auto"(按 CPU核数÷并发进程数 自动摊分, GPU/CPU两条
|
||||
# 抽帧路径现在都吃这个值了, 比固定数字更适合核数不一的机队)。已经是 "auto" 的跳过；
|
||||
# 只升级明显偏低的字面值(<5)或完全没配的情况, 已手动调更高的字面值保留不动。
|
||||
old = sf.get("ffmpeg_threads", 2)
|
||||
_old_is_auto = isinstance(old, str) and old.strip().lower() == "auto"
|
||||
if not _old_is_auto and (not isinstance(old, (int, float)) or old < 5):
|
||||
sf["ffmpeg_threads"] = "auto"
|
||||
changes.append(f"ffmpeg_threads {old!r} -> auto")
|
||||
|
||||
# 2) 超长视频补跑列表 (缺失才加)
|
||||
if "long_video_list" not in sub:
|
||||
sub["long_video_list"] = "./超长待重跑.txt"
|
||||
changes.append("long_video_list -> ./超长待重跑.txt (新增)")
|
||||
|
||||
# 2b) 降低 OCR 拼接帧数 (大图OCR>12s会超时; 只往下调, 已更小的保留)
|
||||
_mt = sub.get("max_tiles_per_composite", 40)
|
||||
if _mt > 24:
|
||||
sub["max_tiles_per_composite"] = 24
|
||||
changes.append("max_tiles_per_composite %d -> 24 (拼接图减小, 防OCR超时)" % _mt)
|
||||
|
||||
# 2b2) 降全局OCR并发, 削掉过载突发尾延迟(端点平均快但突发会排队爆30s); 只往下调
|
||||
_mg = sub.get("max_global_ocr", 80)
|
||||
if _mg > 50:
|
||||
sub["max_global_ocr"] = 50
|
||||
changes.append("max_global_ocr %d -> 50 (降负载削尾延迟)" % _mg)
|
||||
|
||||
# 2b3) OCR图片像素上限: 超大合成图(区域过高)缩小再OCR, 防单张>30s超时
|
||||
if "ocr_max_image_side" not in sub:
|
||||
sub["ocr_max_image_side"] = 4000
|
||||
changes.append("ocr_max_image_side -> 4000 (新增, 超大图缩小防超时)")
|
||||
|
||||
# 2c) 不丢帧: OCR多轮重试, 仍失败则整段视频判失败去重跑(不产出带洞SRT)
|
||||
for k, v in [("ocr_no_drop", True), ("ocr_max_passes", 5),
|
||||
("ocr_retry_sleep", 2), ("ocr_fail_list", "./OCR失败待重跑.txt")]:
|
||||
if k not in sub:
|
||||
sub[k] = v
|
||||
changes.append("%s -> %s (新增, 不丢帧)" % (k, v))
|
||||
|
||||
# 2d) 低置信占比过滤: 幻灯片/编程视频OCR出垃圾(低置信50%+)且吃3-5倍OCR, 整段判非字幕跳过
|
||||
for k, v in [("filter_lowconf", True), ("lowconf_thr", 80),
|
||||
("lowconf_filter_frac", 0.6), ("lowconf_min_det", 200)]:
|
||||
if k not in sub:
|
||||
sub[k] = v
|
||||
changes.append("%s -> %s (新增, 过滤幻灯片/代码类非字幕视频)" % (k, v))
|
||||
|
||||
# 2e) 状态归类清单: 这些原因是"无可用字幕的正常过滤", 应记 no_subtitle 而非 error。
|
||||
# 旧逻辑把字幕太少/冗余/置信度/非字幕等错记成 error(线上 error 中~79%其实是它们)。缺失才加。
|
||||
if "no_subtitle_reasons" not in sub:
|
||||
sub["no_subtitle_reasons"] = [
|
||||
"字幕太少", "冗余文字过多", "冗余文字太多", "无字幕区域",
|
||||
"非字幕内容", "置信度太低", "未检测", "特殊字体",
|
||||
"视频过短", "视频过长",
|
||||
]
|
||||
changes.append("no_subtitle_reasons -> [10项] (新增, 过滤型结论归 no_subtitle 不计 error)")
|
||||
|
||||
# 3) 抽帧自动选空闲GPU (避开被OCR服务占用的卡, 缺失才加)
|
||||
if "gpu_auto_select" not in sf:
|
||||
sf["gpu_auto_select"] = True
|
||||
changes.append("gpu_auto_select -> True (新增, 抽帧自动避开被占用的GPU)")
|
||||
if "gpu_busy_mem_mb" not in sf:
|
||||
sf["gpu_busy_mem_mb"] = 4096
|
||||
changes.append("gpu_busy_mem_mb -> 4096 (新增, 显存占用超此判为忙)")
|
||||
sf.setdefault("gpu_skip_indices", []) # 手动跳过列表(默认空, 交给自动探测)
|
||||
|
||||
if changes:
|
||||
json.dump(d, open(P, "w", encoding="utf-8"), ensure_ascii=False, indent=2)
|
||||
print("✓ config 已更新:")
|
||||
for c in changes:
|
||||
print(" -", c)
|
||||
else:
|
||||
print("✓ config 无需改动(已是最新)")
|
||||
41
tools/release_machine.py
Normal file
41
tools/release_machine.py
Normal file
@@ -0,0 +1,41 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""人工救援: 当某台机器【永久报废】(不会再重启续做)时, 把它名下所有未完成分片的锁删掉,
|
||||
任务退回 todo 让别的机器重新认领。
|
||||
|
||||
⚠️ 只在确认机器真报废后用。机器只是重启的话, 它自己会续做, 不要跑这个。
|
||||
|
||||
用法:
|
||||
python tools/release_machine.py <机器ID> # 实际释放
|
||||
python tools/release_machine.py <机器ID> --dry-run # 只列出, 不删
|
||||
"""
|
||||
import sys
|
||||
import os
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from tool import job_cos # noqa: E402
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) < 2:
|
||||
print("用法: python tools/release_machine.py <机器ID> [--dry-run]")
|
||||
sys.exit(1)
|
||||
target = sys.argv[1]
|
||||
dry = "--dry-run" in sys.argv[2:]
|
||||
|
||||
shards = job_cos.my_open_claims(target)
|
||||
if not shards:
|
||||
print(f"机器 {target} 名下没有未完成分片。")
|
||||
return
|
||||
print(f"机器 {target} 名下未完成分片 {len(shards)} 个:")
|
||||
for s in shards:
|
||||
print(f" - {s}")
|
||||
if dry:
|
||||
print("(--dry-run, 未删除)")
|
||||
return
|
||||
for s in shards:
|
||||
job_cos._delete(job_cos._key("claimed", s)) # 只删锁; todo 仍在 -> 别人会重领
|
||||
print(f"✓ 已释放 {len(shards)} 个分片的锁, 它们会被其他机器重新认领。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
99
tools/rerun_srt.py
Normal file
99
tools/rerun_srt.py
Normal file
@@ -0,0 +1,99 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
从已有 OCR 数据重新生成 SRT，不重新调用 OCR API。
|
||||
用法: python rerun_srt.py
|
||||
"""
|
||||
import sys, os, json
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
|
||||
from difflib import SequenceMatcher
|
||||
from tool.log.log_manager import UnifiedLogger
|
||||
from tool.getsrt import (
|
||||
analyze_subtitle_blocks, analyze_h,
|
||||
extract_highest_subtitle, check_subtitle_quality_enhanced,
|
||||
evaluate_text_quality_rules, generate_srt_from_results
|
||||
)
|
||||
|
||||
COMPOSITE_DIR = sys.argv[1] if len(sys.argv) > 1 else "./composite/BV1aL4y1p7Bi_p11"
|
||||
JSONL_PATH = os.path.join(COMPOSITE_DIR, "ocr_result.jsonl")
|
||||
META_PATH = os.path.join(COMPOSITE_DIR, "metadata.json")
|
||||
OUTPUT_SRT = os.path.join(COMPOSITE_DIR, "BV1aL4y1p7Bi_p11.new.srt")
|
||||
INTERVAL = 0.33
|
||||
|
||||
logger = UnifiedLogger()
|
||||
logger.reset()
|
||||
logger.init_logging("rerun_log.json")
|
||||
logger.start_program("rerun")
|
||||
|
||||
# 读取已有 OCR 数据，将旧路径修正为当前实际路径
|
||||
all_pic_ocr_pair = []
|
||||
with open(JSONL_PATH, 'r', encoding='utf-8') as f:
|
||||
lines = f.readlines()
|
||||
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
if i + 1 < len(lines) and lines[i].startswith("pic_path: "):
|
||||
raw_path = lines[i].strip()[len("pic_path: "):]
|
||||
filename = os.path.basename(raw_path)
|
||||
actual_path = os.path.join(COMPOSITE_DIR, filename)
|
||||
try:
|
||||
ocr_json = json.loads(lines[i + 1])
|
||||
all_pic_ocr_pair.append([actual_path, ocr_json])
|
||||
i += 2
|
||||
continue
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
i += 1
|
||||
|
||||
print(f"读取 {len(all_pic_ocr_pair)} 帧的OCR数据")
|
||||
|
||||
# 分析字幕网格位置
|
||||
best_grids = analyze_subtitle_blocks(all_pic_ocr_pair, META_PATH, logger)
|
||||
h = analyze_h(all_pic_ocr_pair, META_PATH)
|
||||
|
||||
if not best_grids.get("composite"):
|
||||
print("ERROR: 未找到合适的字幕区域")
|
||||
sys.exit(1)
|
||||
|
||||
# 逐帧提取字幕
|
||||
results = []
|
||||
DEBUG_KEYWORD = "曹小月从角落里拿出雨伞"
|
||||
for idx in range(len(all_pic_ocr_pair)):
|
||||
print(f"\r处理 {idx + 1}/{len(all_pic_ocr_pair)}", end="", flush=True)
|
||||
text, conf, flag, sim = extract_highest_subtitle(
|
||||
idx, all_pic_ocr_pair, best_grids, META_PATH, h, logger
|
||||
)
|
||||
if text and not check_subtitle_quality_enhanced(text) and conf <= 90:
|
||||
text = ""
|
||||
# 打印包含关键词的帧的原始OCR和提取结果
|
||||
if DEBUG_KEYWORD in (text or ""):
|
||||
pic_path, ocr_json = all_pic_ocr_pair[idx]
|
||||
print(f"\n[DEBUG 帧{idx}] 提取: {repr(text)}")
|
||||
for td in ocr_json['TextDetections']:
|
||||
print(f" box: {repr(td['DetectedText'])} conf={td['Confidence']} watermark={td.get('is_watermark','?')}")
|
||||
results.append((idx, text or ""))
|
||||
|
||||
non_empty = sum(1 for _, t in results if t)
|
||||
print(f"\n共 {non_empty}/{len(results)} 帧有字幕")
|
||||
|
||||
# 生成 SRT
|
||||
generate_srt_from_results(results, OUTPUT_SRT, interval_seconds=INTERVAL)
|
||||
print(f"SRT 已输出: {OUTPUT_SRT}")
|
||||
|
||||
# 展示问题字幕附近的结果
|
||||
with open(OUTPUT_SRT, 'r', encoding='utf-8') as f:
|
||||
srt_content = f.readlines()
|
||||
|
||||
print("\n=== 修复验证（'曹小月从角落里拿出雨伞' 附近） ===")
|
||||
found = False
|
||||
for j, line in enumerate(srt_content):
|
||||
if "曹小月从角落里拿出雨伞" in line:
|
||||
start = max(0, j - 3)
|
||||
end = min(len(srt_content), j + 10)
|
||||
print("".join(srt_content[start:end]))
|
||||
print("---")
|
||||
found = True
|
||||
break
|
||||
|
||||
if not found:
|
||||
print("未在输出 SRT 中找到目标字幕，请检查提取结果。")
|
||||
89
tools/seed_jobs.py
Normal file
89
tools/seed_jobs.py
Normal file
@@ -0,0 +1,89 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""把一份/多份视频清单切成分片, 上传到 COS 的 todo/, 供常驻 worker 认领。
|
||||
|
||||
用法:
|
||||
python tools/seed_jobs.py 清单1.txt [清单2.txt ...]
|
||||
python tools/seed_jobs.py ./text2 # 目录: 取其中所有 *.txt
|
||||
python tools/seed_jobs.py ./text2 --skip-processed # 灌之前先过滤掉全局已处理过的视频
|
||||
分片大小取 config.dispatch.shard_size(默认2000)。分片名: shard_<起始序号>。
|
||||
幂等性: 同名分片已存在(todo 里有)则跳过, 不覆盖。
|
||||
|
||||
--skip-processed: 聚合 30 台机器各自最新一天的 数据.jsonl 快照(dispatch.daily_backup_path),
|
||||
合并成全局已处理视频集合再过滤清单 —— 避免同一视频被不同机器重复处理(本机 数据.jsonl 只能
|
||||
挡住"这台机器"重复跑, 挡不住跨机器重复)。有最长约1天(daily_backup_hour)的数据延迟。
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from config.config_utils import config # noqa: E402
|
||||
from tool import job_cos # noqa: E402
|
||||
|
||||
|
||||
def _collect_lines(paths):
|
||||
lines = []
|
||||
for p in paths:
|
||||
files = glob.glob(os.path.join(p, "*.txt")) if os.path.isdir(p) else [p]
|
||||
for fp in files:
|
||||
with open(fp, encoding="utf-8") as f:
|
||||
lines += [ln.strip() for ln in f if ln.strip()]
|
||||
# 去重并保序
|
||||
seen, out = set(), []
|
||||
for ln in lines:
|
||||
if ln not in seen:
|
||||
seen.add(ln)
|
||||
out.append(ln)
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description="切片灌任务到 COS todo/")
|
||||
ap.add_argument("paths", nargs="+", help="清单.txt 或目录(取其中所有 *.txt)")
|
||||
ap.add_argument("--skip-processed", action="store_true",
|
||||
help="灌之前先过滤掉全局(所有机器)已处理过的视频，见文件头说明")
|
||||
ap.add_argument("--all-dates", action="store_true",
|
||||
help="配合 --skip-processed: 不只取每台机器最新一天,把所有历史日期的已处理清单都拉下来合并判断"
|
||||
"(机群已停机、想更保险地核对时用;正常灌任务不需要,只会多下载很多文件)")
|
||||
args = ap.parse_args()
|
||||
|
||||
shard_size = int((config.get_dispatch_config() or {}).get("shard_size", 2000))
|
||||
lines = _collect_lines(args.paths)
|
||||
if not lines:
|
||||
print("✗ 没有读到任何视频路径")
|
||||
sys.exit(1)
|
||||
|
||||
if args.skip_processed:
|
||||
if args.all_dates:
|
||||
print("正在聚合所有机器【全部历史日期】的已处理清单(daily_backup_path)...")
|
||||
else:
|
||||
print("正在聚合各机器最新一天的已处理清单(daily_backup_path)...")
|
||||
processed = job_cos.global_processed_set(verbose=True, all_dates=args.all_dates)
|
||||
before = len(lines)
|
||||
lines = [ln for ln in lines if ln not in processed]
|
||||
print(f"全局已处理 {len(processed)} 条 -> 过滤掉 {before - len(lines)} 条，剩余 {len(lines)} 条待灌。")
|
||||
if not lines:
|
||||
print("✓ 全部已处理过，无需灌任务。")
|
||||
return
|
||||
|
||||
existing = set(job_cos.list_todo())
|
||||
n_shards = (len(lines) + shard_size - 1) // shard_size
|
||||
print(f"共 {len(lines)} 条 -> {n_shards} 个分片(每片≤{shard_size})。已存在 {len(existing)} 个分片。")
|
||||
|
||||
added = skipped = 0
|
||||
for i in range(0, len(lines), shard_size):
|
||||
shard = f"shard_{i:08d}"
|
||||
if shard in existing:
|
||||
skipped += 1
|
||||
continue
|
||||
body = "\n".join(lines[i:i + shard_size]).encode("utf-8")
|
||||
job_cos._put(job_cos._key("todo", shard), body) # 普通写(任务清单, 不需禁止覆盖)
|
||||
added += 1
|
||||
if added % 50 == 0:
|
||||
print(f" 已上传 {added} 个分片...")
|
||||
print(f"✓ 完成: 新增 {added} 个分片, 跳过已存在 {skipped} 个。worker 会自动认领。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
30
tools/split_txt.py
Normal file
30
tools/split_txt.py
Normal file
@@ -0,0 +1,30 @@
|
||||
"""
|
||||
将 txt 文件按指定行数拆分为多个子文件。
|
||||
用法：python split_txt.py [输入文件] [每份行数]
|
||||
默认：python split_txt.py text/0004_has_srt.txt 100
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
def split_txt(input_path: str, chunk_size: int = 100):
|
||||
with open(input_path, 'r', encoding='utf-8') as f:
|
||||
lines = [line.strip() for line in f if line.strip()]
|
||||
|
||||
total = len(lines)
|
||||
num_files = (total + chunk_size - 1) // chunk_size
|
||||
out_dir = os.path.dirname(input_path)
|
||||
base_name = os.path.splitext(os.path.basename(input_path))[0]
|
||||
|
||||
for i in range(num_files):
|
||||
chunk = lines[i * chunk_size:(i + 1) * chunk_size]
|
||||
out_path = os.path.join(out_dir, f"{base_name}_{i + 1:04d}.txt")
|
||||
with open(out_path, 'w', encoding='utf-8') as f:
|
||||
f.write('\n'.join(chunk) + '\n')
|
||||
print(f" {out_path} ({len(chunk)} 条)")
|
||||
|
||||
print(f"\n共 {total} 条，拆分为 {num_files} 个文件（每份 {chunk_size} 条）")
|
||||
|
||||
if __name__ == '__main__':
|
||||
input_file = sys.argv[1] if len(sys.argv) > 1 else 'text/0004_has_srt.txt'
|
||||
chunk_size = int(sys.argv[2]) if len(sys.argv) > 2 else 100
|
||||
split_txt(input_file, chunk_size)
|
||||
73
tools/stats_jsonl.py
Normal file
73
tools/stats_jsonl.py
Normal file
@@ -0,0 +1,73 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
统计 数据.jsonl：成功条数、总条数、成功数据总秒数、全部数据总秒数。
|
||||
用法: python stats_jsonl.py [数据.jsonl路径]
|
||||
默认: python stats_jsonl.py 数据.jsonl
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
|
||||
JSONL_PATH = sys.argv[1] if len(sys.argv) > 1 else "数据.jsonl"
|
||||
|
||||
|
||||
def fmt(seconds):
|
||||
"""秒 → H时M分S秒，便于阅读"""
|
||||
seconds = int(round(seconds))
|
||||
h, rem = divmod(seconds, 3600)
|
||||
m, s = divmod(rem, 60)
|
||||
return f"{h}时{m}分{s}秒"
|
||||
|
||||
|
||||
def main():
|
||||
total = 0
|
||||
success = 0
|
||||
total_sec = 0.0
|
||||
success_sec = 0.0
|
||||
status_counter = Counter()
|
||||
|
||||
try:
|
||||
f = open(JSONL_PATH, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {JSONL_PATH}")
|
||||
return
|
||||
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
|
||||
total += 1
|
||||
status = rec.get("status")
|
||||
status_counter[status] += 1
|
||||
|
||||
dur = rec.get("duration") or 0
|
||||
try:
|
||||
dur = float(dur)
|
||||
except (TypeError, ValueError):
|
||||
dur = 0.0
|
||||
total_sec += dur
|
||||
|
||||
if status == "success":
|
||||
success += 1
|
||||
success_sec += dur
|
||||
|
||||
rate = (success / total * 100) if total else 0.0
|
||||
|
||||
print(f"文件: {JSONL_PATH}")
|
||||
print("-" * 40)
|
||||
print(f"总条数: {total}")
|
||||
print(f"成功条数: {success} ({rate:.1f}%)")
|
||||
print(f"各状态明细: {dict(status_counter)}")
|
||||
print("-" * 40)
|
||||
print(f"成功数据秒数: {success_sec:.2f} 秒 ({fmt(success_sec)})")
|
||||
print(f"全部数据秒数: {total_sec:.2f} 秒 ({fmt(total_sec)})")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
117
tools/test_ocr.py
Normal file
117
tools/test_ocr.py
Normal file
@@ -0,0 +1,117 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
向 OCR 服务发一次请求做 OCR，验证端点可用、看识别结果。
|
||||
|
||||
用法:
|
||||
python test_ocr.py <图片路径>
|
||||
python test_ocr.py <图片路径> --url http://其它IP:18081
|
||||
python test_ocr.py <图片路径> --raw # 额外打印原始 JSON 响应
|
||||
|
||||
不给图片时，自动在 ./composite、./split 下找一张 .jpg 试。
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import time
|
||||
|
||||
import requests
|
||||
|
||||
DEFAULT_URL = "http://asr.ocr.general.smart.l20.production.polaris:18081"
|
||||
OCR_PATH = "/youtu/ocrapi/hpgeneralocr_v3"
|
||||
|
||||
|
||||
def find_sample_image(given):
|
||||
if given:
|
||||
return given if os.path.exists(given) else None
|
||||
for root in ("./composite", "./split"):
|
||||
hits = glob.glob(os.path.join(root, "**", "*.jpg"), recursive=True)
|
||||
if hits:
|
||||
return hits[0]
|
||||
return None
|
||||
|
||||
|
||||
def img2base64(img_path):
|
||||
with open(img_path, "rb") as f:
|
||||
return base64.b64encode(f.read()).decode()
|
||||
|
||||
|
||||
def ocr_request(base_url, image_b64, timeout=30):
|
||||
"""向端点发一次 OCR 请求，返回 (原始响应文本, 耗时秒)。"""
|
||||
url = base_url.rstrip("/") + OCR_PATH
|
||||
app_id = f"{int(time.time() * 1000)}{random.randint(100000, 999999)}"
|
||||
payload = {
|
||||
"app_id": app_id,
|
||||
"image": image_b64,
|
||||
"ocr_template": "OCR",
|
||||
"options": {"pdf_page_index": 0, "detect_text": True},
|
||||
}
|
||||
t0 = time.time()
|
||||
resp = requests.post(url, headers={"Content-Type": "application/json"},
|
||||
json=payload, timeout=timeout)
|
||||
dt = time.time() - t0
|
||||
resp.raise_for_status()
|
||||
return resp.text, dt
|
||||
|
||||
|
||||
def extract_texts(raw):
|
||||
"""从 Youtu OCR 响应里抽出每行识别文字 (content, conf)。"""
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except (json.JSONDecodeError, TypeError):
|
||||
return None
|
||||
if data.get("error_code", 0) != 0:
|
||||
return f"OCR 返回错误: error_code={data.get('error_code')}, msg={data.get('error_msg')}"
|
||||
lines = []
|
||||
for rec in data.get("recognize_list", []):
|
||||
for item in rec.get("item_content", {}).get("item_list", []):
|
||||
lines.append((item.get("content", ""), item.get("conf", 0)))
|
||||
return lines
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("image", nargs="?", default=None, help="图片路径")
|
||||
ap.add_argument("--url", default=DEFAULT_URL, help="OCR 端点 (默认 Polaris 地址)")
|
||||
ap.add_argument("--raw", action="store_true", help="打印原始 JSON 响应")
|
||||
args = ap.parse_args()
|
||||
|
||||
img = find_sample_image(args.image)
|
||||
if not img:
|
||||
print("找不到图片，请把一张 jpg 路径作为参数传入。")
|
||||
return
|
||||
|
||||
print(f"端点: {args.url}{OCR_PATH}")
|
||||
print(f"图片: {img}")
|
||||
print("-" * 60)
|
||||
|
||||
try:
|
||||
raw, dt = ocr_request(args.url, img2base64(img))
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f"✗ 请求失败: {e}")
|
||||
return
|
||||
|
||||
print(f"✓ 请求成功，耗时 {dt * 1000:.0f}ms")
|
||||
|
||||
if args.raw:
|
||||
print("\n原始响应:")
|
||||
print(raw)
|
||||
print("-" * 60)
|
||||
|
||||
result = extract_texts(raw)
|
||||
if result is None:
|
||||
print("响应不是合法 JSON")
|
||||
elif isinstance(result, str):
|
||||
print(result)
|
||||
elif not result:
|
||||
print("（未识别到文字）")
|
||||
else:
|
||||
print(f"\n识别到 {len(result)} 行文字:")
|
||||
for content, conf in result:
|
||||
print(f" [{conf:.2f}] {content}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
159
tools/throughput_jsonl.py
Normal file
159
tools/throughput_jsonl.py
Normal file
@@ -0,0 +1,159 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
分析 数据.jsonl 的处理吞吐：单台机器一天能处理多少小时的视频数据。
|
||||
|
||||
两种口径：
|
||||
A) 墙钟吞吐（真实）：用 process_start/process_end 求整批的实际墙钟跨度，
|
||||
总视频时长 ÷ 墙钟跨度 = 机器真实吞吐（含当时的并发/排队效果）。
|
||||
B) 单视频平均（理想并发）：按每个视频 duration/elapsed 的实时倍率，
|
||||
再乘并发数估算（假设无 OCR/CPU 排队，即"OCR 速度够"的上限）。
|
||||
|
||||
用法: python throughput_jsonl.py [数据.jsonl路径] [并发数]
|
||||
默认: python throughput_jsonl.py 数据.jsonl 1
|
||||
"""
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime
|
||||
|
||||
JSONL_PATH = sys.argv[1] if len(sys.argv) > 1 else "数据.jsonl"
|
||||
WORKERS = int(sys.argv[2]) if len(sys.argv) > 2 else 1
|
||||
OCR_QPS_PER_MACHINE = float(sys.argv[3]) if len(sys.argv) > 3 else 14.0 # 单台 OCR 的 QPS
|
||||
|
||||
|
||||
def parse_dt(s):
|
||||
if not s:
|
||||
return None
|
||||
try:
|
||||
return datetime.fromisoformat(s)
|
||||
except (ValueError, TypeError):
|
||||
return None
|
||||
|
||||
|
||||
def fmt_h(hours):
|
||||
return f"{hours:.1f} 小时"
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
f = open(JSONL_PATH, encoding="utf-8")
|
||||
except FileNotFoundError:
|
||||
print(f"找不到文件: {JSONL_PATH}")
|
||||
return
|
||||
|
||||
n = 0
|
||||
total_dur = 0.0 # 全部视频时长(秒)
|
||||
total_elapsed = 0.0 # 全部单视频处理耗时(秒)，含并发
|
||||
starts, ends = [], []
|
||||
per_video_ratio = [] # 每视频 duration/elapsed
|
||||
stage_sum = {} # 各阶段耗时累计
|
||||
with_timing = 0
|
||||
total_ocr = 0 # 全部 OCR 请求次数累计
|
||||
with_ocr = 0
|
||||
|
||||
with f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
n += 1
|
||||
dur = rec.get("duration") or 0
|
||||
try:
|
||||
dur = float(dur)
|
||||
except (TypeError, ValueError):
|
||||
dur = 0.0
|
||||
total_dur += dur
|
||||
|
||||
el = rec.get("elapsed_seconds")
|
||||
if el:
|
||||
total_elapsed += float(el)
|
||||
if dur > 0:
|
||||
per_video_ratio.append(dur / float(el))
|
||||
|
||||
st = parse_dt(rec.get("process_start"))
|
||||
en = parse_dt(rec.get("process_end"))
|
||||
if st:
|
||||
starts.append(st)
|
||||
if en:
|
||||
ends.append(en)
|
||||
|
||||
sd = rec.get("stage_durations") or {}
|
||||
if sd:
|
||||
with_timing += 1
|
||||
for k, v in sd.items():
|
||||
try:
|
||||
stage_sum[k] = stage_sum.get(k, 0.0) + float(v)
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
ic = rec.get("image_counts") or {}
|
||||
toc = ic.get("total_ocr_count")
|
||||
if toc:
|
||||
try:
|
||||
total_ocr += int(toc)
|
||||
with_ocr += 1
|
||||
except (TypeError, ValueError):
|
||||
pass
|
||||
|
||||
print(f"文件: {JSONL_PATH}")
|
||||
print("=" * 52)
|
||||
print(f"记录数: {n}")
|
||||
print(f"视频总时长: {total_dur/3600:.1f} 小时 ({total_dur:.0f} 秒)")
|
||||
print(f"单视频耗时累加: {total_elapsed/3600:.1f} 小时 ({total_elapsed:.0f} 秒)")
|
||||
if with_ocr:
|
||||
print(f"OCR 请求总数: {total_ocr} (平均 {total_ocr/with_ocr:.0f} 次/视频, {with_ocr} 条有统计)")
|
||||
|
||||
# ---- 口径A：真实墙钟吞吐 ----
|
||||
print("\n[A] 真实墙钟吞吐（按 process_start/end 的整批实际跨度）")
|
||||
if starts and ends:
|
||||
span = (max(ends) - min(starts)).total_seconds()
|
||||
if span > 0:
|
||||
tp = total_dur / span # 视频秒/墙钟秒
|
||||
print(f" 整批墙钟跨度: {span/3600:.2f} 小时")
|
||||
print(f" 实时倍率: {tp:.2f}x （每 1 秒墙钟处理 {tp:.2f} 秒视频）")
|
||||
print(f" → 一天可处理: {fmt_h(tp*24)} 的视频")
|
||||
vids_per_day = n / span * 86400 if span else 0
|
||||
print(f" → 一天可处理: {vids_per_day:.0f} 个视频")
|
||||
|
||||
# ---- OCR 需求：这台机器实际产生多少 OCR/秒、能喂几台 OCR ----
|
||||
if total_ocr:
|
||||
ocr_per_sec = total_ocr / span
|
||||
machines_fed = ocr_per_sec / OCR_QPS_PER_MACHINE
|
||||
print(f"\n [OCR 配比]（单台 OCR 按 {OCR_QPS_PER_MACHINE:.0f} QPS 计）")
|
||||
print(f" 本机 OCR 速率: {ocr_per_sec:.2f} OCR/秒")
|
||||
print(f" → 本机能喂满: {machines_fed:.2f} 台 OCR")
|
||||
if machines_fed > 0:
|
||||
print(f" → 一台 OCR 可喂: {1/machines_fed:.1f} 台这样的处理机")
|
||||
else:
|
||||
print(" 墙钟跨度为0，无法计算")
|
||||
else:
|
||||
print(" 缺少 process_start/process_end 字段，跳过")
|
||||
|
||||
# ---- 口径B：按单视频平均实时倍率 × 并发 ----
|
||||
print(f"\n[B] 理想吞吐（单视频平均倍率 × 并发{WORKERS}，假设无排队=OCR够快）")
|
||||
if per_video_ratio:
|
||||
avg_ratio = sum(per_video_ratio) / len(per_video_ratio)
|
||||
# 也给一个总量口径：总视频时长/总耗时（更稳，受长短视频加权）
|
||||
weighted = total_dur / total_elapsed if total_elapsed else 0
|
||||
print(f" 单视频平均倍率: {avg_ratio:.2f}x")
|
||||
print(f" 加权倍率(总/总):{weighted:.2f}x")
|
||||
print(f" → 单并发一天: {fmt_h(weighted*24)} 的视频")
|
||||
print(f" → {WORKERS}并发一天: {fmt_h(weighted*24*WORKERS)} 的视频（理想线性，实际受CPU限制）")
|
||||
else:
|
||||
print(" 缺少 elapsed_seconds 字段，跳过")
|
||||
|
||||
# ---- 各阶段占比 ----
|
||||
if stage_sum:
|
||||
print(f"\n[阶段耗时占比]（{with_timing} 条有计时）")
|
||||
main_t = stage_sum.get("main", sum(v for k, v in stage_sum.items() if k != "main"))
|
||||
for k in ["split_frame", "merge", "getsrt"]:
|
||||
if k in stage_sum:
|
||||
pct = stage_sum[k] / main_t * 100 if main_t else 0
|
||||
print(f" {k:12s}: {stage_sum[k]/3600:6.1f} 小时 ({pct:.0f}%)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
91
tools/upload_composites.py
Normal file
91
tools/upload_composites.py
Normal file
@@ -0,0 +1,91 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
扫描本地 ./composite/ 下的视频目录，逐个上传到其在 COS 上对应视频目录的 composite/ 子目录，
|
||||
上传成功后删除本地。视频的 COS 路径从 数据.jsonl 的 video_cos_path 反查。
|
||||
|
||||
用法: python upload_composites.py [composite根目录] [数据.jsonl路径]
|
||||
默认: python upload_composites.py ./composite 数据.jsonl
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
from config.config_utils import config
|
||||
|
||||
COMPOSITE_ROOT = sys.argv[1] if len(sys.argv) > 1 else "./composite"
|
||||
JSONL_PATH = sys.argv[2] if len(sys.argv) > 2 else "数据.jsonl"
|
||||
|
||||
|
||||
def build_cosdir_map(jsonl_path):
|
||||
"""从 数据.jsonl 建立 video_name -> cos_dir 映射。"""
|
||||
mapping = {}
|
||||
if not os.path.exists(jsonl_path):
|
||||
print(f"警告: 找不到 {jsonl_path}，无法反查 COS 路径")
|
||||
return mapping
|
||||
with open(jsonl_path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
vid = rec.get("video_cos_path")
|
||||
if not vid:
|
||||
continue
|
||||
cos_dir = vid[:vid.rfind("/") + 1] # 含末尾 /
|
||||
name = os.path.splitext(os.path.basename(vid))[0]
|
||||
mapping[name] = cos_dir
|
||||
return mapping
|
||||
|
||||
|
||||
def upload_dir(local_dir, cos_dest_dir, coscli_path, cos_config_path):
|
||||
if not cos_dest_dir.endswith("/"):
|
||||
cos_dest_dir += "/"
|
||||
src = local_dir if local_dir.endswith(os.sep) else local_dir + os.sep
|
||||
cmd = [coscli_path, "-c", cos_config_path, "cp", "-r", src, cos_dest_dir]
|
||||
result = subprocess.run(cmd, capture_output=True, text=True, check=False)
|
||||
return (cos_dest_dir if result.returncode == 0 else None), result.stderr
|
||||
|
||||
|
||||
def main():
|
||||
if not os.path.isdir(COMPOSITE_ROOT):
|
||||
print(f"目录不存在: {COMPOSITE_ROOT}")
|
||||
return
|
||||
|
||||
cfg = config.get_splitframe_config()
|
||||
coscli_path = cfg.get("coscli_path", "./coscli-linux-amd64")
|
||||
cos_config_path = cfg.get("cosconfig_path", ".cos.yaml")
|
||||
|
||||
cosdir_map = build_cosdir_map(JSONL_PATH)
|
||||
|
||||
dirs = [d for d in os.listdir(COMPOSITE_ROOT)
|
||||
if os.path.isdir(os.path.join(COMPOSITE_ROOT, d))]
|
||||
print(f"发现 {len(dirs)} 个本地 composite 目录，开始上传...")
|
||||
|
||||
ok = fail = skip = 0
|
||||
for i, name in enumerate(dirs, 1):
|
||||
local_dir = os.path.join(COMPOSITE_ROOT, name)
|
||||
cos_dir = cosdir_map.get(name)
|
||||
if not cos_dir:
|
||||
skip += 1
|
||||
print(f"[{i}/{len(dirs)}] {name} ✗ 在 数据.jsonl 中找不到 COS 路径，跳过")
|
||||
continue
|
||||
cos_dest = cos_dir + "composite/"
|
||||
uploaded, err = upload_dir(local_dir, cos_dest, coscli_path, cos_config_path)
|
||||
if uploaded:
|
||||
shutil.rmtree(local_dir, ignore_errors=True)
|
||||
ok += 1
|
||||
print(f"[{i}/{len(dirs)}] {name} ✓ → {cos_dest}（已删本地）")
|
||||
else:
|
||||
fail += 1
|
||||
print(f"[{i}/{len(dirs)}] {name} ✗ 上传失败，保留本地: {err.strip()[:120]}")
|
||||
|
||||
print(f"\n完成 — 成功: {ok} 失败: {fail} 跳过(无COS路径): {skip}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
35
tools/upload_daily.py
Normal file
35
tools/upload_daily.py
Normal file
@@ -0,0 +1,35 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""手动/定时把本地 数据.jsonl 备份到 COS(文件名带机器号+日期)。
|
||||
|
||||
常驻 worker(run_worker.py)内部已有每天10点自动备份线程; 本脚本用于:
|
||||
- 不跑常驻 worker、想用系统 crontab 定时备份的场景;
|
||||
- 想立刻手动备份一次。
|
||||
|
||||
crontab 例(每天10点):
|
||||
0 10 * * * cd /项目根 && python3 tools/upload_daily.py >> log/upload_daily.log 2>&1
|
||||
"""
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from tool import job_cos # noqa: E402
|
||||
|
||||
JSONL = "数据.jsonl"
|
||||
|
||||
|
||||
def main():
|
||||
mid = job_cos.get_machine_id()
|
||||
dst = job_cos.upload_daily_backup(JSONL, mid)
|
||||
if dst:
|
||||
print(f"✓ 数据.jsonl 已上传 -> {dst}")
|
||||
else:
|
||||
print(f"✗ 本地无 {JSONL}, 未上传")
|
||||
ll = job_cos.upload_longlist_backup(mid)
|
||||
if ll:
|
||||
print(f"✓ 超长待重跑列表已上传 -> {ll}")
|
||||
else:
|
||||
print("✗ 本地无超长待重跑列表, 未上传")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
207
tools/upload_data_standalone.py
Normal file
207
tools/upload_data_standalone.py
Normal file
@@ -0,0 +1,207 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""独立备份脚本: 把本地 数据.jsonl + 超长待重跑列表 上传到 COS。
|
||||
|
||||
不 import 项目里任何其他 .py(不依赖 tool/、main.py 等), 只用标准库 + 第三方 qcloud_cos SDK,
|
||||
直接读 config/config.json —— 哪怕代码刚更新完、其他模块有问题, 这个脚本也能独立跑, 适合放在
|
||||
重新部署/更新代码前后单独执行一次, 确保数据先落到 COS。
|
||||
|
||||
用法(项目根目录下运行):
|
||||
python tools/upload_data_standalone.py
|
||||
|
||||
行为跟 tools/upload_daily.py + tool/job_cos.py 里的 upload_daily_backup/upload_longlist_backup
|
||||
一致(上传目标、命名、覆盖安全性都对齐):
|
||||
数据.jsonl -> dispatch.daily_backup_path/<日期>/数据_<机器号>.jsonl (全量快照,覆盖安全)
|
||||
已处理清单(轻量) -> dispatch.daily_backup_path/<日期>/已处理_<机器号>.txt
|
||||
(从jsonl提取的纯 video_cos_path 列表, 一行一个; 全局去重 global_processed_set()
|
||||
只拉这个小文件, 不用为了去重下载/解析整份带状态耗时字段的jsonl)
|
||||
超长待重跑列表 -> dispatch.long_list_backup_path/todo/<原文件名>_<机器号>_<日期>_<时分秒>.txt
|
||||
(增量式:传完清空本地已传部分; key精确到秒,同一天多次调用不会互相覆盖)
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import socket
|
||||
import subprocess
|
||||
from datetime import datetime
|
||||
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
|
||||
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||
CONFIG_PATH = os.path.join(ROOT, "config", "config.json")
|
||||
JSONL_PATH = os.path.join(ROOT, "数据.jsonl")
|
||||
|
||||
|
||||
def load_config() -> dict:
|
||||
with open(CONFIG_PATH, "r", encoding="utf-8") as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def parse_cos_url(url: str, default_bucket: str):
|
||||
"""cos://bucket/a/b -> (bucket, 'a/b')。无 cos:// 前缀时按传入的默认桶补全。"""
|
||||
url = url.strip()
|
||||
if url.startswith("cos://"):
|
||||
rest = url[len("cos://"):]
|
||||
bucket, _, key = rest.partition("/")
|
||||
return bucket, key.strip("/")
|
||||
return default_bucket, url.strip("/")
|
||||
|
||||
|
||||
def detect_ip() -> str:
|
||||
"""取本机对外默认路由的网卡IP(UDP connect触发内核选路由,不实际发包,天然跳过docker网桥/回环);
|
||||
取不到退化到 `hostname -I` 第一个。跟 tool/job_cos.py 的 _detect_ip() 逻辑一致。"""
|
||||
try:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
|
||||
try:
|
||||
s.connect(("8.8.8.8", 80))
|
||||
ip = s.getsockname()[0]
|
||||
finally:
|
||||
s.close()
|
||||
if ip and not ip.startswith("127."):
|
||||
return ip
|
||||
except OSError:
|
||||
pass
|
||||
try:
|
||||
r = subprocess.run(["hostname", "-I"], capture_output=True, text=True, timeout=5)
|
||||
ips = (r.stdout or "").split()
|
||||
return ips[0] if ips else ""
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return ""
|
||||
|
||||
|
||||
def get_machine_id(dcfg: dict) -> str:
|
||||
"""优先级: config.dispatch.machine_id > 环境变量 MACHINE_ID > 本机固定IP > 主机名。
|
||||
跟 tool/job_cos.py 的 get_machine_id() 保持一致, 这样跟常驻 worker 认领的是同一个身份。"""
|
||||
mid = (dcfg.get("machine_id") or "").strip()
|
||||
if mid:
|
||||
return mid
|
||||
mid = (os.environ.get("MACHINE_ID") or "").strip()
|
||||
if mid:
|
||||
return mid
|
||||
ip = detect_ip()
|
||||
if ip:
|
||||
return ip
|
||||
return socket.gethostname()
|
||||
|
||||
|
||||
# 数据.jsonl 是全量快照, 产能高的机器上百 MB/上 GB。put_object 单请求默认超时 30 秒, 大到
|
||||
# 传不完就一直 "The write operation timed out"。超过这个大小改用 coscli 分块上传(几 GB 也行),
|
||||
# coscli 失败自动回退 put_object。跟 tool/job_cos.py 的 _upload_dated_file 是同一套策略。
|
||||
_COSCLI_UPLOAD_THRESHOLD = 50 * 1024 * 1024 # 50MB
|
||||
|
||||
|
||||
def _coscli_put(sf_cfg: dict, local_path: str, cos_url: str) -> bool:
|
||||
"""用 coscli 上传大文件。成功 True 失败 False。sf_cfg = config 的 splitframe 段。"""
|
||||
try:
|
||||
coscli = sf_cfg.get("coscli_path", "./coscli-linux-amd64")
|
||||
coscfg = sf_cfg.get("cosconfig_path", ".cos.yaml")
|
||||
r = subprocess.run([coscli, "-c", coscfg, "cp", local_path, cos_url],
|
||||
capture_output=True, text=True, timeout=1800)
|
||||
if r.returncode == 0:
|
||||
return True
|
||||
print(f" coscli 上传失败(回退 put_object): {(r.stderr or '').strip()[:200]}")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f" coscli 上传异常(回退 put_object): {type(e).__name__}: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def upload_jsonl(client, default_bucket: str, dcfg: dict, sf_cfg: dict, machine_id: str):
|
||||
daily_root = dcfg.get("daily_backup_path")
|
||||
if not daily_root:
|
||||
print("✗ 未配置 dispatch.daily_backup_path, 跳过 数据.jsonl")
|
||||
return
|
||||
if not os.path.exists(JSONL_PATH):
|
||||
print(f"✗ 本地无 {JSONL_PATH}, 跳过")
|
||||
return
|
||||
bkt, base = parse_cos_url(daily_root, default_bucket)
|
||||
date = datetime.now().strftime("%Y%m%d")
|
||||
key = f"{base}/{date}/数据_{machine_id}.jsonl"
|
||||
cos_url = f"cos://{bkt}/{key}"
|
||||
with open(JSONL_PATH, "rb") as f:
|
||||
data = f.read()
|
||||
|
||||
size = len(data)
|
||||
uploaded = False
|
||||
if size > _COSCLI_UPLOAD_THRESHOLD:
|
||||
print(f" 数据.jsonl 有 {size / 1024 / 1024:.0f}MB, 用 coscli 分块上传")
|
||||
uploaded = _coscli_put(sf_cfg, JSONL_PATH, cos_url)
|
||||
if not uploaded:
|
||||
client.put_object(Bucket=bkt, Body=data, Key=key)
|
||||
print(f"✓ 数据.jsonl 已上传 -> {cos_url}")
|
||||
|
||||
# 顺带生成一份轻量"已处理清单"(只有 video_cos_path, 一行一个), 供 global_processed_set()
|
||||
# 只拉这个小文件去重用, 不用为了去重下载/解析整份带状态/耗时字段的 jsonl。
|
||||
paths = []
|
||||
for line in data.decode("utf-8", errors="ignore").splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
vid = json.loads(line).get("video_cos_path")
|
||||
if vid:
|
||||
paths.append(vid)
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
if paths:
|
||||
list_key = f"{base}/{date}/已处理_{machine_id}.txt"
|
||||
client.put_object(Bucket=bkt, Body=("\n".join(paths) + "\n").encode("utf-8"), Key=list_key)
|
||||
print(f"✓ 已处理清单已上传 -> cos://{bkt}/{list_key} ({len(paths)} 条)")
|
||||
|
||||
|
||||
def upload_long_list(client, default_bucket: str, dcfg: dict, sub_cfg: dict, machine_id: str):
|
||||
if dcfg.get("run_long_video"):
|
||||
print("本机 run_long_video=true(专职长视频机), 不回传超长列表, 跳过")
|
||||
return
|
||||
long_root = dcfg.get("long_list_backup_path")
|
||||
if not long_root:
|
||||
print("✗ 未配置 dispatch.long_list_backup_path, 跳过超长待重跑列表")
|
||||
return
|
||||
local_long = sub_cfg.get("long_video_list", "./超长待重跑.txt")
|
||||
if not os.path.isabs(local_long):
|
||||
local_long = os.path.join(ROOT, local_long)
|
||||
if not os.path.exists(local_long):
|
||||
print(f"✗ 本地无 {local_long}, 跳过")
|
||||
return
|
||||
with open(local_long, "r", encoding="utf-8") as f:
|
||||
snapshot = f.readlines()
|
||||
if not any(ln.strip() for ln in snapshot):
|
||||
print("✗ 超长待重跑列表为空, 跳过")
|
||||
return
|
||||
|
||||
bkt, base = parse_cos_url(long_root, default_bucket)
|
||||
stem = os.path.splitext(os.path.basename(local_long))[0]
|
||||
ts = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
key = f"{base}/todo/{stem}_{machine_id}_{ts}.txt"
|
||||
client.put_object(Bucket=bkt, Body="".join(snapshot).encode("utf-8"), Key=key)
|
||||
print(f"✓ 超长待重跑列表已上传 -> cos://{bkt}/{key}")
|
||||
|
||||
if dcfg.get("long_list_clear_after_upload", True):
|
||||
try:
|
||||
with open(local_long, "r", encoding="utf-8") as f:
|
||||
cur = f.readlines()
|
||||
with open(local_long, "w", encoding="utf-8") as f:
|
||||
f.writelines(cur[len(snapshot):])
|
||||
print(f" 本地已清空已传的 {len(snapshot)} 行, 保留上传期间新增的 {len(cur) - len(snapshot)} 行")
|
||||
except OSError as e:
|
||||
print(f" ⚠ 清空本地文件失败(不影响已上传的部分): {e}")
|
||||
|
||||
|
||||
def main():
|
||||
cfg = load_config()
|
||||
cos_cfg = cfg.get("tencentcloud", {}).get("cos", {})
|
||||
dcfg = cfg.get("dispatch", {}) or {}
|
||||
sub_cfg = cfg.get("subtitle", {}) or {}
|
||||
sf_cfg = cfg.get("splitframe", {}) or {} # coscli 路径/配置在这段
|
||||
|
||||
client = CosS3Client(CosConfig(
|
||||
Region=cos_cfg["region"], SecretId=cos_cfg["secret_id"], SecretKey=cos_cfg["secret_key"],
|
||||
))
|
||||
default_bucket = cos_cfg["bucket"]
|
||||
machine_id = get_machine_id(dcfg)
|
||||
print(f"机器ID: {machine_id}")
|
||||
|
||||
upload_jsonl(client, default_bucket, dcfg, sf_cfg, machine_id)
|
||||
upload_long_list(client, default_bucket, dcfg, sub_cfg, machine_id)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
150
tools/video_bench.py
Normal file
150
tools/video_bench.py
Normal file
@@ -0,0 +1,150 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
下载一个视频 → 抽帧 → 用这些【真实帧】对 OCR 端点做压测。
|
||||
比随机生成的图更贴近线上负载（真实字幕内容，OCR 干的活和实际一致）。
|
||||
|
||||
用法（项目根目录运行）:
|
||||
python temp/video_bench.py <视频...> --url http://IP:18081 [选项]
|
||||
|
||||
<视频...> 一个或多个，cos://bucket/path/x.mp4 或本地 ./video/x.mp4
|
||||
多个视频时各抽一部分帧合并成图池(max-frames 在视频间均分)
|
||||
也可用 --list 文件 从文件读视频列表(每行一个)
|
||||
|
||||
选项:
|
||||
--url URL OCR 端点(可多次，soak 只用第一个)
|
||||
--concurrency N 并发(默认 100；单台拐点约 80)
|
||||
--mode soak|sweep soak=持续满速(默认)；sweep=自动加压找极限
|
||||
--fps F 抽帧频率(默认 1，压测用不需要 3fps，省时省内存)
|
||||
--max-frames N 最多用多少帧做图池(默认 2000)
|
||||
--interval N soak 打印间隔秒(默认 5)
|
||||
--ffmpeg PATH ffmpeg 路径(默认 ./bin/ffmpeg)
|
||||
--coscli PATH coscli 路径(默认 ./coscli-linux-amd64)
|
||||
--cos-config PATH coscli 配置(默认 .cos.yaml)
|
||||
--keep 跑完不删临时视频/帧
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
|
||||
# 复用同目录 bench 脚本的压测逻辑
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import bench_ocr_endpoints as bench # noqa: E402
|
||||
|
||||
|
||||
def download_video(src, coscli, cos_config, dst):
|
||||
"""cos:// 路径用 coscli 下载到 dst；本地路径直接返回。"""
|
||||
if not src.startswith("cos://"):
|
||||
if not os.path.exists(src):
|
||||
print(f"本地视频不存在: {src}")
|
||||
return None
|
||||
return src
|
||||
print(f"下载视频: {src}")
|
||||
r = subprocess.run([coscli, "-c", cos_config, "cp", src, dst],
|
||||
capture_output=True, text=True)
|
||||
if r.returncode != 0 or not os.path.exists(dst):
|
||||
print(f"下载失败: {r.stderr.strip()[:300]}")
|
||||
return None
|
||||
print(f" → {dst} ({os.path.getsize(dst)/1024/1024:.0f} MB)")
|
||||
return dst
|
||||
|
||||
|
||||
def extract_frames(ffmpeg, video, frame_dir, fps):
|
||||
"""用 ffmpeg 抽帧到 frame_dir，返回帧文件列表。"""
|
||||
os.makedirs(frame_dir, exist_ok=True)
|
||||
print(f"抽帧 (fps={fps}) ...")
|
||||
cmd = [ffmpeg, "-i", video, "-vf", f"fps={fps}", "-q:v", "2",
|
||||
os.path.join(frame_dir, "f_%06d.jpg"), "-loglevel", "error", "-y"]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
print(f"抽帧失败: {r.stderr.strip()[:400]}")
|
||||
return []
|
||||
frames = sorted(glob.glob(os.path.join(frame_dir, "*.jpg")))
|
||||
print(f" 抽出 {len(frames)} 帧")
|
||||
return frames
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("videos", nargs="*", default=[], help="一个或多个视频(cos://... 或本地路径)")
|
||||
ap.add_argument("--list", default=None, help="从文件读视频列表(每行一个，可代替直接传)")
|
||||
ap.add_argument("--url", action="append", default=None, help="OCR 端点(可多次)")
|
||||
ap.add_argument("--concurrency", type=int, default=100)
|
||||
ap.add_argument("--mode", choices=["soak", "sweep"], default="soak")
|
||||
ap.add_argument("--fps", type=float, default=1.0)
|
||||
ap.add_argument("--max-frames", type=int, default=2000)
|
||||
ap.add_argument("--interval", type=int, default=5)
|
||||
ap.add_argument("--ffmpeg", default="./bin/ffmpeg")
|
||||
ap.add_argument("--coscli", default="./coscli-linux-amd64")
|
||||
ap.add_argument("--cos-config", default=".cos.yaml")
|
||||
ap.add_argument("--keep", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
endpoints = bench.get_endpoints(args.url)
|
||||
if not endpoints:
|
||||
print("没有 OCR 端点，用 --url 指定")
|
||||
return
|
||||
|
||||
# 收集视频列表：命令行参数 + --list 文件
|
||||
videos = list(args.videos)
|
||||
if args.list and os.path.exists(args.list):
|
||||
with open(args.list, encoding="utf-8") as f:
|
||||
videos += [ln.strip() for ln in f if ln.strip()]
|
||||
if not videos:
|
||||
print("没有视频，传一个或多个视频路径，或用 --list 文件")
|
||||
return
|
||||
# 每个视频均分帧额度，避免一个长视频占满图池
|
||||
per_video_cap = max(1, args.max_frames // len(videos))
|
||||
|
||||
work = tempfile.mkdtemp(prefix="video_bench_")
|
||||
try:
|
||||
all_frames = []
|
||||
for i, src in enumerate(videos):
|
||||
print(f"\n[{i+1}/{len(videos)}] {src}")
|
||||
vlocal = os.path.join(work, f"video_{i}.mp4")
|
||||
fdir = os.path.join(work, f"frames_{i}")
|
||||
v = download_video(src, args.coscli, args.cos_config, vlocal)
|
||||
if not v:
|
||||
continue
|
||||
fr = extract_frames(args.ffmpeg, v, fdir, args.fps)
|
||||
if len(fr) > per_video_cap:
|
||||
fr = fr[:per_video_cap]
|
||||
print(f" 取前 {per_video_cap} 帧")
|
||||
all_frames += fr
|
||||
# 处理完即删该视频，省空间
|
||||
if v == vlocal:
|
||||
try:
|
||||
os.remove(vlocal)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
if not all_frames:
|
||||
print("没抽到任何帧")
|
||||
return
|
||||
|
||||
# 帧 → base64 图池
|
||||
print(f"\n加载 {len(all_frames)} 帧为图池 ...")
|
||||
pool = [bench.img2base64(f) for f in all_frames]
|
||||
print("=" * 60)
|
||||
print(f"图池: {len(pool)} 张真实帧(来自 {len(videos)} 个视频) | "
|
||||
f"端点: {endpoints[0]} | 并发: {args.concurrency}")
|
||||
print("=" * 60)
|
||||
|
||||
# 4) 压测
|
||||
if args.mode == "sweep":
|
||||
bench.sweep_endpoint(endpoints[0], pool[0], 10, 320, 6)
|
||||
else:
|
||||
bench.soak_endpoint(endpoints[0], pool, args.concurrency, args.interval)
|
||||
|
||||
finally:
|
||||
if args.keep:
|
||||
print(f"\n临时文件保留在: {work}")
|
||||
else:
|
||||
shutil.rmtree(work, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
128
tools/回溯过滤.py
Normal file
128
tools/回溯过滤.py
Normal file
@@ -0,0 +1,128 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
回溯过滤: 对【已跑出来】的成功视频, 用低置信占比信号找出"非字幕内容"(幻灯片/代码/满屏文字),
|
||||
和新加的实时过滤同一套标准。默认只列出(dry-run); --delete 才删 COS 上的垃圾 SRT。
|
||||
|
||||
用法:
|
||||
python3 tools/回溯过滤.py # 扫全部成功视频, 只列出
|
||||
python3 tools/回溯过滤.py --recent 200 # 只扫最近200个(快速试)
|
||||
python3 tools/回溯过滤.py --frac 0.35 --thr 80 # 阈值(同config默认)
|
||||
python3 tools/回溯过滤.py --delete # ★删掉判定为非字幕的 COS SRT★(不可逆!)
|
||||
产物: 非字幕清单.txt (video_cos_path, 低置信占比, srt路径)
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
from config.config_utils import config
|
||||
|
||||
cos = config.get_cos_config()
|
||||
B = cos["bucket"]
|
||||
_client = CosS3Client(CosConfig(Region=cos["region"], SecretId=cos["secret_id"],
|
||||
SecretKey=cos["secret_key"]))
|
||||
|
||||
|
||||
def key_of(cos_url):
|
||||
return cos_url.split("://", 1)[1].split("/", 1)[1]
|
||||
|
||||
|
||||
def lowconf_ratio(composite_cos_path, thr):
|
||||
"""下载该视频的 ocr_result.jsonl, 返回 (低置信占比, 检测数); 失败返回 (None, 0)。"""
|
||||
key = key_of(composite_cos_path).rstrip("/") + "/ocr_result.jsonl"
|
||||
try:
|
||||
body = _client.get_object(Bucket=B, Key=key)["Body"].get_raw_stream().read()
|
||||
confs = []
|
||||
for line in body.decode("utf-8", "ignore").splitlines():
|
||||
line = line.strip()
|
||||
if not line.startswith("{"):
|
||||
continue
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
for det in d.get("TextDetections", []):
|
||||
if det.get("DetectedText"):
|
||||
confs.append(det.get("Confidence", 0))
|
||||
if not confs:
|
||||
return None, 0
|
||||
return sum(1 for c in confs if c < thr) / len(confs), len(confs)
|
||||
except Exception:
|
||||
return None, 0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--jsonl", default="数据.jsonl")
|
||||
ap.add_argument("--frac", type=float, default=0.6)
|
||||
ap.add_argument("--thr", type=int, default=80)
|
||||
ap.add_argument("--min-det", type=int, default=200)
|
||||
ap.add_argument("--recent", type=int, default=0, help="只扫最近N个(0=全部)")
|
||||
ap.add_argument("--workers", type=int, default=16)
|
||||
ap.add_argument("--delete", action="store_true", help="删掉非字幕视频的 COS SRT(不可逆!)")
|
||||
ap.add_argument("--out", default="非字幕清单.txt")
|
||||
args = ap.parse_args()
|
||||
|
||||
latest = {}
|
||||
for line in open(args.jsonl, encoding="utf-8"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
v = r.get("video_cos_path")
|
||||
if v:
|
||||
latest[v] = r
|
||||
vids = [r for r in latest.values()
|
||||
if r.get("status") == "success" and r.get("composite_cos_path")]
|
||||
if args.recent:
|
||||
vids = vids[-args.recent:]
|
||||
print(f"待扫描成功视频: {len(vids)} | 阈值: 低置信<{args.thr} 占比>{args.frac*100:.0f}% | "
|
||||
f"{'★删除模式★' if args.delete else 'dry-run(只列出)'}")
|
||||
|
||||
bad, done, errored = [], 0, 0
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as ex:
|
||||
fut = {ex.submit(lowconf_ratio, r["composite_cos_path"], args.thr): r for r in vids}
|
||||
for f in as_completed(fut):
|
||||
r = fut[f]
|
||||
ratio, ndet = f.result()
|
||||
done += 1
|
||||
if ratio is None:
|
||||
errored += 1
|
||||
elif ndet >= args.min_det and ratio > args.frac:
|
||||
bad.append((r["video_cos_path"], ratio, ndet, r.get("srt_cos_path")))
|
||||
if done % 100 == 0:
|
||||
print(f" ...{done}/{len(vids)} 已扫, 命中非字幕 {len(bad)}")
|
||||
|
||||
bad.sort(key=lambda x: -x[1])
|
||||
with open(args.out, "w", encoding="utf-8") as f:
|
||||
for v, ratio, ndet, srt in bad:
|
||||
f.write(f"{v}\t低置信{ratio*100:.0f}%\t{ndet}检测\t{srt}\n")
|
||||
print(f"\n扫描完成: {done} 个 | 非字幕(应过滤) {len(bad)} 个 | 读取失败 {errored} 个")
|
||||
print(f"清单 → {args.out}")
|
||||
print("命中样例(低置信占比最高的几个):")
|
||||
for v, ratio, ndet, srt in bad[:8]:
|
||||
print(f" {ratio*100:3.0f}% {ndet:5d}检测 {os.path.basename(v)}")
|
||||
|
||||
if args.delete and bad:
|
||||
print(f"\n★删除模式: 删除 {len(bad)} 个非字幕视频的 COS SRT...")
|
||||
dn = 0
|
||||
for v, ratio, ndet, srt in bad:
|
||||
if srt:
|
||||
try:
|
||||
_client.delete_object(Bucket=B, Key=key_of(srt))
|
||||
dn += 1
|
||||
except Exception as e:
|
||||
print(" 删除失败", srt, e)
|
||||
print(f"已删除 {dn} 个 COS SRT。(清单已存 {args.out}, 需要可重跑这些视频)")
|
||||
elif bad:
|
||||
print(f"\n这是 dry-run。确认清单后, 加 --delete 删除这些垃圾SRT。")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
74
tools/导出成功xlsx.py
Normal file
74
tools/导出成功xlsx.py
Normal file
@@ -0,0 +1,74 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
把 数据/ 下所有 jsonl 的【成功】记录去重汇总成一个 xlsx。
|
||||
列: 视频地址 | 字幕地址 | 时长(秒) | 时长(分:秒)
|
||||
|
||||
用法: python3 tools/导出成功xlsx.py [--dir 数据] [--out 成功字幕汇总.xlsx]
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
from openpyxl import Workbook
|
||||
|
||||
|
||||
def fmt_dur(sec):
|
||||
try:
|
||||
s = int(round(float(sec)))
|
||||
except (TypeError, ValueError):
|
||||
return ""
|
||||
if s <= 0:
|
||||
return ""
|
||||
h, rem = divmod(s, 3600)
|
||||
m, ss = divmod(rem, 60)
|
||||
return f"{h}:{m:02d}:{ss:02d}" if h else f"{m}:{ss:02d}"
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--dir", default="数据")
|
||||
ap.add_argument("--out", default="成功字幕汇总.xlsx")
|
||||
args = ap.parse_args()
|
||||
|
||||
files = glob.glob(os.path.join(args.dir, "*.jsonl"))
|
||||
print(f"读取 {len(files)} 个 jsonl ...")
|
||||
latest = {}
|
||||
for fp in files:
|
||||
for line in open(fp, encoding="utf-8"):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
v = r.get("video_cos_path")
|
||||
if v:
|
||||
latest[v] = r
|
||||
|
||||
rows = [r for r in latest.values() if r.get("status") == "success"]
|
||||
# 按视频地址排序, 输出稳定
|
||||
rows.sort(key=lambda r: r.get("video_cos_path", ""))
|
||||
print(f"去重视频 {len(latest)} | 成功 {len(rows)} → 写 xlsx")
|
||||
|
||||
wb = Workbook(write_only=True) # 流式写, 省内存(几十万行也扛得住)
|
||||
ws = wb.create_sheet("成功字幕")
|
||||
ws.append(["视频地址", "字幕地址", "时长(秒)", "时长(分:秒)"])
|
||||
n = 0
|
||||
for r in rows:
|
||||
dur = r.get("duration")
|
||||
try:
|
||||
dsec = round(float(dur), 1) if dur is not None else ""
|
||||
except (TypeError, ValueError):
|
||||
dsec = ""
|
||||
ws.append([r.get("video_cos_path", ""),
|
||||
r.get("srt_cos_path", ""),
|
||||
dsec,
|
||||
fmt_dur(dur)])
|
||||
n += 1
|
||||
wb.save(args.out)
|
||||
print(f"完成: {n} 行 → {args.out} ({os.path.getsize(args.out)//1024//1024} MB)")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
92
tools/拆分超长.py
Normal file
92
tools/拆分超长.py
Normal file
@@ -0,0 +1,92 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
从已有 数据.jsonl 里把【超长视频】的 COS 路径提取到一个 txt，供以后补跑。
|
||||
|
||||
用途：本次改动之前，被"视频过长"过滤掉的视频没有单独留列表，源 txt 又已删除。
|
||||
但 数据.jsonl 每条都存了 duration，可据此把它们捞回来。
|
||||
|
||||
判定为"待补跑的超长视频"：
|
||||
- 该视频最新一条记录 reason == "视频过长"（新版记录有此字段）；或
|
||||
- duration > 阈值 且 status != success（旧记录没有 reason，靠时长兜底）
|
||||
成功处理过的(status=success)不导出。
|
||||
|
||||
用法:
|
||||
python tools/拆分超长.py # 读 ./数据.jsonl，阈值取 config 的 max_duration_seconds
|
||||
python tools/拆分超长.py 数据.jsonl --min 2400 # 指定文件和阈值(秒)
|
||||
python tools/拆分超长.py a.jsonl b.jsonl -o 超长待重跑.txt
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
|
||||
|
||||
def _default_threshold():
|
||||
try:
|
||||
from config.config_utils import config
|
||||
return float(config.get_subtitle_config().get('max_duration_seconds', 5400) or 5400)
|
||||
except Exception:
|
||||
return 5400.0
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('jsonl', nargs='*', default=['数据.jsonl'],
|
||||
help='一个或多个 数据.jsonl(默认 ./数据.jsonl)')
|
||||
ap.add_argument('--min', type=float, default=None,
|
||||
help='超长阈值(秒)，默认取 config.max_duration_seconds')
|
||||
ap.add_argument('-o', '--out', default='超长待重跑.txt', help='输出 txt')
|
||||
args = ap.parse_args()
|
||||
|
||||
thr = args.min if args.min is not None else _default_threshold()
|
||||
files = args.jsonl or ['数据.jsonl']
|
||||
|
||||
# 多文件合并，按 video_cos_path 取最新一条
|
||||
latest = {}
|
||||
for path in files:
|
||||
if not os.path.exists(path):
|
||||
print(f"[跳过] 不存在: {path}")
|
||||
continue
|
||||
with open(path, encoding='utf-8') as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
v = r.get('video_cos_path')
|
||||
if v:
|
||||
latest[v] = r
|
||||
|
||||
out_paths, seen = [], set()
|
||||
n_reason, n_dur = 0, 0
|
||||
for v, r in latest.items():
|
||||
if r.get('status') == 'success':
|
||||
continue
|
||||
reason = r.get('reason')
|
||||
dur = r.get('duration')
|
||||
hit = False
|
||||
if reason == '视频过长':
|
||||
hit = True; n_reason += 1
|
||||
elif dur is not None and dur > thr:
|
||||
hit = True; n_dur += 1
|
||||
if hit and v not in seen:
|
||||
seen.add(v); out_paths.append(v)
|
||||
|
||||
with open(args.out, 'w', encoding='utf-8') as f:
|
||||
for p in out_paths:
|
||||
f.write(p + '\n')
|
||||
|
||||
print(f"阈值 {thr:.0f}s | 扫描 {len(latest)} 个唯一视频")
|
||||
print(f"导出 {len(out_paths)} 个超长视频 → {args.out} (reason命中 {n_reason}, 时长兜底 {n_dur})")
|
||||
print("\n补跑提示: 放进 text2/ 并【换一个 数据.jsonl】(否则会被当已处理跳过):")
|
||||
print(f" mv {args.out} text2/long_rerun.txt")
|
||||
print(" # 建议在 GPU 机器上跑(长视频GPU硬解快); 放大 config.max_duration_seconds 让它们不再被过滤")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
93
tools/查合成图尺寸.py
Normal file
93
tools/查合成图尺寸.py
Normal file
@@ -0,0 +1,93 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
统计合成图的【像素尺寸】, 看有多少张超大(最长边>阈值) → 判断是不是OCR超时的根因。
|
||||
纯标准库, 不依赖 PIL / cv2 / 项目代码。只读JPEG文件头, 不解码, 很快。
|
||||
|
||||
用法:
|
||||
python3 tools/查合成图尺寸.py # 默认扫 ./split/**/composite/, 阈值4000
|
||||
python3 tools/查合成图尺寸.py --dir ./split --max 4000
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
|
||||
|
||||
def jpeg_size(path):
|
||||
"""只读 JPEG 文件头拿 (宽,高), 不解码。失败返回 None。"""
|
||||
try:
|
||||
with open(path, 'rb') as f:
|
||||
if f.read(2) != b'\xff\xd8':
|
||||
return None
|
||||
while True:
|
||||
b = f.read(1)
|
||||
while b and b != b'\xff':
|
||||
b = f.read(1)
|
||||
marker = f.read(1)
|
||||
while marker == b'\xff':
|
||||
marker = f.read(1)
|
||||
if not marker:
|
||||
return None
|
||||
m = marker[0]
|
||||
if 0xC0 <= m <= 0xCF and m not in (0xC4, 0xC8, 0xCC):
|
||||
f.read(3)
|
||||
h = int.from_bytes(f.read(2), 'big')
|
||||
w = int.from_bytes(f.read(2), 'big')
|
||||
return w, h
|
||||
seg = int.from_bytes(f.read(2), 'big')
|
||||
if seg < 2:
|
||||
return None
|
||||
f.seek(seg - 2, 1)
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('--dir', default='./split')
|
||||
ap.add_argument('--max', type=int, default=4000, help='超大阈值(最长边像素)')
|
||||
args = ap.parse_args()
|
||||
|
||||
pats = [os.path.join(args.dir, '**', 'composite', '*.jpg'),
|
||||
os.path.join(args.dir, '**', '*.jpg')]
|
||||
files = []
|
||||
for p in pats:
|
||||
files = glob.glob(p, recursive=True)
|
||||
if files:
|
||||
break
|
||||
files = [f for f in files if 'composite' in f]
|
||||
if not files:
|
||||
print(f"在 {args.dir} 下没找到 composite/*.jpg")
|
||||
return
|
||||
|
||||
tot = over = mx = 0
|
||||
sizes = []
|
||||
for f in files:
|
||||
sz = jpeg_size(f)
|
||||
if sz:
|
||||
tot += 1
|
||||
m = max(sz)
|
||||
mx = max(mx, m)
|
||||
sizes.append((os.path.getsize(f), sz, f))
|
||||
if m > args.max:
|
||||
over += 1
|
||||
|
||||
print("=" * 56)
|
||||
print(f"合成图 {tot} 张 | 最长边>{args.max} 的 {over} 张 "
|
||||
f"({over/tot*100:.0f}%) | 最大最长边 {mx}px")
|
||||
print("=" * 56)
|
||||
print("最大的 8 张:")
|
||||
for fsize, (w, h), f in sorted(sizes, reverse=True)[:8]:
|
||||
flag = ' <== 超大' if max(w, h) > args.max else ''
|
||||
print(f" {fsize//1024:5d}KB {w}x{h} 最长边={max(w,h)}{flag}")
|
||||
print()
|
||||
if over > tot * 0.3:
|
||||
print(f"⚠ {over}/{tot} 张超大 → 实锤: 超大合成图就是OCR超时根因。")
|
||||
print(f" 部署 ocr_max_image_side={args.max} 后, 这些缩到 {args.max}px, OCR 从30s降到几秒。")
|
||||
elif over == 0:
|
||||
print("✓ 没有超大图(尺寸都正常) → 超时不是图大, 是端点问题(共享/波动)。")
|
||||
else:
|
||||
print(f"部分超大({over}张), 部署尺寸上限能缓解这部分。")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
147
tools/查端点存活.py
Normal file
147
tools/查端点存活.py
Normal file
@@ -0,0 +1,147 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
逐个探测 config 里的 OCR 端点, 看有多少台【真的在服务】。
|
||||
用来确认"120台只活了20台"这类问题——死端点越多, 活端点被压得越狠 → 超时。
|
||||
|
||||
用法:
|
||||
python3 tools/查端点存活.py # 自动找一张真实帧图做OCR探测
|
||||
python3 tools/查端点存活.py --tcp # 只测TCP端口连通(更快, 不发OCR)
|
||||
python3 tools/查端点存活.py --timeout 5
|
||||
python3 tools/查端点存活.py --list-dead # 额外列出死端点IP
|
||||
"""
|
||||
import argparse
|
||||
import base64
|
||||
import glob
|
||||
import os
|
||||
import socket
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
import requests
|
||||
from config.config_utils import config
|
||||
|
||||
OCR_PATH = "/youtu/ocrapi/hpgeneralocr_v3"
|
||||
DEFAULT_PORT = 18081
|
||||
|
||||
|
||||
def norm(e):
|
||||
e = str(e).strip().rstrip("/")
|
||||
if not e:
|
||||
return None
|
||||
if not e.startswith("http"):
|
||||
e = "http://" + e
|
||||
host = e.split("://", 1)[1]
|
||||
if ":" not in host:
|
||||
e = e + ":" + str(DEFAULT_PORT)
|
||||
return e
|
||||
|
||||
|
||||
def host_port(ep):
|
||||
h = ep.split("://", 1)[1]
|
||||
host, _, port = h.partition(":")
|
||||
return host, int(port or DEFAULT_PORT)
|
||||
|
||||
|
||||
def find_image_b64():
|
||||
"""找一张真实帧图转 base64; 找不到返回 None(改用TCP探测)。"""
|
||||
for pat in ("./split/**/*.jpg", "./composite/**/*.jpg", "./split/*/*.jpg"):
|
||||
for p in glob.glob(pat, recursive=True):
|
||||
try:
|
||||
with open(p, "rb") as f:
|
||||
return base64.b64encode(f.read()).decode()
|
||||
except OSError:
|
||||
continue
|
||||
return None
|
||||
|
||||
|
||||
def probe_ocr(ep, img_b64, timeout):
|
||||
url = ep + OCR_PATH
|
||||
payload = {"app_id": str(int(time.time() * 1000)), "image": img_b64,
|
||||
"ocr_template": "OCR", "options": {"pdf_page_index": 0, "detect_text": True}}
|
||||
t0 = time.time()
|
||||
try:
|
||||
r = requests.post(url, json=payload, timeout=timeout,
|
||||
headers={"Content-Type": "application/json"})
|
||||
dt = time.time() - t0
|
||||
return (ep, "alive", dt, "") if r.status_code == 200 else (ep, "http_err", dt, f"HTTP {r.status_code}")
|
||||
except requests.exceptions.ConnectTimeout:
|
||||
return (ep, "dead", time.time() - t0, "连接超时")
|
||||
except requests.exceptions.ReadTimeout:
|
||||
return (ep, "slow", time.time() - t0, "读超时(连得上不回)")
|
||||
except requests.exceptions.ConnectionError:
|
||||
return (ep, "dead", time.time() - t0, "拒绝/不可达")
|
||||
except Exception as e:
|
||||
return (ep, "dead", time.time() - t0, str(e)[:40])
|
||||
|
||||
|
||||
def probe_tcp(ep, timeout):
|
||||
host, port = host_port(ep)
|
||||
t0 = time.time()
|
||||
try:
|
||||
with socket.create_connection((host, port), timeout=timeout):
|
||||
return (ep, "alive", time.time() - t0, "TCP通")
|
||||
except socket.timeout:
|
||||
return (ep, "dead", time.time() - t0, "TCP超时")
|
||||
except Exception as e:
|
||||
return (ep, "dead", time.time() - t0, str(e)[:40])
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--timeout", type=float, default=5.0)
|
||||
ap.add_argument("--tcp", action="store_true", help="只测TCP端口连通(不发OCR)")
|
||||
ap.add_argument("--list-dead", action="store_true")
|
||||
args = ap.parse_args()
|
||||
|
||||
eps = config.get_tencent_ocr_config().get("endpoints") or []
|
||||
eps = [u for u in (norm(e) for e in eps) if u]
|
||||
|
||||
img = None
|
||||
if not args.tcp:
|
||||
img = find_image_b64()
|
||||
if img is None:
|
||||
print("没找到帧图, 改用 TCP 连通探测(--tcp)。")
|
||||
mode = "TCP连通" if (args.tcp or img is None) else "真实OCR请求"
|
||||
print(f"探测 {len(eps)} 个端点 (方式: {mode}, 超时 {args.timeout}s)...")
|
||||
|
||||
def work(e):
|
||||
if args.tcp or img is None:
|
||||
return probe_tcp(e, args.timeout)
|
||||
return probe_ocr(e, img, args.timeout)
|
||||
|
||||
res = []
|
||||
with ThreadPoolExecutor(max_workers=64) as ex:
|
||||
for f in as_completed([ex.submit(work, e) for e in eps]):
|
||||
res.append(f.result())
|
||||
|
||||
c = Counter(r[1] for r in res)
|
||||
alive = [r for r in res if r[1] in ("alive", "http_err")]
|
||||
lats = sorted(r[2] for r in alive)
|
||||
total = len(eps)
|
||||
good = c["alive"] + c["http_err"]
|
||||
print("=" * 52)
|
||||
print(f" 可用(alive/http_err): {good}")
|
||||
print(f" 慢(读超时): {c['slow']}")
|
||||
print(f" 死(连不上/拒绝): {c['dead']}")
|
||||
if lats:
|
||||
print(f" 可用端点延迟: 中位 {lats[len(lats)//2]*1000:.0f}ms 最慢 {lats[-1]*1000:.0f}ms")
|
||||
print("=" * 52)
|
||||
print(f"结论: {total} 个端点里 {good} 个可用 ({good/total*100:.0f}%)")
|
||||
if good < total * 0.8:
|
||||
print(f"⚠️ {total-good} 个端点不可用! 负载全压在 {good} 个上 → 这极可能就是OCR超时根因。")
|
||||
print(f" 把死端点服务拉起 / 或从config删掉死的, OCR容量最高可提升 {total/max(good,1):.1f} 倍。")
|
||||
else:
|
||||
print("端点基本都活着 → OCR超时不是'端点少', 是'整体容量/CPU抢占'问题。")
|
||||
|
||||
if args.list_dead:
|
||||
print("\n死/慢端点:")
|
||||
for ep, st, dt, msg in sorted(res, key=lambda x: x[1]):
|
||||
if st in ("dead", "slow"):
|
||||
print(f" {st:6s} {ep:32s} {msg}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
111
tools/检查srt质量.py
Normal file
111
tools/检查srt质量.py
Normal file
@@ -0,0 +1,111 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
扫描 SRT 文件, 报告字幕质量指标 + 抽样, 帮你快速判断OCR字幕好不好。
|
||||
纯标准库。
|
||||
|
||||
用法:
|
||||
python3 tools/检查srt质量.py ./srt # 扫目录下所有 .srt
|
||||
python3 tools/检查srt质量.py a.srt b.srt # 扫指定文件
|
||||
python3 tools/检查srt质量.py ./srt --sample 8 # 多抽几条看
|
||||
"""
|
||||
import argparse
|
||||
import glob
|
||||
import os
|
||||
import re
|
||||
|
||||
CJK = re.compile(r'[一-鿿]')
|
||||
|
||||
|
||||
def parse_srt(path):
|
||||
"""返回 [(idx, start, end, text), ...]"""
|
||||
try:
|
||||
raw = open(path, encoding='utf-8', errors='ignore').read()
|
||||
except OSError:
|
||||
return []
|
||||
blocks = re.split(r'\n\s*\n', raw.strip())
|
||||
out = []
|
||||
for b in blocks:
|
||||
lines = [x for x in b.splitlines() if x.strip()]
|
||||
if len(lines) < 2:
|
||||
continue
|
||||
m = re.search(r'(\d+):(\d+):(\d+)[,.](\d+)\s*-->\s*(\d+):(\d+):(\d+)[,.](\d+)', b)
|
||||
if not m:
|
||||
continue
|
||||
g = list(map(int, m.groups()))
|
||||
st = g[0]*3600 + g[1]*60 + g[2] + g[3]/1000.0
|
||||
en = g[4]*3600 + g[5]*60 + g[6] + g[7]/1000.0
|
||||
ti = next((i for i, l in enumerate(lines) if '-->' in l), 0)
|
||||
text = ' '.join(lines[ti+1:]).strip()
|
||||
out.append((st, en, text))
|
||||
return out
|
||||
|
||||
|
||||
def has_self_repeat(s, minlen=4):
|
||||
"""检测整条里前半段==后半段这种自我重复(如 AABB → AB 重复)。"""
|
||||
n = len(s)
|
||||
if n < minlen * 2:
|
||||
return False
|
||||
h = n // 2
|
||||
return s[:h] == s[h:2*h] and len(s[:h]) >= minlen
|
||||
|
||||
|
||||
def scan(paths, sample):
|
||||
files = []
|
||||
for p in paths:
|
||||
if os.path.isdir(p):
|
||||
files += glob.glob(os.path.join(p, '**', '*.srt'), recursive=True)
|
||||
elif p.endswith('.srt'):
|
||||
files.append(p)
|
||||
if not files:
|
||||
print("没找到 .srt")
|
||||
return
|
||||
|
||||
tot_e = empty = dup = selfrep = tooshort = lowcjk = badtime = 0
|
||||
chars = 0
|
||||
samples = []
|
||||
for f in files:
|
||||
subs = parse_srt(f)
|
||||
prev = None
|
||||
for st, en, text in subs:
|
||||
tot_e += 1
|
||||
chars += len(text)
|
||||
if not text:
|
||||
empty += 1
|
||||
if prev is not None and text and text == prev:
|
||||
dup += 1
|
||||
if has_self_repeat(text):
|
||||
selfrep += 1
|
||||
if text and len(text) <= 1:
|
||||
tooshort += 1
|
||||
if text and len(CJK.findall(text)) < len(text) * 0.3 and len(text) > 4:
|
||||
lowcjk += 1 # 中文占比过低 → 可能乱码/纯符号
|
||||
if en <= st or (en - st) > 30:
|
||||
badtime += 1
|
||||
prev = text
|
||||
if subs and len(samples) < sample:
|
||||
mid = len(subs) // 2
|
||||
for s in subs[mid:mid+2]:
|
||||
samples.append((os.path.basename(f), s))
|
||||
|
||||
print("=" * 58)
|
||||
print(f"SRT 文件 {len(files)} 个 | 字幕条目 {tot_e} 条 | 平均 {chars/max(tot_e,1):.1f} 字/条")
|
||||
print("-" * 58)
|
||||
def pct(x): return f"{x} ({x/max(tot_e,1)*100:.1f}%)"
|
||||
print(f" 空条目: {pct(empty)} <- 应≈0")
|
||||
print(f" 连续重复(同上条): {pct(dup)} <- 高=去重没做好")
|
||||
print(f" 自我重复(AABB): {pct(selfrep)} <- 高=自重复bug")
|
||||
print(f" 单字条目: {pct(tooshort)}")
|
||||
print(f" 低中文占比(疑乱码):{pct(lowcjk)} <- 高=OCR乱/纯符号")
|
||||
print(f" 时间轴异常: {pct(badtime)} <- 重叠/0时长/超30s")
|
||||
print("=" * 58)
|
||||
print("抽样字幕:")
|
||||
for fn, (st, en, text) in samples[:sample]:
|
||||
print(f" [{st:6.1f}-{en:6.1f}] {text[:50]} ({fn})")
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument('paths', nargs='+')
|
||||
ap.add_argument('--sample', type=int, default=6)
|
||||
a = ap.parse_args()
|
||||
scan(a.paths, a.sample)
|
||||
92
tools/看OCR分布.py
Normal file
92
tools/看OCR分布.py
Normal file
@@ -0,0 +1,92 @@
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
聚合 ./ocr_dispatch/<pid>.json (各 worker 进程写的派发计数), 看 119 个 OCR 端点各分到多少,
|
||||
直接验证 P2C 负载均衡是否均匀。
|
||||
|
||||
用法:
|
||||
python3 tools/看OCR分布.py # 默认只统计最近5分钟内更新过的进程(活进程)
|
||||
python3 tools/看OCR分布.py --all # 包含所有(含已退出的旧进程文件)
|
||||
python3 tools/看OCR分布.py --fresh 120 # 只算最近120秒更新的
|
||||
python3 tools/看OCR分布.py --top 20 # 只看派发最多/最少各20个
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
import glob
|
||||
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--dir", default="./ocr_dispatch", help="计数文件目录")
|
||||
ap.add_argument("--fresh", type=int, default=300, help="只统计最近N秒更新过的进程文件(默认300)")
|
||||
ap.add_argument("--all", action="store_true", help="统计所有文件(忽略新鲜度)")
|
||||
ap.add_argument("--top", type=int, default=0, help="只显示派发最多/最少各N个(0=全部)")
|
||||
args = ap.parse_args()
|
||||
|
||||
if not os.path.isdir(args.dir):
|
||||
print(f"没有目录 {args.dir} —— 还没产生派发计数。")
|
||||
print("确认 config.tencentcloud.ocr.dispatch_log_every > 0, 且程序已跑一会儿(每1000次派发才落盘)。")
|
||||
raise SystemExit(0)
|
||||
|
||||
now = time.time()
|
||||
files = glob.glob(os.path.join(args.dir, "*.json"))
|
||||
counts = {} # endpoint -> 总派发数
|
||||
lat_sum = {} # endpoint -> 延迟加权和(按该进程total加权)
|
||||
lat_w = {}
|
||||
nproc = 0
|
||||
grand = 0
|
||||
for fp in files:
|
||||
try:
|
||||
if not args.all and (now - os.path.getmtime(fp)) > args.fresh:
|
||||
continue
|
||||
d = json.load(open(fp, encoding="utf-8"))
|
||||
except Exception:
|
||||
continue
|
||||
nproc += 1
|
||||
for ep, c in (d.get("counts") or {}).items():
|
||||
counts[ep] = counts.get(ep, 0) + c
|
||||
grand += c
|
||||
for ep, l in (d.get("lat_ewma") or {}).items():
|
||||
tot = d.get("total", 0) or 0
|
||||
lat_sum[ep] = lat_sum.get(ep, 0.0) + l * tot
|
||||
lat_w[ep] = lat_w.get(ep, 0) + tot
|
||||
|
||||
if not counts:
|
||||
print(f"没有可统计的计数(活进程文件0个)。用 --all 看历史, 或确认程序在跑。")
|
||||
raise SystemExit(0)
|
||||
|
||||
n = len(counts)
|
||||
avg = grand / n if n else 0
|
||||
rows = []
|
||||
for ep, c in counts.items():
|
||||
lat = (lat_sum.get(ep, 0) / lat_w[ep]) if lat_w.get(ep) else 0.0
|
||||
rows.append((c, ep, lat))
|
||||
rows.sort(reverse=True)
|
||||
|
||||
mx = rows[0][0] if rows else 0
|
||||
mn = rows[-1][0] if rows else 0
|
||||
import statistics
|
||||
cs = [r[0] for r in rows]
|
||||
std = statistics.pstdev(cs) if len(cs) > 1 else 0
|
||||
|
||||
print("=" * 60)
|
||||
print(f"统计进程数: {nproc} 端点数: {n} 总派发: {grand}")
|
||||
print(f"每端点平均: {avg:.0f} 最多: {mx} 最少: {mn} max/min: {(mx/mn) if mn else float('inf'):.2f}")
|
||||
print(f"标准差: {std:.0f} 变异系数(越小越均匀): {(std/avg*100) if avg else 0:.1f}%")
|
||||
print("=" * 60)
|
||||
|
||||
show = rows
|
||||
note = ""
|
||||
if args.top and len(rows) > 2 * args.top:
|
||||
show = rows[:args.top] + [(-1, "...", -1)] + rows[-args.top:]
|
||||
note = f"(只显示最多/最少各 {args.top} 个)"
|
||||
print(f"端点派发明细 {note}:")
|
||||
print(f" {'端点':<24}{'派发数':>8}{'占比':>8}{'延迟ms':>8}")
|
||||
for c, ep, lat in show:
|
||||
if c < 0:
|
||||
print(" ...")
|
||||
continue
|
||||
print(f" {ep:<24}{c:>8}{c/grand*100:>7.1f}%{lat*1000:>7.0f}")
|
||||
|
||||
print()
|
||||
print("判读: 变异系数 <15% 算很均匀; max/min 接近1最好。")
|
||||
print(" 某端点'派发少+延迟高'=它慢被P2C自动避开(正常); '派发少+延迟正常'=可能刚冷却过。")
|
||||
57
常用命令.txt
Normal file
57
常用命令.txt
Normal file
@@ -0,0 +1,57 @@
|
||||
==================== 常用命令(在项目根目录执行)====================
|
||||
|
||||
【1. 从 COS 下载 tar-gz 目录的所有文件】
|
||||
./coscli-linux-amd64 -c .cos.yaml cp -r cos://asr-dig-data-1253181912/0004/tar-gz/ ./tar-gz/
|
||||
|
||||
# 提高并发(文件多时更快)
|
||||
./coscli-linux-amd64 -c .cos.yaml cp -r cos://asr-dig-data-1253181912/0004/tar-gz/ ./tar-gz/ --routines 16
|
||||
|
||||
|
||||
【2. 上传到 COS 的 tar-gz 目录】
|
||||
# 上传单个文件(保留原名)
|
||||
./coscli-linux-amd64 -c .cos.yaml cp ./文件名.tar.gz cos://asr-dig-data-1253181912/0004/tar-gz/
|
||||
|
||||
# 上传整个目录(加 -r)
|
||||
./coscli-linux-amd64 -c .cos.yaml cp -r ./本地目录/ cos://asr-dig-data-1253181912/0004/tar-gz/
|
||||
|
||||
|
||||
【3. 查看 tar-gz 目录里有哪些文件】
|
||||
./coscli-linux-amd64 -c .cos.yaml ls cos://asr-dig-data-1253181912/0004/tar-gz/
|
||||
|
||||
|
||||
==================== 项目启动 / 停止 ====================
|
||||
|
||||
【启动(进程池, 后台, 写日志到 app.log)】
|
||||
nohup python main.py --source-list ./text2 --max-workers 56 --executor process > app.log 2>&1 &
|
||||
|
||||
# 96核机器内存充裕可调大: --max-workers 64
|
||||
|
||||
【实时看日志】
|
||||
tail -f app.log
|
||||
|
||||
【停止(连 ffmpeg 子进程一起杀)】
|
||||
bash scripts/stop.sh -y
|
||||
|
||||
# 或手动:
|
||||
pkill -9 -f main.py ; pkill -9 -f multiprocessing ; pkill -9 ffmpeg
|
||||
|
||||
|
||||
==================== 监控 / 统计 ====================
|
||||
|
||||
【后台采集 CPU/IO/负载/网络】
|
||||
nohup bash scripts/monitor.sh 5 monitor.log &
|
||||
|
||||
【看每小时吞吐 + 有效并发(对照 max-workers 看吃满没)】
|
||||
python tools/hourly_stats.py 数据.jsonl --last 6
|
||||
|
||||
【看每天处理量】
|
||||
python tools/daily_stats.py 数据.jsonl
|
||||
|
||||
|
||||
==================== 备注 ====================
|
||||
- Linux 首次用 coscli 先给执行权限: chmod +x coscli-linux-amd64
|
||||
- 下载/上传目录必须带 -r
|
||||
- 目标路径以 / 结尾=保留原名; 带文件名=改成该名
|
||||
- 启动靠 数据.jsonl 断点续传, 重启会跳过已处理的视频
|
||||
|
||||
|
||||
Reference in New Issue
Block a user