初始提交：抖音视频批量采集流水线
App 端免签名接口（设备注册 + detail + CDN 直链），一体化流水线 下载→抽音→传COS→记录→删本地，AI 作品四档判定，四份文档： README（使用指南）、PROGRESS（批次进度）、RE（逆向全过程）、AI（判定规则）。 Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
28
.gitignore
vendored
Normal file
28
.gitignore
vendored
Normal file
@@ -0,0 +1,28 @@
|
||||
# 媒体产物（体积大，不入库）
|
||||
downloads/
|
||||
audio/
|
||||
*.mp4
|
||||
*.mp3
|
||||
*.m4a
|
||||
*.tmp
|
||||
|
||||
# 数据与产物（体积极大，不入库；lists/ 是凭据，入库）
|
||||
data/
|
||||
records/
|
||||
exports/
|
||||
work/
|
||||
|
||||
# 运行态 / 断点续传记录
|
||||
.device.json
|
||||
.downloaded.json
|
||||
uploaded.txt
|
||||
*.failed.txt
|
||||
*.log
|
||||
|
||||
# 凭证
|
||||
cookie.txt
|
||||
.env
|
||||
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.idea/
|
||||
215
AI.md
Normal file
215
AI.md
Normal file
@@ -0,0 +1,215 @@
|
||||
# AI 判定规则详解
|
||||
|
||||
> 本文档逐条解释 `douyin/aiclass.py` 的判定逻辑：信号从哪来、四档怎么定、
|
||||
> 每个词表为什么这么选、已知误收和否决理由。改规则之前必读；
|
||||
> 每个坑在 `test_aiclass.py` 里都有对应测试（29 个），改完务必跑通。
|
||||
> 各批筛选结果和下载进度见 [PROGRESS.md](PROGRESS.md) §4.5。
|
||||
|
||||
---
|
||||
|
||||
## 1. 判定对象：三个文本源，地位不同
|
||||
|
||||
每条记录抽出三段文本，权重严格递减：
|
||||
|
||||
| 来源 | 构成 | 地位 |
|
||||
|---|---|---|
|
||||
| **body**（作品文本） | `desc` + 所有 hashtags 的 name | 主信号场，能定 strong / topic |
|
||||
| **author**（作者昵称） | `author.nickname` | 只能兜底成 author 档，**绝不往上顶** |
|
||||
| **music**（音乐标题） | `music.title`，但先剥壳（见 §6） | 同上 |
|
||||
|
||||
「作者昵称 / 音乐标题」回答的是**谁发的**，不是**作品是什么**。实测
|
||||
@牛牛Ai 发普法解说、@你好，我是大福 发真人游泳视频——昵称带 AI 的作品
|
||||
多数不是 AI。所以 author/music 的命中只能在作品自身毫无信号时兜底成
|
||||
`author` 档（`classify()` 里 `if tier == "none"` 才采纳）。
|
||||
|
||||
## 2. 四档体系
|
||||
|
||||
档位之间不重叠，一条记录取**最强**的一档：
|
||||
|
||||
```
|
||||
platform > strong > topic > author > none
|
||||
```
|
||||
|
||||
| 档 | 依据 | 含义 | 可信度 |
|
||||
|---|---|---|---|
|
||||
| `platform` | 爬虫打的 `ai.is_aigc` | 抖音官方 AIGC 标 | 最高，且唯一能抓到「作者什么都没标但确实是 AI」的作品——实测 1744 条文案零 AI 字样 |
|
||||
| `strong` | body 里有明确「这是 AI 做的」信号 | 见 §3 的四路信号 | 高 |
|
||||
| `topic` | body 只提到 AI 本身 | 在讲 AI（教程/新闻/大模型），不一定是 AI 做的 | 中 |
|
||||
| `author` | 只有昵称/音乐标题带 AI | 兜底档，会误收 | 低 |
|
||||
| `none` | 无信号 | | |
|
||||
|
||||
⚠️ **已知 bug**：爬虫层把「`risk_infos` 有内容」就当 `is_aigc=true`，把
|
||||
虚构演绎/个人观点等免责声明也算进了 platform 档，全库误算 1029 条。正确做法
|
||||
是认 `risk_infos.type ∈ {44, 47, 76, 328}`（44=作者声明、47=疑似、76/328=平台判定），
|
||||
修法在 PROGRESS.md §7 待办。`raw.is_aigc_media` 全库皆 false，别信。
|
||||
|
||||
## 3. strong 档的四路信号（body 里任一成立）
|
||||
|
||||
### 3.1 `aigc` token
|
||||
|
||||
独立的 `aigc` 字样直接 strong。边界规则见 §5。
|
||||
|
||||
### 3.2 生成工具名单 `_GEN_TOOLS`——命中即 strong，不要求 ai 同现
|
||||
|
||||
用了生成工具 = 片子是 AI 做的，这是最强的文本证据。名单 70+ 个，分三类：
|
||||
|
||||
- **中文工具**：即梦、可灵、海螺（注意这个在歧义表，见 3.5）、海艺、小云雀、
|
||||
妙鸭、一帧秒创、腾讯智影、讯飞智作、硅基智能、魔音工坊、元气口播、万兴播爆、
|
||||
闪剪、声音克隆/语音克隆/克隆音色/音色克隆……
|
||||
- **英文工具**：dreamina、jimeng、seedance、seedream、kling、vidu、pixverse、
|
||||
pika、luma、runway、sora、viggle、heygen、midjourney、comfyui、flux、
|
||||
nanobanana、stable diffusion、liblib、suno、udio、veo3、hunyuanvideo、
|
||||
cogvideo、controlnet、domoai、synthesia、deepbrain……
|
||||
- **生成动词**：文生视频、图生视频、文生图、图生图、生成式
|
||||
|
||||
**选词标准**（2026-08 扩充时定的）：「这串字在中文短视频文案里不可能有别的意思」。
|
||||
全部在 12.7 万条真实数据上验过——要么零命中（备着以后用），要么命中全是真 AI。
|
||||
反过来，这些被明确**拒收**：`tripo`（撞 tripod 三脚架）、`seaweed`（海藻）、
|
||||
`lora`（撞 valorant，实测命中 3 条全是无畏契约）、`captions`、`playground`。
|
||||
|
||||
### 3.3 AI 漫剧专有名词 `_MANJU`——单独出现即 strong
|
||||
|
||||
AI 漫剧已形成固定圈子，这些词在 18.4 万条数据里几乎只出现在 AI 作品上
|
||||
（括号=全库命中/仍漏掉的）：
|
||||
|
||||
- 圈内词：漫剧(1568/64)、治愈系漫画(80/0)、渡灵书(62/0)、超级裂变(46/0)、
|
||||
二次元动漫(33/0)、月落小溪(25/4)、七罪创世(21/0)、晚夏小梦(67/4)、
|
||||
灵绘漫客(12/0)、neowow(12/0)、猫箱跨次元相遇(114/12) 等
|
||||
- **红果片名**（字节免费短剧 App 的 AI 漫剧）：万妖图录传、发配边关、
|
||||
罪妻开荒养出战神、斩仙台、杂灵根……**来源是新闻报道热榜，不是红果完整片库**，
|
||||
红果不公开片库，遇到新片名往这加
|
||||
|
||||
实测**拒收**的（题材 ≠ AI 标记）：西游记(+26 真·玄奘故里游记)、孙悟空(+16)、
|
||||
宫崎骏(+10 龙猫童年)、民间故事、睡前故事、国漫推荐(+4《剑来》真人制作)、
|
||||
原创动漫(+18 手绘二次元)、创作者激励计划、未来导演扶持计划、开放赛道（后三个是
|
||||
抖音通用活动）。
|
||||
|
||||
**为什么「红果」两个字不能进表**：红果主体是真人短剧，AI 漫剧只是一个赛道。
|
||||
实测 3 条命中 2 条真人短剧（《家里家外2》《冷御失守》）、1 条演员面试。
|
||||
要抓红果的 AI 漫剧只能靠具体片名。
|
||||
|
||||
### 3.4 `ai` + 创作语境词 `_GEN_CTX`——同现才成立
|
||||
|
||||
单独的 `ai` 只能算 topic。必须跟 41 个创作/生成语境词之一同现才升级 strong：
|
||||
短剧、漫剧、动画、视频、绘画、创作、制作、生成、分身、配音、换脸、特效、导演、
|
||||
数字人、虚拟人、成片、写真、转绘、扩图、提示词……
|
||||
|
||||
反过来，中文泛词（动画/绘画/分身/动漫）**不跟 ai 同现就一律不算**——830 条
|
||||
`#原创动画` 全是手绘，早期版本单独算信号误收几千条。`剪映`(375)/`关键帧`(314)/
|
||||
`蒙版`(203)/`转场` 是剪辑术语，根本不是 AI 信号。
|
||||
|
||||
### 3.5 歧义工具 `_AMBIG_GEN`——必须跟 ai/aigc 同现
|
||||
|
||||
工具名本身是常用词，单独出现几乎全是误判（数据来自全库统计）：
|
||||
|
||||
| 词 | 误判实例 | 真命中 |
|
||||
|---|---|---|
|
||||
| 豆包 | 224 条：狗名、豆沙包、「反强包之神」 | 145 条，全带 AI 字样 |
|
||||
| 海螺 | 13 条：三峰「海螺」帐篷 | 全带 AI 字样 |
|
||||
| 元宝 | 11 条：316 不锈钢元宝勺 | 全带 AI 字样 |
|
||||
| 万相 | 17 条：《龙神万相》、万相集 | 全带 AI 字样 |
|
||||
| 星流 | 「明星流量」子串 | 全带 AI 字样 |
|
||||
| 清影 | 「起舞弄清影」 | 全带 AI 字样 |
|
||||
|
||||
真命中**无一例外**同时带 AI 字样，所以这道门槛不掉召回。同机制的还有
|
||||
`_AMBIG_CHAT`（豆包、元宝、混元，归 topic 档）。
|
||||
|
||||
## 4. topic 档：在讲 AI，不一定是 AI 做的
|
||||
|
||||
- **模型名单** `_CHAT_TOOLS`：chatgpt、deepseek、kimi、通义、千问、qwen、文心、
|
||||
ernie、智谱、glm、copilot、gemini、grok，及中文泛词 人工智能、大模型、
|
||||
神经网络、机器学习、深度学习
|
||||
- **独立成词的** `_CHAT_RE`：gpt、claude、llm、agi、ai agent（要求词边界，
|
||||
避免 gpt 撞上别的串）
|
||||
|
||||
命中即 topic；若同时撞了 §3 的 strong 信号，取 strong。
|
||||
|
||||
## 5. 归一化与词边界：AI 的三种写法、词中词的两难
|
||||
|
||||
### 5.1 AI 的冒充写法
|
||||
|
||||
`ai` 的第二个字符实际匹配四种：`i`、小写 `l`、罗马数字 `Ⅰ`/`ⅰ`（U+2160/U+2170）：
|
||||
|
||||
- `Al`（大写 A + 小写 L）：标签 `抖音al` 出现 129 次，文案有 `#Al创作`
|
||||
- `#AⅠ漫剧`：罗马数字冒充，与 `Al` 同套路
|
||||
|
||||
### 5.2 独立词边界
|
||||
|
||||
- `_AI` / `_AIGC`：前后不能接字母数字（挡 email、detail、again、hair），
|
||||
但**允许接中文**（`AI绘画` 要命中）
|
||||
- `_alt()`（工具/模型/漫剧名单共用的编译器）：纯 ASCII 词只挡字母、
|
||||
**不挡数字和中文**——数据里满地 `Seedance2.0`、`sora2`、`runninghub0.21`、
|
||||
`libtv创作者`，一刀切 `\b` 会丢 3500+ 条真命中
|
||||
- 但真正的词中词要挡：`udio` 藏在 `Grass Studio`/`theonestudio` 里 17 次、
|
||||
`flux` 藏在 `Fluxxwave` 曲名里 1 次（都是音乐标题混进 author 档的）
|
||||
|
||||
## 6. 音乐标题剥壳
|
||||
|
||||
抖音默认原声的标题就是个壳，里面是作者昵称复读，不含作品信息：
|
||||
|
||||
```
|
||||
jsonl: @<昵称>创作的原声
|
||||
csv: @<昵称>创作的原声 - <昵称> ← 后缀那半截也要剥
|
||||
```
|
||||
|
||||
**不剥会怎样**：昵称带 AI 会连「创作」一起命中 `_GEN_CTX`，作者昵称就从
|
||||
music 这条路偷偷升到 strong——实测 771 条这样混进来（@冠宇Aigc 的猴子炒菜、
|
||||
@牛牛Ai 的大案纪实）。剥完后音乐标题只剩 16 条还有命中，这条路本来就没什么
|
||||
价值，只配给 author 档兜底。注意 csv 的壳多一截 ` - 昵称`，剥壳正则必须两种都覆盖，
|
||||
否则 bug 在 csv 输入上复发。
|
||||
|
||||
## 7. 明确否决的信号（放开只进误判，实测一条真的都捞不到）
|
||||
|
||||
| 否决词 | 污染来源 |
|
||||
|---|---|
|
||||
| 数字人 | 数字人民币 |
|
||||
| 分身 | 微信分身、五五分身材 |
|
||||
| 换脸 | 「这套穿搭能换脸」 |
|
||||
| lora | 无畏契约 valorant |
|
||||
| tripo | tripod 三脚架 |
|
||||
| seaweed | 海藻 |
|
||||
| 度加 | 「亏损幅度加大」 |
|
||||
| 万彩 | 「29万彩礼」 |
|
||||
| 生数 | 「诞生数副名场面」 |
|
||||
| 节点 | 「关键节点」 |
|
||||
| 抽卡 | 原神 |
|
||||
| 捏脸 | 燕云十六声 |
|
||||
| 国漫 | 《剑来》真人制作 |
|
||||
| 对口型 | 真人对口型 |
|
||||
|
||||
测试锚点：`test_rejected_candidates_stay_none`、`test_digital_rmb_is_not_a_digital_human`、
|
||||
`test_hongguo_platform_name_is_not_an_ai_signal`。
|
||||
|
||||
## 8. 维护指南
|
||||
|
||||
**加新工具名**：
|
||||
|
||||
1. 先问「这串字在中文短视频文案里有没有别的意思」。没有 → `_GEN_TOOLS`；
|
||||
是常用词但真命中都带 AI → `_AMBIG_GEN`；是对话模型 → `_CHAT_TOOLS`；
|
||||
常用词模型 → `_AMBIG_CHAT`
|
||||
2. 在真实数据上跑一遍验证：命中应全为真 AI，或至少零命中
|
||||
3. 加测试用例到 `test_aiclass.py`（用真实文案，别造）
|
||||
4. 红果新片名 → `_MANJU` 片名区；通用活动标签别收
|
||||
|
||||
**判档结果怎么用**：`clean_ai.py --tier platform,strong` 是默认推荐组合
|
||||
（可下载清单的标准切法）；`topic` 档是「讲 AI 的新闻科普」，多数采集场景不要；
|
||||
`--platform-needs-text` 用于卡「平台标了 AIGC 但文案零信号」的带货视频，但会误伤
|
||||
@栖光 类 AI 古风美图作者（文案一个 AI 字都不写），这批要按是否带货再分
|
||||
（`lists/platform无佐证_*.txt`）。
|
||||
|
||||
## 9. 测试对照表（改规则的回归网）
|
||||
|
||||
| 测试 | 守什么 |
|
||||
|---|---|
|
||||
| `test_platform_label_wins_even_with_no_ai_text` | platform 档优先，文案零信号也成立 |
|
||||
| `test_strong_*`（5 个） | 四路 strong 信号：复合词、工具名、aigc、Al 冒充 |
|
||||
| `test_topic_*`（2 个） | 纯 AI 提及、模型名只给 topic |
|
||||
| `test_author_never_upgrades_a_scored_item` | 昵称/music 绝不把已定级的作品往上顶 |
|
||||
| `test_default_original_sound_is_not_a_work_signal` / `test_sound_wrapper_csv_form_is_stripped` | 音乐壳剥壳（jsonl + csv 两种写法） |
|
||||
| `test_english_words_containing_ai_are_not_signals` | email/detail/hair 不算 |
|
||||
| `test_editing_jargon_is_not_ai` / `test_generic_chinese_words_alone_are_not_ai` | 剪辑术语、中文泛词单独不成立 |
|
||||
| `test_ambiguous_tool_*`（2 个） | 歧义工具单独不算、带 AI 才算 |
|
||||
| `test_roman_numeral_impersonates_ai` / `test_strong_handles_capital_A_lowercase_L` | 两种 AI 冒充写法 |
|
||||
| `test_rejected_candidates_stay_none` + 同组 | §7 否决清单 |
|
||||
| `test_manju_*`（3 个） | 漫剧词单独成立、红果不成立、题材词不收 |
|
||||
| `test_veo3_beats_a_weak_author_signal` | 强信号压弱信号 |
|
||||
390
PROGRESS.md
Normal file
390
PROGRESS.md
Normal file
@@ -0,0 +1,390 @@
|
||||
# 任务进度 / 踩坑记录
|
||||
|
||||
> 项目介绍和各代码文件说明见 [README.md](README.md)。本文档记录接口踩坑、
|
||||
> 各批筛选与下载进度、已知问题与待办，随批次滚动更新。
|
||||
|
||||
## 抖音视频批量采集 → 音频提取 → COS 上传
|
||||
|
||||
把一批 `douyin.com/video/...` 链接跑成 ASR 可用的数据集：
|
||||
**下载无水印 mp4 → 抽 mp3 → 探时长 → 传腾讯 COS → 出清单（jsonl / xlsx）**。
|
||||
|
||||
全程纯 Python，不依赖第三方解析服务、不依赖 JS 运行时、**不需要登录**。
|
||||
|
||||
---
|
||||
|
||||
## 1. 结论先行：接口怎么走通的
|
||||
|
||||
> 这些结论的验证过程和完整时间线见 [RE.md](RE.md)（逆向全过程复盘）。
|
||||
> 这里只放结论，改代码前先看。
|
||||
|
||||
| 路线 | 状态 | 说明 |
|
||||
|---|---|---|
|
||||
| web `/aweme/v1/web/aweme/detail/` | **要登录** | 匿名请求返回 HTTP 200 **空 body**。真实原因在响应头 `X-Whale-Throughput-Abort-Data`（base64 JSON）：`{"content":"anonymous","name":"强制登录"}`。跟 a_bogus 签名对不对**无关**——伪造的随机 a_bogus 和完全不签名，表现一模一样。 |
|
||||
| App `/aweme/v1/aweme/detail/` | **可用，且不用签名** | 先打 `/service/2/device_register/` 拿 `device_id` / `install_id`，带上就能取详情。**不需要 X-Gorgon、不需要 Cookie、不需要登录。** 这是本项目默认路径。 |
|
||||
| CDN `play_addr` 直链 | 免鉴权 | `aweme.video.play_addr.url_list` 里的 douyinvod 地址没有防盗链，裸 `requests` 直接下。 |
|
||||
| `/aweme/v1/play/?video_id={uri}` | 免鉴权 | 302 跳 CDN，零鉴权。下载走的就是它。 |
|
||||
|
||||
### 两层限流（跑批量必然撞上）
|
||||
|
||||
1. **设备级 detail 配额**：单个 `device_id` 约 **575 次** detail 调用后被限，表现为 HTTP 200 空 body。
|
||||
→ `douyin/batch.py` 的 `DevicePool` 到配额就换设备；用 `generation` 标记当前设备，避免多个 worker 同时发现失败把设备连换好几次。
|
||||
2. **IP 级注册配额**：短时间内注册约 **11~12** 次后，**所有** register host 一律返回 `device_id='0'`。
|
||||
→ `Device.register()` 指数退避重试（默认 5 次，30s 起步翻倍），让批处理**停下来等**，而不是空转着把剩余任务全标成失败。
|
||||
|
||||
第 2 条是踩过的坑：早期没有退避，一次注册受限直接烧掉 678 条任务。
|
||||
|
||||
### 作品拿不到时怎么区分
|
||||
|
||||
看 `filter_detail.filter_reason`：
|
||||
|
||||
- `images_base` → 图文作品（图集），本项目不处理，抛 `NotAVideo`
|
||||
- `status_self_see` / `status_audit_self_see` → 仅作者自见，实际不可用
|
||||
|
||||
---
|
||||
|
||||
## 2. 目录结构
|
||||
|
||||
```
|
||||
├── pipeline.py 【核心】一体化流水线：下载→抽音→上传→记录→删本地
|
||||
├── scripts/
|
||||
│ ├── clean_ai.py 从链接 jsonl 里筛出 AI 相关作品，出链接清单
|
||||
│ ├── jsonl_to_xlsx.py jsonl → xlsx（认 records/records.jsonl 和 records/manifest.jsonl 两种）
|
||||
│ └── export_sample_xlsx.py records.jsonl → 交付用样本 xlsx（支持 --ai-only / --limit）
|
||||
├── tests/
|
||||
│ ├── test_smoke.py 离线冒烟测试（不联网）
|
||||
│ ├── test_meta.py 标题/标签拆分、记录追加的测试
|
||||
│ ├── test_aiclass.py AI 判定的测试（用例全取自真实数据）
|
||||
│ └── test_pipeline.py 流水线端到端测试（ffmpeg 真跑，网络和 COS 打桩）
|
||||
│
|
||||
├── douyin/
|
||||
│ ├── app.py 【核心】设备注册 + App 端 detail
|
||||
│ ├── batch.py 并发批处理、DevicePool 设备轮换
|
||||
│ ├── download.py 带 Range 续传的下载、去重 Registry
|
||||
│ ├── meta.py 标题/标签拆分、records.jsonl 追加写
|
||||
│ ├── aiclass.py 判定作品是不是 AI 相关，分四档，见 §4.5
|
||||
│ ├── cos.py COS 上传 + 上传后大小校验
|
||||
│ ├── audio.py ffmpeg/ffprobe 薄封装：找可执行文件、拼命令、探时长
|
||||
│ ├── urls.py 短链还原、aweme_id 提取
|
||||
│ ├── client.py Aweme/DouyinError 等共用类型 + web 后端请求路径
|
||||
│ ├── signer.py a_bogus 签名（仅 web 后端用，当前跑不起来，见下）
|
||||
│ └── crypto.py SM3 / RC4 / 自定义 Base64（未完成的纯 Python 签名器的积木）
|
||||
│
|
||||
├── data/ 源数据（外部抓来的，clean_ai.py 的输入。删了没法重新筛，别动）
|
||||
│ ├── 抖音链接20260827_40087.jsonl 40087 条
|
||||
│ ├── 抖音ai视频_20260831_87110.jsonl 87110 条（其实是露营帐篷爬取，AI 占 3.3%）
|
||||
│ ├── 抖音ai视频_20260901_57261.jsonl 57261 条（同上，AI 占 2.8%）
|
||||
│ ├── 抖音ai视频_20260902_42391.jsonl 42391 条（9 月 AI 批，AI 占 90.9%）
|
||||
│ ├── 抖音ai视频_20260904_62171.jsonl 62171 条（同上，AI 占 88.3%）
|
||||
│ ├── 抖音ai视频20260908_118469.jsonl 118469 条（同上，AI 占 95.1%）
|
||||
│ ├── 所有作品.csv 4839 条，字段与 jsonl 不同，见 §4.5 第 8 条
|
||||
│
|
||||
├── records/ 流水线产物与断点记录
|
||||
│ ├── records.jsonl 全字段元信息（含 raw），断点续跑靠它
|
||||
│ ├── pipeline_failed.txt 失败清单：aweme_id / 阶段 / 原因（重跑据此续传）
|
||||
│ └── manifest.jsonl 老产物：第一批 5393 条的 3 字段清单，见 §5
|
||||
│
|
||||
├── exports/ 交付/导出产物（xlsx / csv）
|
||||
├── lists/ 各批的链接清单，也是"哪批下了什么"的凭据
|
||||
└── logs/ 各批跑批日志
|
||||
`work/` 是流水线的临时目录，跑的时候自建、传完即删，仓库里不留。
|
||||
|
||||
**web 后端目前是跑不起来的，别把它当后路。** 具体状态：
|
||||
|
||||
- `client.py` 里 `Aweme` / `DouyinClient` / `DouyinError` 这几个类型被 `app.py`、`batch.py`
|
||||
正常用着，**不是死代码**；只有它的 web 请求路径闲置。
|
||||
- `signer.py` 两个后端都不可用：`NodeSigner` 要 `node_signer/sign.js`，**这个目录不在项目里**；
|
||||
`ABogusSigner` 是纯 Python 实现，它自己的文档写着"剩字节拼装顺序待从 ab.js 逆出来"，没写完。
|
||||
- `crypto.py` 的 SM3 / RC4 / 自定义 Base64 对过官方测试向量（`test_smoke.py` 每次都验），
|
||||
但它只是那个**没写完的** `ABogusSigner` 的积木，`signer.py` 并不 import 它。
|
||||
|
||||
所以真要走 web 端，得先补 `node_signer/`（或者把 ab.js 逆完）**再**准备登录 Cookie。
|
||||
App 端被封的话这里没有现成退路，得先做上面这件事。
|
||||
|
||||
**本地媒体文件已全部删除**（`downloads/` 的 5393 个 mp4、`audio/` 的 5393 个 mp3），东西都在 COS 上。旧的分阶段脚本依赖本地文件，要用得先重新下载；`pipeline.py` 不受影响——它本来就是边跑边删。
|
||||
|
||||
输入链接文件 `all_video_links(1).txt` 也已不在，重跑需要重新提供。
|
||||
|
||||
---
|
||||
|
||||
## 3. 环境
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
另需系统里有 `ffmpeg` / `ffprobe`。实测环境：Python 3.11.0 + ffmpeg 2025-07-31 (gyan.dev full build)。
|
||||
|
||||
COS 凭证走环境变量，别写进命令行（会留在 shell history 里）：
|
||||
|
||||
```powershell
|
||||
$env:COS_SECRET_ID = "..."
|
||||
$env:COS_SECRET_KEY = "..."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 一体化流水线（推荐）
|
||||
|
||||
```bash
|
||||
$env:COS_SECRET_ID="..."; $env:COS_SECRET_KEY="..."
|
||||
|
||||
# 先小批试跑，加 --keep-local 不删本地文件，人工看一眼产出
|
||||
python pipeline.py --from-file links.txt --region ap-chongqing --limit 20 --keep-local
|
||||
|
||||
# 确认没问题再放开
|
||||
python pipeline.py --from-file links.txt --region ap-chongqing --workers 8
|
||||
|
||||
# 出 xlsx（--columns full 带上标题、作者、标签、发布时间）
|
||||
python scripts/jsonl_to_xlsx.py -i records/records.jsonl --columns full -o "exports/抖音-四川话.xlsx"
|
||||
|
||||
# 出交付样本 xlsx（--ai-only 只导 AI 档，--limit 限量）
|
||||
python scripts/export_sample_xlsx.py --ai-only -o "exports/AI视频样本.xlsx"
|
||||
```
|
||||
|
||||
每条视频端到端走完 9 步才算完：下载 → 抽 mp3 → 探时长 → 校验 → 传两个对象
|
||||
→ 核对大小 → 写 record → 删本地。**磁盘峰值只有 `workers × (mp4+mp3)`，
|
||||
约 100 MB，与总量无关**——分阶段跑得先摊 53 GB。
|
||||
|
||||
元信息全进 `records/records.jsonl`，一行一条，`raw` 内嵌，字段见下。断点续跑靠它，
|
||||
重跑同一批会跳过已完成的（连下载都不会发生）。COS 上同名对象一律覆盖。
|
||||
`pipeline.py` 的 `--records` / `--failed` 默认就指向 `records/` 目录。
|
||||
|
||||
### 两道闸门
|
||||
|
||||
删本地文件是唯一不可逆的操作，所以卡两道，任一不过就保留文件、记 failed、继续下一条：
|
||||
|
||||
1. **抽取闸门** —— mp3 用 `ffprobe -count_frames` **实际解码**算时长，与 mp4 比对，
|
||||
差超过 `max(1s, 2%)` 判定截断。
|
||||
注意不能用 `format=duration`：它读的是 LAME 的 Xing 头，声明的是原始总帧数。
|
||||
实测把 mp3 截断到 20%，`format=duration` 报的仍是完整的 10.0078s，一字不差；
|
||||
`ffmpeg -f null -` 同样拦不住（rc=0、无 stderr）。只有实解码能发现。
|
||||
2. **上传闸门** —— 不信 `upload_file` 没抛异常，而是 `head_object` 取回
|
||||
`Content-Length` 跟本地字节数比对，两个对象都对上才删。
|
||||
|
||||
### records.jsonl 字段
|
||||
|
||||
```json
|
||||
{"aweme_id":"...","title":"剥掉标签的纯标题","desc":"服务端原样",
|
||||
"tags":["..."],"author":{"nickname":"...","uid":"...","sec_uid":"..."},
|
||||
"create_time":1582635538,"create_time_str":"2020-02-25 21:38:58",
|
||||
"duration_sec":41.702993,"duration_ms_api":41750,
|
||||
"stats":{"digg":0,"comment":0,"collect":0,"share":0,"play":0},
|
||||
"music":{...},"cover_url":"...","video":{"uri":"...","width":1280,"height":720},
|
||||
"cos_mp4":"cos://...","cos_mp3":"cos://...","size_mp4":0,"size_mp3":0,
|
||||
"source_url":"...","fetched_at":0,"raw":{ ...服务端原始 JSON... }}
|
||||
```
|
||||
|
||||
三条要注意的：
|
||||
|
||||
- **`aweme_id` 必须是每行第一个字段**。续跑扫描用正则从行首抠 id，不解析整行
|
||||
（几百 MB 的文件差别是 ~1 秒对十几秒）。重排字段不会报错，只会让启动悄悄变慢。
|
||||
- **`duration_sec` 和 `duration_ms_api` 不是一回事**。前者是 ffprobe 实测，后者是
|
||||
服务端声明值，实测同一条 41703 vs 41750 ms。切分要用 `duration_sec`。
|
||||
- **`tags` 取自 `desc` 的正则全集，不是 `text_extra`**。实测 12 条样本里有 2 条的
|
||||
标签在 `desc` 里有、`text_extra` 里没有，以后者为准会静默丢标签。
|
||||
|
||||
`raw` 内嵌导致单行几十 KB、整个文件十几 GB，所以读取一律逐行流式，
|
||||
`jsonl_to_xlsx.py` / `export_sample_xlsx.py` 已经改成流式了。
|
||||
|
||||
---
|
||||
|
||||
## 4.5 从链接 jsonl 里筛 AI 作品
|
||||
|
||||
> 判定规则的逐条详解（词表选词标准、误收数据、否决清单、维护指南）见 [AI.md](AI.md)。
|
||||
|
||||
`clean_ai.py` 吃的是外部抓来的链接 jsonl（带 desc / hashtags / author / ai 等字段），
|
||||
判定逻辑在 `douyin/aiclass.py`，分四档，取最强的一档：
|
||||
|
||||
| 档 | 依据 | 含义 |
|
||||
|---|---|---|
|
||||
| `platform` | `ai.is_aigc` | 抖音自己打的 AIGC 标 |
|
||||
| `strong` | 文案/标签有 AIGC、AI 生成工具名、ai+创作类词、AI 赛事标签 | 作品声明是 AI 做的 |
|
||||
| `topic` | 只提到 AI 本身（ai 教程、人工智能、大模型、ChatGPT） | 在讲 AI，不一定是 AI 做的 |
|
||||
| `author` | 只有作者昵称或音乐标题带 AI | 最弱，会误收 |
|
||||
|
||||
```bash
|
||||
python scripts/clean_ai.py -i 抖音链接xxx.jsonl --stats # 先看分布
|
||||
python scripts/clean_ai.py -i 抖音链接xxx.jsonl --tier platform,strong -o ai # 出 ai.jsonl + ai.txt
|
||||
python pipeline.py --from-file ai.txt --region ap-chongqing # 直接接流水线
|
||||
```
|
||||
|
||||
**要注意的**（都是从真实数据里挖出来的，改规则前先看 `test_aiclass.py`）：
|
||||
|
||||
1. **`platform` 是唯一能抓到"作者没标但确实是 AI"的信号。** 实测 1744 条被平台标了
|
||||
AIGC，文案标签里一个 ai 字都没有（脑筋急转弯、历史解说、原创动画）。纯关键词过滤
|
||||
会整片漏掉。反过来 9000+ 条有明确 AI 信号却没被平台标——两路互补，都得要。
|
||||
2. **`Al`（大写 A + 小写 L）冒充 AI。** 标签 `抖音al` 出现 129 次，文案里也有 `#Al创作`。
|
||||
不归一化就漏。反过来 `ai` 是 email/detail/again/hair 的常见片段，只能认前后不接
|
||||
字母数字的独立 `ai`。
|
||||
3. **音乐标题默认是 `@<作者昵称>创作的原声`**，40087 条里 32555 条是这个壳。昵称带 AI
|
||||
会连"创作"一起命中，等于让作者昵称从 music 这条路偷偷升到 `strong`——实测 771 条
|
||||
这样混进来。现在先剥壳，剥完只当弱信号（全库剥完只剩 16 条还有命中，本来就没价值）。
|
||||
4. **中文泛词不单独成立。** `动画`/`绘画`/`分身`/`动漫` 必须跟 AI 词同现，否则 830 条
|
||||
`#原创动画` 全被误收。`剪映`(375)/`关键帧`(314)/`蒙版`(203)/`转场` 是剪辑术语，不是 AI。
|
||||
5. **工具名不含 ai 字样的一大票**：即梦 / seedance / libtv / flovaai / 小云雀 /
|
||||
runninghub / oiioii / shotlab / veo3，只能靠名单，名单要跟着平台活动更新。
|
||||
6. **一批工具名本身是常用词，必须要求跟 ai/aigc 同现**（见 `_AMBIG_GEN` / `_AMBIG_CHAT`）。
|
||||
实测误判：豆包 224 条（狗名、豆沙包）、万相 17 条（《龙神万相》）、海螺 13 条
|
||||
（三峰的"海螺"帐篷）、元宝 11 条（316 不锈钢元宝勺）、星流 1 条（"明星流量"子串）、
|
||||
清影 1 条（"起舞弄清影"）。而真命中的**无一例外**都同时带 AI 字样，所以这道门槛
|
||||
不掉召回——旧文件重跑 15218 → 15204，掉的正好是这些。
|
||||
7. **英文工具名要挡词中词，但只挡字母，不挡数字和中文。** `udio` 藏在 `Grass Studio`
|
||||
里 17 次、`flux` 藏在 `Fluxxwave` 曲名里。可是数据里满地都是 `Seedance2.0`、`sora2`、
|
||||
`runninghub0.21`、`libtv创作者`、`flovaai超创`，一刀切加 `` 会丢 3500+ 条真命中。
|
||||
8. **别放开"数字人"/"换脸"/"分身"当独立信号。** 数字人民币含"数字人"、微信分身含"分身"、
|
||||
穿搭视频也说"这套穿搭能换脸"。实测放开只进误判，一条真的都捞不到。
|
||||
同理否掉的还有：lora（撞 valorant）、tripo（撞 tripod 三脚架）、seaweed（海藻）、
|
||||
度加（"亏损幅度加大"）、万彩（"29万彩礼"）、生数（"诞生数副名场面"）、节点（"关键节点"）、
|
||||
抽卡（原神）、捏脸（燕云十六声）、国漫（《剑来》真人制作）、对口型（真人对口型）。
|
||||
这 8 条否决理由都写成了测试，见 `test_rejected_candidates_stay_none`。
|
||||
9. **AI 还有第三种冒充写法：罗马数字。** `#AⅠ漫剧` 用的是 U+2160，跟 `Al` 一个套路。
|
||||
10. **csv 的原声标题多一截 ` - 昵称`**（`@X创作的原声 - X`），剥壳正则要一起覆盖，
|
||||
否则第 3 条那个作者昵称升档的 bug 会在 csv 输入上复发。
|
||||
|
||||
### platform 档要不要求文案佐证
|
||||
|
||||
抖音给 **AI 配音 / AI 封面**的普通带货视频也打 AIGC 标。露营帐篷那两批里，
|
||||
`platform` 档有一半（2045 条）文案和作者昵称都零信号，点开是"四脚伞帐篷带广告位"
|
||||
"抖音私密账号是什么"。`--platform-needs-text` 就是卡这个的，作者昵称带 AI 也算佐证。
|
||||
|
||||
**但这个开关会误伤一类真 AI**：@栖光 那种做 AI 古风美图的，文案只写
|
||||
`#古风 #国风古韵 #意境`，一个 AI 字都不会写。这类只有平台标签能抓到。
|
||||
所以无佐证的那批不是垃圾，得按"是不是带货"再分一刀（见 `lists/platform无佐证_*.txt`）。
|
||||
|
||||
### 抖音的 AIGC 字段：认 risk_infos.type，别认 is_aigc_media
|
||||
|
||||
`raw.risk_infos.type` 是数字码，一一对应：
|
||||
|
||||
| type | content | 含义 |
|
||||
|---|---|---|
|
||||
| 44 | 作者声明：内容由 AI 生成 | 作者主动声明，19946 条 |
|
||||
| 47 | 疑似使用了 AI 生成技术，请谨慎甄别 | 平台检测，措辞不确定，515 条 |
|
||||
| 76 / 328 | 作品含 AI 生成内容 | 平台判定，448 条 |
|
||||
|
||||
**这几个才是 AI。**下面这些是别的免责声明，跟 AI 无关：
|
||||
|
||||
| type | content | 条数 |
|
||||
|---|---|---|
|
||||
| 45 | 作者声明：虚构演绎，仅供娱乐 | 868 |
|
||||
| 336 / 351 | 个人观点，仅供参考 | 111 |
|
||||
| 54 | 作者声明：内容来源于网络 | 13 |
|
||||
| 335 | 作者声明：内容含营销推广信息 | 10 |
|
||||
|
||||
**已知 bug**：爬虫那层把「`risk_infos` 有内容」就当 `is_aigc=true`，把上面这些
|
||||
免责声明也算进 `platform` 档了，全库误算 **1029 条**。修法见 §7。
|
||||
|
||||
另外 `raw.is_aigc_media` 全库 24611 条**全是 `false`**，名字最像却完全没用，别信。
|
||||
|
||||
### 各批筛选结果
|
||||
|
||||
| 源文件 | 总条数 | AI 占比 | 出的清单 |
|
||||
|---|---|---|---|
|
||||
| 抖音链接20260827_40087.jsonl | 40087 | 39.1% | `ai_videos.txt` 15218 |
|
||||
| 抖音ai视频_20260831_87110.jsonl | 87110 | **3.3%** | `ai_87110_严格.txt` 1366 |
|
||||
| 抖音ai视频_20260901_57261.jsonl | 57261 | **2.8%** | `ai_57261.txt` 776 |
|
||||
| 抖音ai视频_20260902_42391.jsonl | 42391 | 90.9% | `ai_42391.txt` 37703 |
|
||||
| 抖音ai视频_20260904_62171.jsonl | 62171 | 88.3% | `ai_62171.txt` 52360 |
|
||||
| 抖音ai视频20260908_118469.jsonl | 118469 | 95.1% | `ai_118469.txt` 106447 |
|
||||
| 所有作品.csv | 4839 | 未筛 | `csv_links.txt` 4839（全量下的） |
|
||||
|
||||
前两个文件（87110 / 57261）名字叫"ai视频"，**实际是露营帐篷带货爬取**
|
||||
（标签 TOP：户外露营 10568、帐篷 9346），AI 只占 3% 左右。9 月的三批
|
||||
（42391 / 62171 / 118469）才是真 AI 池，且 platform 档占绝对大头——这三批
|
||||
`allow_download=false` 里 `aigc_forbidden` 多达 2.8 万 / 3.7 万 / 7.5 万条，
|
||||
但 App 端接口不受此限（见下）。拿到新文件先跑 `--stats` 看一眼，别信文件名。
|
||||
|
||||
`aigc_forbidden` 那道限制**只作用于网页端，App 端接口不受影响**——已用 20 条探针
|
||||
验过，9 条 web 侧 `allow_download=false` 的全都正常下下来了。不用 `--exclude-forbidden`。
|
||||
|
||||
### lists/ 里各清单的关系
|
||||
|
||||
| 清单 | 条数 | 是什么 |
|
||||
|---|---|---|
|
||||
| `all_video_links(3).txt` | 3557 | 第一批输入 |
|
||||
| `ai_videos.txt` | 15218 | 40087 批筛出的 AI |
|
||||
| `csv_links.txt` | 4839 | 所有作品.csv 全量（未筛） |
|
||||
| `ai_87110_严格.txt` / `ai_57261.txt` | 1366 / 776 | 露营两批筛出的 AI |
|
||||
| `ai_42391.txt` / `ai_62171.txt` / `ai_118469.txt` | 37703 / 52360 / 106447 | 9 月三批筛出的 AI（platform+strong 档） |
|
||||
| `ai_合并待下载.txt` | 66411 | 62171 全量 + 42391 部分 + 118469 无，合并去重后的跑批输入 |
|
||||
| `ai_待下载.txt` | 2611 | 旧池剩余（platform 无佐证两档 + 零星），已下过 99%，基本作废 |
|
||||
| `platform无佐证_非带货.txt` | 776 | AI 古风/漫剧/语录图文 |
|
||||
| `platform无佐证_带货.txt` | 1331 | 待下（不建议）：帐篷等实拍带货 |
|
||||
|
||||
截至 2026-09-29，9 月三批的下载进度都是 97%+（118469 批 103702/106447、
|
||||
42391 批 37254/37703、62171 批 50989/52360），三批合计还有约 4565 条
|
||||
不在 records.jsonl 里（含图文、已删除等下不动的）。
|
||||
|
||||
## 5. 自检
|
||||
|
||||
```bash
|
||||
python tests/test_smoke.py # 离线冒烟：SM3/RC4/base64 自测、id 提取、Registry
|
||||
python tests/test_meta.py # 标题/标签拆分、records.jsonl 追加与续跑
|
||||
python tests/test_aiclass.py # AI 判定（用例全取自真实数据）
|
||||
python tests/test_pipeline.py # 端到端：ffmpeg 真跑，网络和 COS 打桩
|
||||
```
|
||||
|
||||
### 老产物 manifest.jsonl
|
||||
|
||||
`records/manifest.jsonl` 是第一批 5393 条四川话数据的清单（`cos_mp4` / `cos_mp3` / `duration` 三字段），
|
||||
`exports/抖音-四川话.xlsx` 就是它的 xlsx 版。**它和 `records/records.jsonl` 零重合**——那 5393 条不在
|
||||
`records/records.jsonl` 里，所以这个文件是这批数据在 COS 上的唯一记录，别删。
|
||||
|
||||
生成它的那套分阶段脚本（`main.py` / `extract_audio.py` / `probe_durations.py` /
|
||||
`upload_cos.py`）已经删掉，功能全被 `pipeline.py` 覆盖了。这个项目没有 git，删了就是真没了——
|
||||
但那四个脚本做的事 `pipeline.py` 一条命令全包含，唯一不能复现的是 manifest.jsonl 的三字段老格式。
|
||||
其中 `probe_durations.probe()` 被 `pipeline.py` 用着，已搬进 `douyin/audio.py`。
|
||||
|
||||
---
|
||||
|
||||
## 6. 跑批结果
|
||||
|
||||
`records/records.jsonl` 累计 **219144 条 / 8676.02 小时 / 2483.43 GB（mp4）**，
|
||||
全部在 `cos://asr-dig-data-1253181912/0005/zh/`。
|
||||
|
||||
9 月三批 AI 池（42391 / 62171 / 118469）已基本下完，各批进度 97%+。
|
||||
早期按 AI 判档的拆分（platform 有佐证 5423 条 / strong 9036 条等）是 9/1 时点的
|
||||
快照，那之后 records.jsonl 又进了十几万科目的 9 月 AI 批，旧数字只作历史参考。
|
||||
|
||||
### COS 前缀 0005/zh/ 混了八批
|
||||
|
||||
```
|
||||
5393 四川话 只在 manifest.jsonl 里，不在 records.jsonl
|
||||
3517 第一批链接
|
||||
14884 AI 筛选批（40087 源）
|
||||
4813 csv 全量批
|
||||
2093 87110 + 57261 两批
|
||||
37254 42391 批
|
||||
50989 62171 批
|
||||
103702 118469 批
|
||||
```
|
||||
|
||||
**按前缀取数会一次拿到全部**，要区分只能走 `records.jsonl`。
|
||||
|
||||
## 7. 待办 / 已知问题
|
||||
|
||||
**该修的**
|
||||
|
||||
- **`is_aigc` 把免责声明当成了 AIGC**，全库误算 1029 条（见 §4.5）。爬虫那层应改成
|
||||
`risk_infos.type in {44, 47, 76, 328}`，并把 `type` 一起存进记录——这样才能区分
|
||||
「作者声明」(44) 和「平台检测」(47)，两者可信度不同。改完 `platform` 档会掉约 1029 条，
|
||||
真 AI 一条不少（那些走 strong）。
|
||||
- **图文作品（`aweme_type` 68/61/55/163）不支持**，需要 `images[]` 分支。这是目前失败
|
||||
的绝对大头：87110 批 38 条失败里 33 条是图文，AI 池里累计 241 条下不了。
|
||||
图文在按 aweme_id 排序的清单里**扎堆在头部**（早期 AI 内容多是 Midjourney 出图），
|
||||
所以跑批前几百条失败率会飙到 15%，后面掉到 2-3%，别被吓到。
|
||||
|
||||
**已知取舍**
|
||||
|
||||
- 最长一条 3398.988 秒（约 57 分钟），若 ASR 侧对单条时长有上限需要另行切分
|
||||
- `split_desc` 在标签名以 `#名字` 形式也出现在正文里时会连正文一起删（见
|
||||
`test_split_repeated_tagname_in_body_is_lossy`）；服务端给了可信偏移时不会
|
||||
- 设备配额轮换的触发条件是**每 550 次 detail**，但到底是按 IP 还是按 UA/设备指纹限流
|
||||
一直没验证过。实测跑到第 3 次轮换都正常，暂时不用管
|
||||
- `topic` / `author` 两档从没进过下载清单。`author` 档现在 14 条**全部来自音乐标题**
|
||||
（`【AI配文风景滤镜】的原声` 这类抖音模板），作者昵称一条都没有——因为凡是作者叫 AI 的，
|
||||
作品本身也都标了 AI，昵称这条线提供不了增量，它的价值只是给 `platform` 档做佐证
|
||||
|
||||
**安全**
|
||||
|
||||
- 聊天记录里出现过一组 COS 密钥，**建议轮换**。凭证只能走 `COS_SECRET_ID` /
|
||||
`COS_SECRET_KEY` 环境变量，`CosUploader` 不读命令行参数
|
||||
246
RE.md
Normal file
246
RE.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# 抖音接口逆向全过程
|
||||
|
||||
> 本文按时间顺序复盘这个项目把「批量下载抖音无水印视频」从 0 走通的全过程。
|
||||
> 每条结论都标注了验证方式：【实测】= 打过真实接口，【推断】= 有旁证但没做对照实验。
|
||||
> 对应的代码在 `douyin/` 包里，结论速查表在 [PROGRESS.md](PROGRESS.md) §1。
|
||||
|
||||
---
|
||||
|
||||
## 0. 目标与约束
|
||||
|
||||
把外部抓来的几十万条抖音链接跑成 ASR 数据集：下载无水印 mp4 → 抽 mp3 → 传 COS。
|
||||
三条硬约束：**不登录**（没有号，批量场景号必死）、**不依赖第三方解析服务**（不稳定、有泄露风险）、
|
||||
**不依赖 JS 运行时**（部署环境只有 Python + ffmpeg）。
|
||||
|
||||
---
|
||||
|
||||
## 1. 第一阶段：Web 端——撞在「强制登录」上
|
||||
|
||||
最先试的自然是浏览器在用的 web 接口 `www.douyin.com/aweme/v1/web/aweme/detail/`。
|
||||
这套接口的门槛众所周知：请求参数要带 `a_bogus` 签名（还有 `msToken`、`ttwid`、
|
||||
`s_v_web_id`、`odin_tt` 一串指纹 Cookie），所以第一步是先把签名搞对。
|
||||
|
||||
### 1.1 空 body 的两种成因，花了很久才分开
|
||||
|
||||
匿名请求 detail 的表现是 **HTTP 200 + 空 body**，这有歧义：
|
||||
|
||||
1. **session 没预热**：新 session 的第一个接口请求必然空，第二个才通。
|
||||
解法很猥琐但有效——初始化时先打一发 `/aweme/v1/web/aweme/related/` 把这个
|
||||
「哑弹」消耗掉（`_warm_up()`，见 `douyin/client.py`）。
|
||||
2. **服务端主动拦截**：响应头 `X-Whale-Throughput-Abort-Data` 里有 base64 JSON，
|
||||
解码出来是拦截原因（`decode_abort()`）。
|
||||
|
||||
分不清这两种就会对着可重试的瞬时态疯狂重试，或者把硬门槛当网络抖动。
|
||||
|
||||
### 1.2 关键对照实验：不签名 vs 假签名，表现一模一样
|
||||
|
||||
把签名字段换成完全随机的 garbage a_bogus，和干脆不签名，两种请求的响应**逐字节一致**
|
||||
（都是 200 空 body + `强制登录` 拦截头）。【实测】
|
||||
|
||||
这说明：**匿名拿不到 detail 跟签名对不对无关**。继续抠 a_bogus 算法救不了这条路。
|
||||
|
||||
### 1.3 绕路尝试：related 接口
|
||||
|
||||
`/aweme/v1/web/aweme/related/` 匿名可用、返回完整 `play_addr`——但只含**相关视频**，
|
||||
不含目标视频自身。【实测】此路不通，只白捡了一个预热接口。
|
||||
|
||||
### 1.4 Web 端留档的正确姿势
|
||||
|
||||
web 路径没删干净而是留了 `client.py` / `signer.py` / `crypto.py`，因为：
|
||||
- `Aweme` / `DouyinError` 这些类型被 App 端正常复用；
|
||||
- 几个坑的记录有复用价值（UA 必须与签名 UA 一致、`browser_version` 参数要跟着对齐、
|
||||
cookie 预热、拦截头解码）；
|
||||
- 万一 App 端被封，web 是唯一退路——但要先补签名（见 §2）。
|
||||
|
||||
---
|
||||
|
||||
## 2. 插曲：a_bogus 纯 Python 化，逆了一半
|
||||
|
||||
既然 web 端迟早要用（当时这么想），顺手逆了 a_bogus。它的底层是三样公开原语：
|
||||
|
||||
| 原语 | 说明 | 状态 |
|
||||
|---|---|---|
|
||||
| SM3 | 国标哈希（GB/T 32905-2016） | ✅ 对照官方测试向量验证通过 |
|
||||
| RC4 | 标准流密码 | ✅ 实现并验证 |
|
||||
| 自定义 Base64 | 标准 Base64 换了套字符码表 | ✅ 逆出码表 |
|
||||
|
||||
这三样现在还在 `douyin/crypto.py` 里，文件底部自带官方向量自检，`test_smoke.py`
|
||||
每次跑都会验。**卡住的是最后一层**：三个原语的输出按什么顺序拼进 a_bogus 的
|
||||
字节结构，需要对照混淆过的 `ab.js` 逆出来，没逆完。`ABogusSigner` 文档里原话就是
|
||||
「剩字节拼装顺序待从 ab.js 逆出来」。
|
||||
|
||||
同时期的 `NodeSigner` 路线（本地 node_signer 跑真实 webmssdk）实测能被抖音接受，
|
||||
但 `node_signer/` 目录不在本项目里。所以 **web 端当前没有可用退路**，这是已知风险。
|
||||
|
||||
---
|
||||
|
||||
## 3. 第二阶段：转向 App 端——发现「免签名」路线
|
||||
|
||||
Web 端被判死刑后转向 App 端。App 接口传统上被认为要 X-Gorgon / X-Khronos 签名，
|
||||
这恰恰是「不依赖 JS 运行时」最难啃的方向。但实测发现**新版本的 App 端详情接口
|
||||
根本不验签**。
|
||||
|
||||
### 3.1 最早的误判：伪造 device_id 导致的空 body
|
||||
|
||||
第一次打 App 端 detail 也是空 body，一度以为「果然还是要签名」。
|
||||
换用**注册得来的真实** `device_id` / `install_id` 之后立刻正常。【实测】
|
||||
|
||||
教训：**空 body 不代表要签名**。App 端的门槛是设备身份的真实性，不是签名。
|
||||
|
||||
### 3.2 设备注册：免签名、免 Cookie、免登录
|
||||
|
||||
链路只有两步（`douyin/app.py`）：
|
||||
|
||||
```
|
||||
POST https://{log.snssdk.com|api.amemv.com|...}/service/2/device_register/
|
||||
-> device_id / install_id / openudid / cdid / clientudid
|
||||
|
||||
GET https://aweme.snssdk.com/aweme/v1/aweme/detail/?device_id=...&iid=...
|
||||
-> 完整 aweme_detail
|
||||
```
|
||||
|
||||
- 注册是引导调用，**不需要任何签名**，四个 host 互为备份；
|
||||
- 注册一次身份长期有效，落盘缓存（`.device.json`）；
|
||||
- 从此 detail 随便打，不需要 X-Gorgon、不需要 Cookie、不需要登录。
|
||||
|
||||
#### device_id 是怎么来的：我们自己只造指纹，id 是服务器发的
|
||||
|
||||
`device_id` 不是客户端算出来的——和真 App 首次启动一样，是**服务器在注册响应里
|
||||
分配的**。我们这边生成的是四样客户端指纹（`Device._register_once()`）：
|
||||
|
||||
| 字段 | 生成方式 | 作用 |
|
||||
|---|---|---|
|
||||
| `openudid` | 16 位随机 hex | 设备级指纹 |
|
||||
| `cdid` | 随机 UUID | 设备级指纹（detail 请求也带） |
|
||||
| `clientudid` | 随机 UUID | 会话级指纹 |
|
||||
| 机型档案 | 从 8 款机型池随机抽（型号/品牌/Android 版本/分辨率/dpi） | 决定 UA 和上报参数，注册后跟着设备走 |
|
||||
|
||||
连同伪装成抖音 9.9.9（`com.ss.android.ugc.aweme`，APP_ID=1128）的完整上报体
|
||||
（`magic_tag: ss_app_log`，header 里 `device_id=0, install_id=0` 表示「我是新来的，
|
||||
请发身份」）POST 给注册接口。响应里 `device_id_str` / `install_id_str` 就是分配来的
|
||||
身份，四个 host 按顺序试，拿到第一个非 `'0'` 的就算注册成功。
|
||||
|
||||
服务器拒绝时的行为也是信息：返回 `device_id='0'`（不是错误码），换全新指纹照样被拒、
|
||||
四个 host 同时被拒、约 10 分钟后自行恢复——说明这是**临时限流，且不按指纹分桶、
|
||||
不按单个 host 分桶**。按 IP 还是按 UA/机型特征分桶没验证过（缺第二个出口 IP）。
|
||||
|
||||
### 3.3 设备档案的自洽性
|
||||
|
||||
注册请求里上报机型（device_type / brand / os_version / resolution），后续 detail
|
||||
请求的 UA 和这些字段必须自洽——「注册报小米、请求 UA 是三星」本身就是可疑特征。
|
||||
所以 `Device` 把机型档案跟着设备身份一起走。【推断：基于风控常识，未做对照实验】
|
||||
|
||||
### 3.4 机型随机化：成本极低的两用对冲
|
||||
|
||||
所有注册请求以前在机型维度上完全一致。注册受限到底是按 IP 还是按「UA+机型」
|
||||
特征分桶，**没验证过**（本机没有第二个出口 IP，做不了对照实验）。
|
||||
把机型做成 8 款的随机池：如果是特征分桶它能缓解，如果是纯 IP 它无害。【对冲】
|
||||
|
||||
---
|
||||
|
||||
## 4. 第三阶段：下载链路——比想象中裸
|
||||
|
||||
拿到 `aweme_detail` 之后找下载地址，两个发现都免鉴权：
|
||||
|
||||
1. **`aweme.video.play_addr.url_list`** 里的 `douyinvod.com` CDN 直链没有防盗链，
|
||||
裸 `requests` 直接下，Referer 都不查。【实测】
|
||||
2. **`/aweme/v1/play/?video_id={uri}`** 更干净：免鉴权 302 跳到 CDN。项目下载走的就是它。
|
||||
|
||||
CDN 支持 Range，顺势做了断点续传（`douyin/download.py`），落盘文件名直接取
|
||||
aweme_id——「是否已下载」看文件在不在，不依赖任何去重记录，记录丢了也能无损续跑。
|
||||
|
||||
---
|
||||
|
||||
## 5. 第四阶段：限流——两层，性质不同
|
||||
|
||||
批量一上量必然撞限流。两层限流的表现和处置完全不同：
|
||||
|
||||
### 5.1 设备级 detail 配额【实测】
|
||||
|
||||
- 单个 `device_id` 约 **575 次** detail 调用后被限，表现是接口一直回 200 空 body；
|
||||
- **同一 IP 换个新注册的设备立刻恢复** → 配额是设备级而非 IP 级；
|
||||
- 注册本身免签名、成本极低 → 按量轮换即可，`DevicePool` 每 400 次主动换（留安全余量）。
|
||||
|
||||
工程上有个细节：多 worker 并发下可能同时发现「设备死了」。`DevicePool` 用
|
||||
`generation` 标记当前设备，worker 拿设备时记下代数，发现被限时只有代数还对得上的
|
||||
才允许触发轮换——避免一个事件把设备连换好几次。
|
||||
|
||||
### 5.2 IP 级注册配额【实测】
|
||||
|
||||
- 短时间内注册约 **11~12 次**后，**所有** register host 一律返回 `device_id='0'`；
|
||||
- 这层是 IP 级的，换设备没用，只能**等**。`Device.register()` 指数退避重试
|
||||
（默认 5 次，30 秒起步翻倍），让批处理停下来等，而不是空转着把剩余任务全标失败。
|
||||
|
||||
**这是本项目烧过真金白银的坑**：早期没有退避，一次注册受限直接烧掉 678 条任务。
|
||||
|
||||
### 5.3 未解之谜
|
||||
|
||||
detail 配额轮换后是否真的按「IP 还是 UA/设备指纹」限流，一直没验证过
|
||||
（缺第二个出口 IP）。实测跑到第 3 次轮换都正常，暂搁置。
|
||||
|
||||
---
|
||||
|
||||
## 6. 第五阶段：数据语义层的逆向
|
||||
|
||||
接口通了之后，剩下的坑全在「字段怎么读」上。这些是从几十万条真实数据里统计出来的：
|
||||
|
||||
### 6.1 AIGC 判定：认 `risk_infos.type`，别认 `is_aigc_media`
|
||||
|
||||
`raw.is_aigc_media` 全库 24611 条**全是 false**——名字最像，完全没用。
|
||||
真正的信号是 `risk_infos.type` 数字码：
|
||||
|
||||
| type | 含义 |
|
||||
|---|---|
|
||||
| 44 | 作者声明：内容由 AI 生成（最可信，19946 条） |
|
||||
| 47 | 疑似使用 AI 生成技术（平台检测，措辞存疑，515 条） |
|
||||
| 76 / 328 | 作品含 AI 生成内容（平台判定，448 条） |
|
||||
| 45 / 336 / 351 / 54 / 335 | 虚构演绎 / 个人观点 / 来源于网络 / 营销推广——**与 AI 无关的免责声明** |
|
||||
|
||||
已知 bug：爬虫层把「risk_infos 有内容」就当 `is_aigc=true`，把免责声明也算进
|
||||
AIGC，全库误算 1029 条。修法（存 `type` 字段、按 `{44,47,76,328}` 判）记在
|
||||
PROGRESS.md §7 待办。
|
||||
|
||||
### 6.2 不可下载作品的分类
|
||||
|
||||
`filter_detail.filter_reason` 能区分失败性质：
|
||||
- `images_base` → 图文作品（图集），不是视频，抛 `NotAVideo`（当前失败的绝对大头，
|
||||
AI 池累计 241 条；且 aweme_id 排序下图文扎堆在头部，跑批前几百条失败率会飙到 15%）
|
||||
- `status_self_see` / `status_audit_self_see` → 仅作者自见，实际不可用
|
||||
|
||||
### 6.3 `aigc_forbidden` 只管 web 端
|
||||
|
||||
`allow_download=false` 的 `aigc_forbidden` 原因在 web 端会拦下载，但
|
||||
**App 端接口完全不受影响**。用 20 条探针实测：9 条 web 侧禁下的全部正常下下来了。
|
||||
所以不需要 `--exclude-forbidden`，禁下名单不是真禁下。
|
||||
|
||||
---
|
||||
|
||||
## 7. 最终架构
|
||||
|
||||
逆向成果固化成一条 9 步流水线（`pipeline.py`），两道「删除闸门」保证
|
||||
「传完就删本地」不失数据：
|
||||
|
||||
```
|
||||
fetch_detail(App端免签) → 下载mp4(302直链+Range续传) → ffmpeg抽mp3
|
||||
→ ffprobe实解码探时长 → 闸门①: mp3时长≈mp4时长?
|
||||
→ 上传mp4/mp3 → head_object核对大小 → 闸门②: 字节数一致?
|
||||
→ 写record → 删本地
|
||||
```
|
||||
|
||||
任何一步失败：保留本地、记 failed、继续下一条。断点续跑靠 `records/records.jsonl`
|
||||
（行首 aweme_id 契约，正则抠 id 不解析整行，十几 GB 文件启动扫描 ~1 秒）。
|
||||
|
||||
---
|
||||
|
||||
## 8. 结论速查（给后来改代码的人）
|
||||
|
||||
| 问题 | 答案 |
|
||||
|---|---|
|
||||
| web detail 匿名能拿吗 | 不能，强制登录，与签名无关【实测】 |
|
||||
| App detail 要签名吗 | 不要。要的是**注册来的真实设备身份**【实测】 |
|
||||
| 设备哪里来 | `/service/2/device_register/`，免签名，身份落盘复用 |
|
||||
| 下载走哪 | `/aweme/v1/play/?video_id=` 302 跳 CDN，免鉴权，支持 Range |
|
||||
| 什么时候被限流 | 设备 ~575 次 detail（换设备）；IP ~12 次注册（只能退避等）【实测】 |
|
||||
| 空 body 怎么读 | 先看 `X-Whale-Throughput-Abort-Data`，再怀疑 session 没预热 |
|
||||
| AIGC 看哪个字段 | `risk_infos.type ∈ {44,47,76,328}`；`is_aigc_media` 全库皆 false |
|
||||
| 退路 | web 端要先补 a_bogus（SM3/RC4/Base64 已通，字节拼装未逆完）+ 登录 Cookie |
|
||||
260
README.md
Normal file
260
README.md
Normal file
@@ -0,0 +1,260 @@
|
||||
# 抖音视频批量采集 → 音频提取 → COS 上传
|
||||
|
||||
把一批 `douyin.com/video/...` 链接跑成 ASR 可用的数据集：
|
||||
**下载无水印 mp4 → 抽 mp3 → 探时长 → 传腾讯 COS → 出清单（jsonl / xlsx）**。
|
||||
|
||||
全程纯 Python，不依赖第三方解析服务、不依赖 JS 运行时、**不需要登录**。
|
||||
核心走 App 端接口：先注册设备拿 `device_id`，用它取作品详情，CDN 直链下载。
|
||||
|
||||
任务进度、各批下载情况见 [PROGRESS.md](PROGRESS.md)；
|
||||
接口是怎么逆出来的（web 端为何弃用、App 端免签名链路、两层限流）见 [RE.md](RE.md)。
|
||||
|
||||
---
|
||||
|
||||
## 快速开始
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt # 另需系统装有 ffmpeg / ffprobe
|
||||
```
|
||||
|
||||
```powershell
|
||||
$env:COS_SECRET_ID="..."; $env:COS_SECRET_KEY="..." # COS 凭证只走环境变量
|
||||
|
||||
# 先小批试跑，加 --keep-local 不删本地文件，人工看一眼产出
|
||||
python pipeline.py --from-file lists/ai_62171.txt --region ap-chongqing --limit 20 --keep-local
|
||||
|
||||
# 确认没问题再放开（断点续跑：重启自动跳过已完成的）
|
||||
python pipeline.py --from-file lists/ai_62171.txt --region ap-chongqing --workers 8
|
||||
|
||||
# 自检
|
||||
python tests/test_smoke.py && python tests/test_meta.py && python tests/test_aiclass.py && python tests/test_pipeline.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 流水线详解
|
||||
|
||||
`pipeline.py` 是项目核心：一条命令完成 下载 → 抽音 → 上传 → 记录 → 删本地。
|
||||
和分阶段跑（全下完再全抽完再全传完）的本质区别是磁盘占用：分阶段要把整批
|
||||
mp4 摊在盘上（上一批 53 GB），流水线每条走完即删，**峰值只有
|
||||
`workers × (mp4+mp3)` ≈ 100 MB，与总量无关**。
|
||||
|
||||
### 每条视频的 9 步
|
||||
|
||||
```
|
||||
1. fetch_detail DevicePool 取设备 → App 端 detail → 元信息 + raw
|
||||
2. 下载 mp4 play_addr 直链 / /aweme/v1/play/ 302，Range 续传
|
||||
3. 抽 mp3 ffmpeg 转码（CPU 密集，信号量限到核数）
|
||||
4. 探时长 ffprobe 读 mp4；mp3 用 -count_frames 实解码
|
||||
5. 校验时长 mp3 实解时长 ≈ mp4 时长？ ← 抽取闸门
|
||||
6. 上传 mp4 COS，覆盖同名
|
||||
7. 上传 mp3 同上
|
||||
8. 核对大小 put() 内部 head_object 比对字节数 ← 删除闸门
|
||||
9. 落库 + 删本地 写 records.jsonl，然后才 unlink mp4/mp3
|
||||
```
|
||||
|
||||
**为什么第 4 步必须实解码**：LAME 写的 Xing 头声明的是原始总帧数，
|
||||
`ffprobe format=duration` 直接信它不解码——实测把 mp3 截断到 20%，
|
||||
它报的还是完整的 10.0078s，`ffmpeg -f null -` 同样拦不住（rc=0 无报错）。
|
||||
只有 `-count_frames` 真解码能发现截断。代价可忽略：120s 的 mp3 是
|
||||
0.109s 对 0.053s。两道闸门任一不过：保留本地文件、记 failed、继续下一条。
|
||||
|
||||
### 启动阶段：四件事做完才开跑
|
||||
|
||||
1. **解析输入**：`--from-file` 或命令行链接，经 `urls.py` 一律还原成
|
||||
aweme_id，顺手去重、剔除已完成的（`aid in log`）、统计无法解析的。
|
||||
2. **切段**（`plan_batch`，默认 `--max-items 12000`）：单进程跑到 ~1.5 万条，
|
||||
Windows 桌面堆被几万个 ffmpeg/ffprobe 子进程耗光，之后 CreateProcess
|
||||
一律失败（rc=3221225794）。症状好认：完成数冻结、失败数猛涨、work/
|
||||
暴涨。是**累积量**问题不是并发问题。段跑完 `os.execv` 重启自己，
|
||||
断点记录不丢。
|
||||
3. **清扫 work/**（`sweep_work_dir`）：删两类残留——aweme_id 已在
|
||||
records.jsonl 里的（早传上 COS 了）、`.part` 半截下载。**不碰失败项的
|
||||
文件**：上传失败/抽音截断是故意留着的，重跑省一次下载。长批跑起来后
|
||||
每 5 分钟还会自动清扫一次（光靠开跑前扫一次，跑到一半磁盘就满）。
|
||||
4. **预热设备池**（`DevicePool`）：惰性注册，先主动要一次设备。开跑就
|
||||
注册不上 = IP 还在冷却期，按退避策略等待并重启（见下），而不是让每个
|
||||
worker 各自撞一遍把整批标失败。
|
||||
|
||||
### 并发模型：两类资源分开限
|
||||
|
||||
| 资源 | 限流方式 | 默认值 |
|
||||
|---|---|---|
|
||||
| 网络（下载/上传/detail） | `--workers` 线程池 | 8 |
|
||||
| CPU（ffmpeg/ffprobe） | `ffmpeg-workers` 信号量 | CPU 核数 |
|
||||
|
||||
下载用裸 `okhttp/3.12.13` UA 的 thread-local session（CDN 不查设备头）；
|
||||
每个 worker 的 detail 请求走 `DevicePool.current()` 共享设备身份。
|
||||
|
||||
### 限流的两级应对
|
||||
|
||||
**设备级 detail 配额**（~575 次/设备，默认 550 就换）：detail 返回「空 body」
|
||||
即判定设备被限，`pool.report_dead(gen)` 换设备重试一次。`generation` 标记
|
||||
保证多 worker 同时发现失败时只换一轮。
|
||||
|
||||
**IP 级注册配额**（~12 次注册后所有 host 返回 `device_id='0'`）：换设备没用，
|
||||
只能等冷却。此时拉起全局 `abort`：后续任务直接跳过不计失败，当前批收尾后按
|
||||
`next_abort_wait` 指数退避（默认 30 分钟起步、连撞翻倍、上限 4 小时）等待并
|
||||
`os.execv` 重启续跑。连撞计数存在环境变量 `_DY_ABORT_STREAK` 里（execv 换进程
|
||||
映像后靠它记住状态）；但凡这一轮出过活就说明是跑着跑着才触发的限流，下次从
|
||||
底价重等。**早期没有这套机制，一次注册受限直接烧掉 678 条任务。**
|
||||
|
||||
### 失败处理
|
||||
|
||||
- `StageError` 带阶段名（detail / download / audio / probe / upload），
|
||||
连同原因追加进 `records/pipeline_failed.txt`，一行一条：`id\t阶段\t原因`。
|
||||
- 顶层还有 `except Exception` 兜底记 `unknown`，**一条炸不掉整批**。
|
||||
- 失败项的本地文件**故意保留**：重跑同一命令即可续传（已完成的自动跳过，
|
||||
失败项重新走全流程）。跑完退出码：0 全成，4 有失败，3 开跑即注册失败，
|
||||
5 撞上 IP 限流停下等手动重跑。
|
||||
|
||||
### 进度报告
|
||||
|
||||
后台线程每 15 秒打一行：`[已完成+失败/总数] 完成 失败 | GB 上传 | 条/分 | 剩约 N 分`，
|
||||
兼做周期清扫的节拍器。
|
||||
|
||||
### 入口脚本
|
||||
|
||||
`pipeline.py` 在根目录（主入口），其余脚本在 `scripts/`，测试在 `tests/`。
|
||||
|
||||
| 文件 | 作用 |
|
||||
|---|---|
|
||||
| `pipeline.py` | **核心**。一体化流水线，逐步过程、闸门与限流应对见上面「流水线详解」 |
|
||||
| `scripts/clean_ai.py` | 从外部抓来的链接 jsonl 里筛出 AI 相关作品，产出链接清单直接喂 pipeline。判定分四档（platform / strong / topic / author），先 `--stats` 看分布再 `-o` 出清单 |
|
||||
| `scripts/jsonl_to_xlsx.py` | 把 `records/records.jsonl`（或老的 `manifest.jsonl`）清洗导出成交付用 xlsx：剔坏行、剔非法时长/前缀、按 mp4 地址去重。流式读，raw 不进内存 |
|
||||
| `scripts/export_sample_xlsx.py` | 按甲方字段表把 records.jsonl 导成样本 xlsx（样本 ID / 标题 / 生成方式 / 工具模型 / 分辨率 / 发布时间 / AI 标识等），支持 `--ai-only`、`--limit` |
|
||||
|
||||
### 测试（`tests/`）
|
||||
|
||||
| 文件 | 覆盖 |
|
||||
|---|---|
|
||||
| `test_smoke.py` | 离线冒烟：crypto 官方向量自检、链接解析、文件名清洗、去重 Registry |
|
||||
| `test_meta.py` | 标题/标签拆分（split_desc）、records.jsonl 追加写与幂等 |
|
||||
| `test_aiclass.py` | AI 判定，用例全部取自真实数据（含 `Al` 冒充、罗马数字 Ⅰ 等冒充写法） |
|
||||
| `test_pipeline.py` | 流水线端到端：ffmpeg 真跑，网络和 COS 打桩。覆盖两道删除闸门（时长不符 / 上传大小不符必须保留本地） |
|
||||
|
||||
### `douyin/` 包
|
||||
|
||||
| 文件 | 作用 |
|
||||
|---|---|
|
||||
| `app.py` | **核心**。App 端接口客户端：`/service/2/device_register/` 注册设备拿 `device_id`/`install_id`，带它打 `/aweme/v1/aweme/detail/` 取详情。无签名、无 Cookie、无登录。设备级 detail 配额约 575 次，超限换设备 |
|
||||
| `batch.py` | 并发批处理骨架：ThreadPoolExecutor 调度、每 worker 独立 Session、`DevicePool` 用 generation 标记轮换设备（避免多 worker 重复换设备）、IP 级注册限流指数退避 |
|
||||
| `download.py` | 带 Range 的断点续传下载；落盘文件名即 aweme_id，「是否已下」看文件在不在；`Registry` 去重记录 |
|
||||
| `meta.py` | 标题/标签拆分（split_desc，认 desc 正则全集不认 text_extra）+ `RecordLog`：records.jsonl 线程安全追加写、行首 aweme_id 快速扫描续跑。约定：**aweme_id 必须是每行第一个字段** |
|
||||
| `aiclass.py` | 判定作品是否 AI 相关，四档取最强：platform（平台 AIGC 标）> strong（文案/标签明确 AI 信号）> topic（在讲 AI）> author（仅昵称带 AI）。规则详解见 [AI.md](AI.md) |
|
||||
| `cos.py` | 腾讯 COS 上传。`put()` 上传后必 `head_object` 取 Content-Length 与本地字节比对，对不上算失败——删除闸门卡在这 |
|
||||
| `audio.py` | ffmpeg/ffprobe 薄封装：定位可执行文件、mp4 抽音轨（默认转 mp3 192k，`copy` 模式直接封装 m4a 无损）、`-count_frames` 实解码探时长（format=duration 会被 Xing 头骗过，不能用） |
|
||||
| `urls.py` | 链接解析：裸 id / 分享短链 / 整段分享口令 / video / note / modal_id 一律还原成 aweme_id |
|
||||
| `client.py` | Web 端客户端（`Aweme` / `DouyinError` 等共用类型 + web 请求路径）。类型被正常用着；**web 请求路径闲置**——匿名被强制登录，见 PROGRESS.md |
|
||||
| `signer.py` | a_bogus 签名器接口。`NodeSigner` 需要项目外的 `node_signer/sign.js`（不在仓库里）；`ABogusSigner` 纯 Python 版没写完。web 端真要走通得先补这块 |
|
||||
| `crypto.py` | 签名器的密码学积木：SM3 / RC4 / 自定义 Base64，带官方测试向量自检。目前只被 test_smoke 验证，未被 signer 使用 |
|
||||
|
||||
### 数据目录
|
||||
|
||||
| 目录 | 内容 |
|
||||
|---|---|
|
||||
| `data/` | 源数据 jsonl/csv（外部抓取的输入，删了没法重新筛，别动） |
|
||||
| `records/` | `records.jsonl`（流水线产物 + 断点记录，一行一条含 raw）、`manifest.jsonl`（第一批老格式清单）、`pipeline_failed.txt`（失败清单） |
|
||||
| `exports/` | xlsx/csv 交付导出物 |
|
||||
| `lists/` | 各批的链接清单，也是「哪批下了什么」的凭据 |
|
||||
| `logs/` | 各批跑批日志 |
|
||||
|
||||
---
|
||||
|
||||
## 使用指南
|
||||
|
||||
### 环境准备
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
| 依赖 | 用途 | 备注 |
|
||||
|---|---|---|
|
||||
| Python ≥ 3.11 | | 实测 3.11.0 |
|
||||
| `ffmpeg` / `ffprobe` | 抽音轨、探时长 | 需在 PATH，实测 2025-07-31 gyan.dev full build |
|
||||
| `requests` | 接口/下载 | |
|
||||
| `cos-python-sdk-v5` | COS 上传 | 延迟导入，不传 COS 可不装 |
|
||||
| `openpyxl` | xlsx 导出 | 延迟导入，不出 xlsx 可不装 |
|
||||
|
||||
COS 凭证只走环境变量（不进命令行、不进代码）：
|
||||
|
||||
```powershell
|
||||
$env:COS_SECRET_ID = "..."
|
||||
$env:COS_SECRET_KEY = "..."
|
||||
```
|
||||
|
||||
### 典型工作流：从一批源数据到交付
|
||||
|
||||
```bash
|
||||
# ① 拿到外部抓的源 jsonl，先放 data/，看 AI 分布（别信文件名，先 --stats）
|
||||
python scripts/clean_ai.py -i data/抖音ai视频_20260904_62171.jsonl --stats
|
||||
|
||||
# ② 筛出 platform+strong 档，出链接清单到 lists/
|
||||
python scripts/clean_ai.py -i data/抖音ai视频_20260904_62171.jsonl --tier platform,strong -o lists/ai_62171
|
||||
|
||||
# ③ 小批试跑 20 条，--keep-local 不删本地，人工检查 work/ 里的 mp4/mp3
|
||||
python pipeline.py --from-file lists/ai_62171.txt --region ap-chongqing --limit 20 --keep-local
|
||||
|
||||
# ④ 正式跑批。断点续跑靠 records/records.jsonl，中断/重启/换机器都能续
|
||||
python pipeline.py --from-file lists/ai_62171.txt --region ap-chongqing --workers 8
|
||||
|
||||
# ⑤ 失败的重跑：同一条命令再跑一遍即可（已完成的自动跳过，
|
||||
# 失败项的本地文件还留着，省一次下载）
|
||||
python pipeline.py --from-file lists/ai_62171.txt --region ap-chongqing --workers 8
|
||||
|
||||
# ⑥ 出交付表
|
||||
python scripts/jsonl_to_xlsx.py -i records/records.jsonl --columns full -o "exports/全量.xlsx"
|
||||
python scripts/export_sample_xlsx.py --ai-only -o "exports/AI视频样本.xlsx"
|
||||
```
|
||||
|
||||
### pipeline.py 参数速查
|
||||
|
||||
| 参数 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `target...` / `--from-file` | | 链接或 aweme_id，二选一或混用；文件里 `#` 开头行视为注释 |
|
||||
| `--region` | **必填** | COS 地域，如 ap-chongqing |
|
||||
| `--bucket` / `--prefix` | asr-dig-data-1253181912 / 0005/zh/ | COS 落点 |
|
||||
| `--workers` | 8 | 网络并发数 |
|
||||
| `--bitrate` | 192k | mp3 码率 |
|
||||
| `--ffmpeg-workers` | CPU 核数 | ffmpeg 并发上限 |
|
||||
| `--device-quota` | 550 | 单设备 detail 调用上限（实测 ~575 被限，留余量） |
|
||||
| `--limit` | 0 | 只跑前 N 条，试跑用 |
|
||||
| `--max-items` | 12000 | 单进程跑满即自动重启接着跑，防 Windows 桌面堆耗尽；0=不切 |
|
||||
| `--abort-wait` / `--abort-wait-cap` | 1800s / 14400s | 撞 IP 级注册限流后的等待/上限，连撞翻倍 |
|
||||
| `--keep-local` | 关 | 上传后不删本地文件（试跑检查用） |
|
||||
| `--records` / `--failed` | records/ 下两个文件 | 元信息落点 / 失败清单 |
|
||||
|
||||
退出码：0 全成；4 有失败（见 failed 文件，重跑同命令续传）；3 开跑即注册失败；
|
||||
5 撞上 IP 限流已停下，等几小时冷却后重跑同命令。
|
||||
|
||||
### clean_ai.py 参数速查
|
||||
|
||||
| 参数 | 默认 | 说明 |
|
||||
|---|---|---|
|
||||
| `-i` | 必填 | 源 jsonl |
|
||||
| `--stats` | | 只看分布不落盘，拿到新文件第一步 |
|
||||
| `--tier` | platform,strong | 保留哪些档（platform/strong/topic/author/none） |
|
||||
| `-o` | 空 | 输出前缀，生成 `<前缀>.txt`（要 `--with-jsonl` 才另出 jsonl） |
|
||||
| `--platform-needs-text` | 关 | platform 档要求文案也有 AI 信号。会误伤文案零信号的 AI 古风作者，见 AI.md §8 |
|
||||
| `--exclude-forbidden` | 关 | 剔 allow_download=false 的。**不需要**——aigc_forbidden 只限 web 端 |
|
||||
| `--samples N` | 0 | 每档打印 N 条样例到 stderr，复核用 |
|
||||
|
||||
### 导出参数速查
|
||||
|
||||
```bash
|
||||
python scripts/jsonl_to_xlsx.py --list-columns # 先看有哪些列可选
|
||||
python scripts/jsonl_to_xlsx.py --columns full ... # full = 标题/作者/标签/时长等常用全字段
|
||||
python scripts/export_sample_xlsx.py --ai-only --limit 500 ...
|
||||
```
|
||||
|
||||
### 常见问题
|
||||
|
||||
- **跑到一半卡住、完成数冻结、失败猛涨**：撞 Windows 桌面堆上限了。Ctrl-C 后重跑
|
||||
同命令即可（断点续跑），或调低 `--max-items` 让重启更频繁。
|
||||
- **大量「空 body」失败**：设备配额到了属正常（会自动换设备）；若报
|
||||
「IP 级注册配额耗尽」，流水线会自己等待续跑，放着不管即可。
|
||||
- **跑批前几百条失败率 15%**：正常。图文作品（图集）扎堆在 aweme_id 排序的头部，
|
||||
后面会掉到 2-3%。
|
||||
- **想只重试失败项**：把 failed 文件第一列抠出来当输入重跑，或直接重跑原命令
|
||||
（已完成项跳过，代价只是启动时扫一遍 records）。
|
||||
132
docs/superpowers/specs/2026-08-25-download-pipeline-design.md
Normal file
132
docs/superpowers/specs/2026-08-25-download-pipeline-design.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# 抖音采集流水线一体化 —— 设计
|
||||
|
||||
日期：2026-08-25
|
||||
|
||||
## 目标
|
||||
|
||||
把现在的 4 个独立阶段（下载 → 抽 mp3 → 探时长 → 传 COS）合成一条流水线，
|
||||
每条视频端到端走完就删本地文件，并把标题等全部元信息记进一个本地 jsonl。
|
||||
|
||||
## 动机
|
||||
|
||||
1. **磁盘**：分阶段跑必须把全部 mp4 摊在盘上（本次 53 GB），单条流水线的
|
||||
峰值只有 `workers × (mp4+mp3)`，约 100 MB，且与总量无关。
|
||||
2. **元信息丢失**：现在只有 `.downloaded.json` 记了 author/desc/uri，且它跟着
|
||||
输出目录一起被删掉了。元信息需要一个独立于媒体文件的落点。
|
||||
|
||||
## 架构
|
||||
|
||||
一个 worker 端到端负责一条，不再分阶段：
|
||||
|
||||
```
|
||||
1. fetch_detail(id) → Aweme + raw
|
||||
2. 下载 mp4 → {--work-dir}/{id}.mp4
|
||||
3. ffmpeg 抽 mp3 → {--work-dir}/{id}.mp3 [信号量限流]
|
||||
4. ffprobe 探 mp4 时长；mp3 用 -count_frames 实际解码算时长
|
||||
5. 校验 |mp3实解时长 - mp4时长| <= max(1s, 2%) ← 抽取闸门
|
||||
6. 上传 mp4 → COS（覆盖同名对象）
|
||||
7. 上传 mp3 → COS（覆盖同名对象）
|
||||
8. head_object 核对两个对象 Content-Length == 本地字节数 ← 删除闸门
|
||||
9. 追加 record 到 records.jsonl，然后删本地 mp4 / mp3
|
||||
```
|
||||
|
||||
工作目录由 `--work-dir` 指定，默认 `work/`，mp4 和 mp3 同放一处——
|
||||
反正传完就删，不需要像原来那样分 `downloads/` 和 `audio/`。
|
||||
|
||||
### 并发
|
||||
|
||||
- 下载、上传是网络密集 → 按 `--workers` 并发（默认 8）
|
||||
- ffmpeg 是 CPU 密集 → 单独用信号量限到 `cpu_count`，避免 8 个 worker
|
||||
同时撞进第 3 步把 CPU 打满
|
||||
|
||||
### 失败处理
|
||||
|
||||
任何一步失败：**保留本地文件不删**，写入 `pipeline_failed.txt`（id + 阶段 + 原因），
|
||||
继续下一条。不中断整批。
|
||||
|
||||
### 断点续跑
|
||||
|
||||
启动时扫 `records.jsonl` 已完成的 id。为避免对数百 MB 的文件逐行 `json.loads`，
|
||||
**`aweme_id` 必须是每行的第一个字段**，扫描用 `^\{"aweme_id":"(\d+)"` 直接抠。
|
||||
这是一个契约，重排字段会让启动扫描从 ~1s 退化到十几秒。
|
||||
|
||||
## 数据
|
||||
|
||||
### records.jsonl（唯一落点，仅本地，不上传 COS）
|
||||
|
||||
每行一条，字段顺序固定，`aweme_id` 在最前：
|
||||
|
||||
```json
|
||||
{"aweme_id":"6797367856917990659",
|
||||
"title":"剥掉标签后的纯标题",
|
||||
"desc":"服务端原样 desc，一字不改",
|
||||
"tags":["标签1","标签2"],
|
||||
"author":{"nickname":"...","uid":"...","sec_uid":"..."},
|
||||
"create_time":1582635538,
|
||||
"create_time_str":"2020-02-25 21:38:58",
|
||||
"duration_sec":41.702993,
|
||||
"duration_ms_api":41750,
|
||||
"stats":{"digg":0,"comment":0,"collect":0,"share":0,"play":0},
|
||||
"music":{"title":"...","author":"...","url":"..."},
|
||||
"cover_url":"...",
|
||||
"video":{"uri":"...","width":1280,"height":720},
|
||||
"cos_mp4":"cos://asr-dig-data-1253181912/0005/zh/{id}.mp4",
|
||||
"cos_mp3":"cos://asr-dig-data-1253181912/0005/zh/{id}.mp3",
|
||||
"size_mp4":4026560,
|
||||
"size_mp3":1001230,
|
||||
"source_url":"https://www.douyin.com/video/{id}",
|
||||
"fetched_at":1756000000,
|
||||
"raw":{ ...服务端 aweme_detail 原样... }}
|
||||
```
|
||||
|
||||
`duration_sec`（ffprobe 精确解码值）与 `duration_ms_api`（服务端声明值）**都留**：
|
||||
实测同一条 41703 ms vs 41750 ms，不等价，前者才能用于 ASR 切分。
|
||||
|
||||
预计体积 200~400 MB / 5400 条。因此所有读取必须逐行流式。
|
||||
|
||||
### 标题拆分
|
||||
|
||||
- `desc` —— 原样照抄，唯一真相来源
|
||||
- `title` —— 用 `text_extra` 的 start/end 偏移剥掉标签，再用正则兜底
|
||||
- `tags` —— **正则从 `desc` 提取全集**，`text_extra` 只用来补 hashtag_id
|
||||
|
||||
理由：实测 12 条样本中有 2 条的标签在 `desc` 里有、`text_extra` 里没有
|
||||
（如 `#非洲搞笑视频`、`#女人`）。以 `text_extra` 为准会静默丢标签。
|
||||
|
||||
## 文件改动
|
||||
|
||||
| 文件 | 改动 |
|
||||
|---|---|
|
||||
| `pipeline.py` | 新增，流水线入口 |
|
||||
| `douyin/meta.py` | 新增：`split_desc()` / `build_record()` / `RecordLog` |
|
||||
| `douyin/cos.py` | 新增：从 `upload_cos.py` 抽出 `CosUploader`（含 `verify()`） |
|
||||
| `douyin/client.py` | `Aweme` 补 tags / duration_ms / uid / sec_uid / stats / width / height |
|
||||
| `jsonl_to_xlsx.py` | 改流式读取，支持 `records.jsonl`，加 `--columns` |
|
||||
| `test_smoke.py` | 加 `split_desc` / `RecordLog` 离线测试 |
|
||||
|
||||
原有 4 个单阶段脚本保留不动，仍可单独使用。
|
||||
|
||||
## 安全
|
||||
|
||||
- 删除是唯一不可逆操作，卡在第 5、8 两道闸门之后
|
||||
- **mp3 截断校验必须实际解码**。`ffprobe format=duration` 读的是 LAME 写的
|
||||
Xing 头，它声明的是原始总帧数：实测把 mp3 截断到 20%，报的仍然是完整的
|
||||
10.0078s，和没截断时一字不差。`ffmpeg -f null -` 也拦不住（rc=0、无 stderr）。
|
||||
只有 `-count_frames` 会真解码，代价是 120s 的 mp3 多花 0.056s（0.109 vs 0.053）
|
||||
- `--keep-local` 关闭删除，供试跑
|
||||
- COS 上已存在同名对象时**直接覆盖**，不做存在性检查、不比对远端大小。
|
||||
本地这一份永远是权威版本。少一个分支，行为也更好预测。
|
||||
- 重复劳动由 `records.jsonl` 的断点续跑拦掉（已完成的 id 整条跳过，
|
||||
连下载都不会发生），所以不需要再靠 COS 存在性去省事
|
||||
- COS 凭证只从环境变量 `COS_SECRET_ID` / `COS_SECRET_KEY` 读
|
||||
|
||||
### 重跑老数据的后果
|
||||
|
||||
拿原来那 5406 条链接重跑，会重新下载约 53 GB 并覆盖 COS 上现有的 68 GB。
|
||||
这是预期行为——覆盖后本地与 COS 一致，且拿到全量元信息。
|
||||
|
||||
## 不做
|
||||
|
||||
- 不回填已下载的 5393 条（除非显式拿老链接重跑）
|
||||
- 不改 `manifest.jsonl` / `抖音-四川话.xlsx` 既有格式
|
||||
- 不支持图文作品
|
||||
37
douyin/__init__.py
Normal file
37
douyin/__init__.py
Normal file
@@ -0,0 +1,37 @@
|
||||
"""抖音单视频无水印下载。"""
|
||||
|
||||
from .app import AppClient, Device, DeviceRegisterError, NotAVideo
|
||||
from .batch import BatchStats, run_batch
|
||||
from .client import (
|
||||
Aweme,
|
||||
DouyinClient,
|
||||
DouyinError,
|
||||
LoginRequired,
|
||||
decode_abort,
|
||||
play_url_from_uri,
|
||||
)
|
||||
from .cos import CosError, CosUploader, SizeMismatch
|
||||
from .download import Registry, download, safe_filename
|
||||
from .meta import RecordLog, build_record, iter_records, split_desc
|
||||
from .signer import (
|
||||
ABogusConfig,
|
||||
ABogusSigner,
|
||||
NodeSigner,
|
||||
SignError,
|
||||
Signer,
|
||||
SignerNotConfigured,
|
||||
)
|
||||
from .urls import ParseError, extract_aweme_id
|
||||
|
||||
__all__ = [
|
||||
"AppClient", "Device", "DeviceRegisterError", "NotAVideo",
|
||||
"BatchStats", "run_batch",
|
||||
"Aweme", "DouyinClient", "DouyinError", "LoginRequired", "decode_abort",
|
||||
"play_url_from_uri",
|
||||
"Registry", "download", "safe_filename",
|
||||
"CosError", "CosUploader", "SizeMismatch",
|
||||
"RecordLog", "build_record", "iter_records", "split_desc",
|
||||
"ABogusConfig", "ABogusSigner", "NodeSigner", "SignError", "Signer",
|
||||
"SignerNotConfigured",
|
||||
"ParseError", "extract_aweme_id",
|
||||
]
|
||||
258
douyin/aiclass.py
Normal file
258
douyin/aiclass.py
Normal file
@@ -0,0 +1,258 @@
|
||||
"""判定一条抖音作品是不是 AI 相关，以及是哪一类相关。
|
||||
|
||||
分四档，档位之间不重叠，取最强的那一档：
|
||||
|
||||
platform 抖音自己打了 AIGC 标（ai.is_aigc）。最可信，而且是唯一能抓到
|
||||
"作者什么都没标、但确实是 AI 生成"的信号 —— 实测 1744 条属于
|
||||
这种，文案标签里一个 ai 字都没有（脑筋急转弯、历史解说、原创动画）。
|
||||
strong 文案/标签里有明确的"AI 生成作品"信号：AIGC、AI 生成类工具名、
|
||||
ai+创作类词、平台的 AI 赛事标签。
|
||||
topic 只提到 AI 本身（ai 教程、人工智能、大模型、ChatGPT 之类），
|
||||
片子讲的是 AI，但不一定是 AI 做的。
|
||||
author 作品本身没信号，只有作者昵称里带 AI/AIGC。最弱，会误收
|
||||
（实测 @牛牛Ai 发的普法解说、@你好，我是大福 的真人游泳视频）。
|
||||
|
||||
坑（都是从这批 40087 条真实数据里挖出来的，别顺手删）：
|
||||
|
||||
1. "Al"（大写 A + 小写 L）被当成 AI 用，标签 `抖音al` 出现 129 次，
|
||||
文案里也有 `#Al创作`。必须归一化，否则整片漏掉。
|
||||
2. 反过来，`ai` 是英文单词的常见片段（email / detail / again / hair），
|
||||
所以只认前后不接字母数字的独立 ai，不能用 in 判断。
|
||||
3. `剪映`/`关键帧`/`蒙版`/`转场`/`剪辑教程` 是剪辑术语，不是 AI。
|
||||
它们在这批数据里量很大（剪映 375、关键帧 314、蒙版 203），
|
||||
早期版本把"动画/绘画/分身"这类词单独当信号，误收了几千条。
|
||||
所以中文泛词一律要求跟 AI 词同现，不单独成立。
|
||||
4. 工具名不含 ai 字样的一大票：即梦 / seedance / libtv / flovaai /
|
||||
小云雀 / 随变 / runninghub / oiioii / shotlab，只能靠名单。
|
||||
5. 音乐标题默认是 `@<作者昵称>创作的原声`，40087 条里有 32555 条是这个壳。
|
||||
昵称带 AI 就会连"创作"一起命中，等于让作者昵称从 music 这条路
|
||||
偷偷升到 strong —— 实测 771 条这样混进来（@冠宇Aigc 的猴子炒菜、
|
||||
@牛牛Ai 的大案纪实）。所以先剥壳，剥完只当弱信号：全库剥完只剩
|
||||
16 条音乐标题还有命中，这条路本来就没什么价值。
|
||||
注意壳有两种写法：jsonl 里是 `@昵称创作的原声`，csv 导出的是
|
||||
`@昵称创作的原声 - 昵称`，后缀那半截也要一起剥。
|
||||
6. 一批工具名本身就是常用词，单独出现几乎全是误判，必须要求跟 ai/aigc
|
||||
同现（见 _AMBIG_*）。实测：豆包 224 条误判（狗名、豆沙包、"反强包
|
||||
之神"）对 145 条真命中；海螺 13 条是三峰的"海螺"帐篷；元宝 11 条是
|
||||
元宝勺辅食餐具；万相 17 条是《龙神万相》和万相集；星流 1 条来自
|
||||
"明星流量"（子串）；清影 1 条来自"起舞弄清影"。而真命中的那些**无一
|
||||
例外**都同时带 AI 字样，所以加这道门槛不掉召回。
|
||||
7. 别把"数字人""换脸""分身"放开成单独信号：数字人民币含"数字人"、
|
||||
微信分身/五五分身材含"分身"、穿搭视频也说"这套穿搭能换脸"。
|
||||
实测放开只进误判，一条真的都捞不到。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Dict, List, Tuple
|
||||
|
||||
# 独立的 ai —— 前后不能接字母数字，挡掉 email、detail、hair。
|
||||
# 第二个字符收了三种冒充写法：i、小写 L、罗马数字 Ⅰ/ⅰ（U+2160/U+2170）。
|
||||
# 实测标签 `抖音al` 129 次、`#AⅠ漫剧` 1 条。别改成简单的 in 判断。
|
||||
_I = r'[ilⅠⅰ]'
|
||||
_AI = re.compile(r'(?<![a-z0-9])a' + _I + r'(?![a-z0-9])', re.I)
|
||||
_AIGC = re.compile(r'(?<![a-z0-9])a' + _I + r'gc(?![a-z0-9])', re.I)
|
||||
|
||||
# 生成类工具 / 平台。命中即 strong —— 用了这些东西，片子就是 AI 做的。
|
||||
_GEN_TOOLS = [
|
||||
"即梦", "dreamina", "jimeng", "seedance", "seedream", "可灵", "kling",
|
||||
"hailuo", "vidu", "pixverse", "pika", "luma", "runway", "sora",
|
||||
"viggle", "heygen", "midjourney", "comfyui", "runninghub", "flux",
|
||||
"nanobanana", "nano banana", "stable diffusion", "liblib",
|
||||
"krea", "ideogram", "recraft", "leonardo",
|
||||
"higgsfield", "suno", "udio", "libtv", "flovaai", "海艺", "小云雀",
|
||||
"shotlab", "sekoai", "movieflow", "updream", "cineart", "oiioii",
|
||||
"minimax", "妙鸭", "一帧秒创", "触手ai", "拍我ai", "美图ai",
|
||||
"文生视频", "图生视频", "文生图", "图生图", "生成式",
|
||||
# --- 下面这批是 2026-08 补的。全部在 12.7 万条真实数据上验过：
|
||||
# 要么零命中（纯属给以后的文件备着），要么命中的全是真 AI，
|
||||
# 没有一个引入新误判。判断标准是"这串字在中文短视频文案里
|
||||
# 不可能有别的意思"，所以像 tripo（三脚架 tripod）、seaweed
|
||||
# （海藻）、lora（撞 valorant）、captions、playground 这些
|
||||
# 一律没收 —— 实测 lora 命中 3 条全是无畏契约电竞。
|
||||
"veo3", "wan2.1", "wanx", "通义万相", "hunyuanvideo", "混元视频",
|
||||
"cogvideo", "open-sora", "animatediff", "controlnet", "智谱清影",
|
||||
"step-video", "hedra", "kaiber", "haiper", "moonvalley", "genmo",
|
||||
"domoai", "synthesia", "akool", "deepbrain", "opusclip", "argil",
|
||||
"kreadoai", "腾讯智影", "讯飞智作", "硅基智能", "魔音工坊", "元气口播",
|
||||
"万兴播爆", "闪剪", "声音克隆", "语音克隆", "克隆音色", "音色克隆",
|
||||
]
|
||||
|
||||
# AI 漫剧圈的专有名词：**单独出现即 strong**，不要求跟 ai 同现。
|
||||
# 抖音的 AI 漫剧已经形成固定圈子，这些词在 18.4 万条真实数据里几乎只出现在
|
||||
# AI 作品上（括号=全库命中/其中现在还漏掉的）。选词标准是"这串字在中文短视频
|
||||
# 文案里不可能有别的意思"：
|
||||
# 漫剧(1568/64) 治愈系漫画(80/0) 渡灵书(62/0) 超级裂变(46/0) 漫剧推荐(38/2)
|
||||
# 二次元动漫(33/0) 月落小溪(25/4) 七罪创世(21/0) 晚夏小梦(67/4)
|
||||
# 灵绘漫客(12/0) neowow(12/0) 造梦国度(2/0) 猫箱跨次元相遇(114/12)
|
||||
# 没收的（实测污染严重，它们是题材不是 AI 标记）：
|
||||
# 西游记(+26 真·玄奘故里游记) 孙悟空(+16) 宫崎骏(+10 龙猫童年) 民间故事(+7《狼来了》)
|
||||
# 睡前故事(+5 爷俩演西游) 神话故事(+2) 国漫推荐(+4《剑来》真人制作)
|
||||
# 原创动漫(+18 手绘二次元) 创作者激励计划(+12 原神/卡西尼号) 未来导演扶持计划(+4)
|
||||
# 开放赛道(+2) —— 后两个是抖音通用活动，不专属 AI
|
||||
_MANJU = [
|
||||
"漫剧", "治愈系漫画", "渡灵书", "超级裂变", "二次元动漫", "月落小溪",
|
||||
"七罪创世", "晚夏小梦", "灵绘漫客", "neowow", "造梦国度", "猫箱跨次元相遇",
|
||||
# 红果（字节的免费短剧 App）上的 AI 漫剧片名。来源是新闻报道的热榜，
|
||||
# **不是红果的完整目录** —— 红果不公开片库，AI 漫剧实际有成百上千部，
|
||||
# 这里只有见诸报道的头部作品，以后遇到新的往这加。
|
||||
# 在 18.4 万条上验过：16/18 零命中（纯属备着），万妖图录传 1 条且正是
|
||||
# 现在漏掉的真 AI 漫剧，零污染。
|
||||
"万妖图录传", "发配边关", "罪妻开荒养出战神", "儿媳的重生棋局",
|
||||
"斩仙台", "菩提临世", "聚宝仙盆", "杂灵根", "渔乡守真心", "金钱照冷暖",
|
||||
"杀猪辣媳", "夫人一个过肩摔", "瘫子总裁站起来", "凡人百世书",
|
||||
"都重生了谁还装富二代",
|
||||
# **"红果"/"红果短剧" 不能进这个表**：红果主体是真人短剧，AI 漫剧只是
|
||||
# 其中一个赛道。实测 3 条命中里 2 条是真人短剧（《家里家外2》《冷御失守》），
|
||||
# 1 条是短剧演员面试。要抓红果的 AI 漫剧只能靠具体片名。
|
||||
]
|
||||
|
||||
# 名字本身是常用词的生成类工具：只有跟 ai/aigc 同现才算 strong。
|
||||
# 见模块开头第 6 条 —— 单独出现基本全是误判，真命中都带 AI 字样。
|
||||
_AMBIG_GEN = [
|
||||
"海螺", "白日梦", "万相", "可图", "清影", "星流", "秒画",
|
||||
"造梦日记", "随变",
|
||||
]
|
||||
|
||||
# 对话/通用大模型。这些更像是"在聊 AI"，归 topic。
|
||||
_CHAT_TOOLS = [
|
||||
"chatgpt", "deepseek", "kimi", "通义", "千问", "qwen", "文心",
|
||||
"ernie", "智谱", "glm", "copilot", "gemini", "grok",
|
||||
"人工智能", "大模型", "神经网络", "机器学习", "深度学习",
|
||||
]
|
||||
# 同上，但归 topic 档的那批常用词模型名
|
||||
_AMBIG_CHAT = ["豆包", "元宝", "混元"]
|
||||
# 单独成词才算，避免 gpt 撞上别的串
|
||||
_CHAT_RE = re.compile(r'(?<![a-z0-9])(?:gpt|claude|llm|agi|ai\s*agent)(?![a-z0-9])')
|
||||
|
||||
# 跟 ai 同现才成立的"创作/生成"语境词。单独出现一律不算。
|
||||
_GEN_CTX = [
|
||||
"短剧", "漫剧", "动画", "动漫", "视频", "短片", "电影", "影视", "绘画",
|
||||
"绘图", "创作", "制作", "生成", "分身", "剧情", "作画", "配音", "换脸",
|
||||
"特效", "导演", "演员", "数字人", "虚拟人", "成片", "新影像", "影像",
|
||||
"浪潮", "大赛", "赛道", "大赏", "工作台", "写真", "画师", "作品", "片子",
|
||||
"微短剧", "番剧", "剪辑", "转绘", "扩图", "修图", "提示词",
|
||||
]
|
||||
_GEN_CTX_RE = re.compile("|".join(map(re.escape, _GEN_CTX)))
|
||||
|
||||
def _alt(words: List[str]) -> re.Pattern:
|
||||
"""把词表编成一个正则。纯 ASCII 的词加字母边界：前后不能紧接字母，
|
||||
但**允许接数字和中文** —— 数据里满地都是 `Seedance2.0`、`sora2`、
|
||||
`runninghub0.21`、`libtv创作者`、`flovaai超创`，一刀切加 \\b 会全丢。
|
||||
只挡真正的词中词：`udio` 藏在 `theonestudio`/`Grass Studio` 里 17 次，
|
||||
`flux` 藏在 `Fluxxwave` 曲名里 1 次，两个都是从 music 混进 author 档的。
|
||||
"""
|
||||
parts = []
|
||||
for w in words:
|
||||
e = re.escape(w)
|
||||
if re.fullmatch(r"[a-z0-9 .\-]+", w):
|
||||
e = r"(?<![a-z])" + e + r"(?![a-z])"
|
||||
parts.append(e)
|
||||
return re.compile("|".join(parts))
|
||||
|
||||
|
||||
_TOOL_RE = _alt(_GEN_TOOLS)
|
||||
_CHATW_RE = _alt(_CHAT_TOOLS)
|
||||
_MANJU_RE = _alt(_MANJU)
|
||||
_AMBIG_GEN_RE = _alt(_AMBIG_GEN)
|
||||
_AMBIG_CHAT_RE = _alt(_AMBIG_CHAT)
|
||||
|
||||
# 抖音默认原声的标题壳，里面就是作者昵称，不含额外信息。
|
||||
# jsonl 是 `@昵称创作的原声`，csv 导出多一截 ` - 昵称`，见模块开头第 5 条。
|
||||
_SOUND_WRAP = re.compile(r'^@.*?(?:创作的原声|的原声)(?:\s*-\s*.*)?$')
|
||||
|
||||
TIERS = ("platform", "strong", "topic", "author", "none")
|
||||
_RANK = {t: i for i, t in enumerate(TIERS)}
|
||||
|
||||
|
||||
def _fields(rec: Dict[str, Any]) -> Tuple[str, str, str]:
|
||||
"""(作品文本, 作者昵称, 音乐标题)。作品文本 = desc + 所有标签名。"""
|
||||
tags = [t.get("name", "") if isinstance(t, dict) else str(t)
|
||||
for t in (rec.get("hashtags") or [])]
|
||||
body = " ".join([rec.get("desc") or ""] + tags)
|
||||
author = (rec.get("author") or {}).get("nickname") or ""
|
||||
music = ((rec.get("music") or {}).get("title") or "").strip()
|
||||
if _SOUND_WRAP.match(music):
|
||||
music = "" # 只是作者昵称的复读，见模块开头第 5 条
|
||||
return body, author, music
|
||||
|
||||
|
||||
def _scan(text: str, where: str) -> Tuple[str, List[str]]:
|
||||
"""扫一段文本，返回 (档位, 命中的信号)。档位只可能是 strong/topic/none。"""
|
||||
low = text.lower()
|
||||
sig: List[str] = []
|
||||
tier = "none"
|
||||
|
||||
has_aigc = bool(_AIGC.search(low))
|
||||
has_ai = bool(_AI.search(low))
|
||||
# 歧义词表的开关：文本里得先有 ai/aigc 字样，那些常用词才作数
|
||||
ai_present = has_ai or has_aigc
|
||||
|
||||
if has_aigc:
|
||||
sig.append(f"{where}:aigc")
|
||||
tier = "strong"
|
||||
|
||||
for m in sorted(set(_TOOL_RE.findall(low))):
|
||||
sig.append(f"{where}:工具:{m}")
|
||||
tier = "strong"
|
||||
|
||||
# 漫剧专名单独成立，不要求 ai 同现
|
||||
for m in sorted(set(_MANJU_RE.findall(low))):
|
||||
sig.append(f"{where}:漫剧:{m}")
|
||||
tier = "strong"
|
||||
|
||||
if ai_present:
|
||||
for m in sorted(set(_AMBIG_GEN_RE.findall(low))):
|
||||
sig.append(f"{where}:工具:{m}")
|
||||
tier = "strong"
|
||||
|
||||
if has_ai:
|
||||
ctx = sorted(set(_GEN_CTX_RE.findall(text)))
|
||||
if ctx:
|
||||
sig.append(f"{where}:ai+{'/'.join(ctx[:4])}")
|
||||
tier = "strong"
|
||||
else:
|
||||
sig.append(f"{where}:ai")
|
||||
if tier == "none":
|
||||
tier = "topic"
|
||||
|
||||
chat = set(_CHATW_RE.findall(low)) | set(_CHAT_RE.findall(low))
|
||||
if ai_present:
|
||||
chat |= set(_AMBIG_CHAT_RE.findall(low))
|
||||
chat = sorted(chat)
|
||||
if chat:
|
||||
sig.append(f"{where}:模型:{'/'.join(chat[:4])}")
|
||||
if tier == "none":
|
||||
tier = "topic"
|
||||
|
||||
return tier, sig
|
||||
|
||||
|
||||
def classify(rec: Dict[str, Any]) -> Dict[str, Any]:
|
||||
"""判一条记录。返回 {tier, signals, is_aigc}。"""
|
||||
body, author, music = _fields(rec)
|
||||
signals: List[str] = []
|
||||
tier = "none"
|
||||
|
||||
is_aigc = bool((rec.get("ai") or {}).get("is_aigc"))
|
||||
label = ((rec.get("ai") or {}).get("label") or "").strip()
|
||||
if is_aigc:
|
||||
tier = "platform"
|
||||
signals.append(f"platform:{label or 'is_aigc'}")
|
||||
|
||||
t, s = _scan(body, "desc")
|
||||
signals.extend(s)
|
||||
if _RANK[t] < _RANK[tier]:
|
||||
tier = t
|
||||
|
||||
# 作者昵称和音乐标题都只是"谁发的"，不是"作品是什么"。
|
||||
# 只在作品自己毫无信号时兜底成 author 档，绝不往上顶。
|
||||
for where, text in (("author", author), ("music", music)):
|
||||
wt, wsig = _scan(text, where)
|
||||
if wt != "none":
|
||||
signals.extend(wsig)
|
||||
if tier == "none":
|
||||
tier = "author"
|
||||
|
||||
return {"tier": tier, "signals": signals, "is_aigc": is_aigc}
|
||||
343
douyin/app.py
Normal file
343
douyin/app.py
Normal file
@@ -0,0 +1,343 @@
|
||||
"""App 端接口客户端 —— 无签名、无 Cookie、无登录。
|
||||
|
||||
实测链路（每一步都验过）：
|
||||
|
||||
1. POST /service/2/device_register/ 引导调用，不需要任何签名
|
||||
-> device_id / install_id
|
||||
2. GET /aweme/v1/aweme/detail/ 带上真实 device_id/iid 即可
|
||||
-> 完整 aweme_detail，含 video.play_addr.uri
|
||||
3. GET /aweme/v1/play/?video_id=URI 免鉴权，302 到 douyinvod CDN
|
||||
-> mp4
|
||||
|
||||
对比 web 端：web 的 /aweme/v1/web/aweme/detail/ 对匿名用户返回
|
||||
{"name":"强制登录"}，且要算 a_bogus。App 端两样都不需要。
|
||||
|
||||
早先测 App 端拿到空 body，是因为 device_id/iid 是伪造的；换成注册得来的
|
||||
真实凭据就正常了 —— 空 body 不代表要签名。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import random
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Optional
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import requests
|
||||
|
||||
from .client import Aweme, DouyinClient, DouyinError
|
||||
|
||||
REGISTER_HOSTS = (
|
||||
"https://log.snssdk.com/service/2/device_register/",
|
||||
"https://api.amemv.com/service/2/device_register/",
|
||||
"https://log-api.amemv.com/service/2/device_register/",
|
||||
"https://toblog.ctobsnssdk.com/service/2/device_register/",
|
||||
)
|
||||
|
||||
API_HOSTS = (
|
||||
"https://aweme.snssdk.com",
|
||||
"https://api.amemv.com",
|
||||
)
|
||||
|
||||
APP_VERSION = "9.9.9"
|
||||
VERSION_CODE = "990"
|
||||
UPDATE_VERSION_CODE = "99013"
|
||||
APP_ID = "1128"
|
||||
CHANNEL = "douyin_official"
|
||||
|
||||
# 机型池。openudid/cdid 本来就每次随机，但 UA、机型、分辨率这些以前是写死的，
|
||||
# 意味着我们所有注册请求在这些维度上完全一致。
|
||||
#
|
||||
# 注册受限时到底是按 IP 还是按"UA+机型"这类特征分桶，我们没验证过——本机没有
|
||||
# 第二个出口 IP，做不了对照实验。把机型随机化是成本极低的两用手段：
|
||||
# 如果是按机型特征分桶，它能缓解；如果是纯 IP，它无害。
|
||||
#
|
||||
# 每项是 (device_type, device_brand, os_version, os_api, resolution, dpi)。
|
||||
DEVICE_POOL = (
|
||||
("SM-G9750", "samsung", "12", "31", "1080*2340", "420"),
|
||||
("SM-S9080", "samsung", "13", "33", "1440*3088", "560"),
|
||||
("PDRM00", "OPPO", "12", "31", "1080*2400", "480"),
|
||||
("V2118A", "vivo", "12", "31", "1080*2376", "480"),
|
||||
("M2012K11AC", "Xiaomi", "13", "33", "1080*2400", "440"),
|
||||
("ELS-AN00", "HUAWEI", "12", "31", "1176*2400", "480"),
|
||||
("21091116AC", "Redmi", "12", "31", "1080*2400", "440"),
|
||||
("NOH-AN01", "HONOR", "12", "31", "1344*2772", "480"),
|
||||
)
|
||||
|
||||
# 兼容旧引用：默认取池子里第一项
|
||||
DEVICE_TYPE, DEVICE_BRAND, OS_VERSION, OS_API = DEVICE_POOL[0][:4]
|
||||
|
||||
|
||||
class DeviceRegisterError(DouyinError):
|
||||
pass
|
||||
|
||||
|
||||
class NotAVideo(DouyinError):
|
||||
"""作品存在，但不是视频（图文/图集）。"""
|
||||
|
||||
|
||||
def _hex(n: int) -> str:
|
||||
return "".join(random.choice("0123456789abcdef") for _ in range(n))
|
||||
|
||||
|
||||
class Device:
|
||||
"""一台虚拟设备的身份。注册一次可长期复用，所以落盘缓存。"""
|
||||
|
||||
def __init__(self, device_id: str, install_id: str, openudid: str,
|
||||
cdid: str, clientudid: str,
|
||||
profile: Optional[tuple] = None) -> None:
|
||||
self.device_id = device_id
|
||||
self.install_id = install_id
|
||||
self.openudid = openudid
|
||||
self.cdid = cdid
|
||||
self.clientudid = clientudid
|
||||
# 机型档案必须跟着设备走：注册时报的是小米、后续 detail 请求的 UA
|
||||
# 却是三星，这种自相矛盾本身就是个可疑特征。
|
||||
self.profile = tuple(profile) if profile else DEVICE_POOL[0]
|
||||
|
||||
@property
|
||||
def device_type(self) -> str:
|
||||
return self.profile[0]
|
||||
|
||||
@property
|
||||
def device_brand(self) -> str:
|
||||
return self.profile[1]
|
||||
|
||||
@property
|
||||
def os_version(self) -> str:
|
||||
return self.profile[2]
|
||||
|
||||
@property
|
||||
def os_api(self) -> str:
|
||||
return self.profile[3]
|
||||
|
||||
@property
|
||||
def resolution(self) -> str:
|
||||
return self.profile[4]
|
||||
|
||||
@property
|
||||
def dpi(self) -> str:
|
||||
return self.profile[5]
|
||||
|
||||
@property
|
||||
def user_agent(self) -> str:
|
||||
return (f"com.ss.android.ugc.aweme/{UPDATE_VERSION_CODE} "
|
||||
f"(Linux; U; Android {self.os_version}; zh_CN; {self.device_type}; "
|
||||
f"Build/QP1A.190711.020; Cronet/TTNetVersion:6c7b9b1a 2022-11-29)")
|
||||
|
||||
def to_dict(self) -> Dict[str, Any]:
|
||||
return {
|
||||
"device_id": self.device_id, "install_id": self.install_id,
|
||||
"openudid": self.openudid, "cdid": self.cdid,
|
||||
"clientudid": self.clientudid, "profile": list(self.profile),
|
||||
}
|
||||
|
||||
@classmethod
|
||||
def from_dict(cls, d: Dict[str, Any]) -> "Device":
|
||||
# profile 是后加的，旧缓存没有这个字段，退回池子第一项
|
||||
return cls(d["device_id"], d["install_id"], d["openudid"],
|
||||
d["cdid"], d["clientudid"], d.get("profile"))
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
@classmethod
|
||||
def register(cls, session: Optional[requests.Session] = None,
|
||||
timeout: float = 20.0, retries: int = 5,
|
||||
backoff: float = 30.0) -> "Device":
|
||||
"""向 ByteDance 注册一台新设备。这一步不需要签名。
|
||||
|
||||
短时间内注册太多次，所有 host 都会返回 device_id='0'。实测：
|
||||
指纹（openudid/cdid）每次都是新的照样被拒，4 个 host 同时被拒，
|
||||
约 10 分钟后自行恢复 —— 所以是临时限流，且不是按设备指纹或按
|
||||
单个 host 分桶的。
|
||||
|
||||
至于分桶的键到底是 IP 还是 UA/机型这类特征，我们没验证过：本机
|
||||
没有第二个出口 IP，做不了对照实验。DEVICE_POOL 的随机化对两种
|
||||
情况都无害，权当两用。
|
||||
|
||||
这里做指数退避重试而不是直接抛错，否则调用方会把后续所有条目
|
||||
瞬间判为失败。
|
||||
"""
|
||||
last: Optional[Exception] = None
|
||||
for attempt in range(retries):
|
||||
try:
|
||||
return cls._register_once(session, timeout)
|
||||
except DeviceRegisterError as exc:
|
||||
last = exc
|
||||
if attempt < retries - 1:
|
||||
wait = backoff * (2 ** attempt)
|
||||
print(f" [注册受限] 等待 {wait:.0f}s 后重试 "
|
||||
f"({attempt + 1}/{retries})", flush=True)
|
||||
time.sleep(wait)
|
||||
raise last # type: ignore[misc]
|
||||
|
||||
@classmethod
|
||||
def _register_once(cls, session: Optional[requests.Session] = None,
|
||||
timeout: float = 20.0) -> "Device":
|
||||
session = session or requests.Session()
|
||||
openudid = _hex(16)
|
||||
cdid = str(uuid.UUID(int=random.getrandbits(128)))
|
||||
clientudid = str(uuid.UUID(int=random.getrandbits(128)))
|
||||
profile = random.choice(DEVICE_POOL)
|
||||
dev_type, dev_brand, os_ver, os_api, resolution, dpi = profile
|
||||
|
||||
ua = (f"com.ss.android.ugc.aweme/{UPDATE_VERSION_CODE} "
|
||||
f"(Linux; U; Android {os_ver}; zh_CN; {dev_type}; "
|
||||
f"Build/QP1A.190711.020; Cronet/TTNetVersion:6c7b9b1a 2022-11-29)")
|
||||
|
||||
query = {
|
||||
"ac": "wifi", "channel": CHANNEL, "aid": APP_ID, "app_name": "aweme",
|
||||
"version_code": VERSION_CODE, "version_name": APP_VERSION,
|
||||
"device_platform": "android", "os": "android", "ssmix": "a",
|
||||
"device_type": dev_type, "device_brand": dev_brand,
|
||||
"language": "zh", "os_api": os_api, "os_version": os_ver,
|
||||
"manifest_version_code": VERSION_CODE, "resolution": resolution,
|
||||
"dpi": dpi, "update_version_code": UPDATE_VERSION_CODE,
|
||||
"_rticket": str(int(time.time() * 1000)),
|
||||
"cdid": cdid, "openudid": openudid,
|
||||
}
|
||||
payload = {
|
||||
"magic_tag": "ss_app_log",
|
||||
"header": {
|
||||
"display_name": "抖音",
|
||||
"update_version_code": int(UPDATE_VERSION_CODE),
|
||||
"manifest_version_code": int(VERSION_CODE), "aid": int(APP_ID),
|
||||
"channel": CHANNEL, "package": "com.ss.android.ugc.aweme",
|
||||
"app_version": APP_VERSION, "version_code": int(VERSION_CODE),
|
||||
"sdk_version": "2.13.5", "os": "Android",
|
||||
"os_version": os_ver, "os_api": int(os_api),
|
||||
"device_model": dev_type, "device_brand": dev_brand,
|
||||
"device_manufacturer": dev_brand, "cpu_abi": "arm64-v8a",
|
||||
"density_dpi": int(dpi), "display_density": "mdpi",
|
||||
"resolution": resolution.replace("*", "x"),
|
||||
"language": "zh", "timezone": 8,
|
||||
"access": "wifi", "not_request_sender": 0, "region": "CN",
|
||||
"tz_name": "Asia/Shanghai", "tz_offset": 28800,
|
||||
"sim_region": "cn", "openudid": openudid,
|
||||
"clientudid": clientudid, "cdid": cdid,
|
||||
"rom": "QP1A.190711.020", "rom_version": f"{dev_brand}/{os_ver}",
|
||||
"device_id": 0, "install_id": 0,
|
||||
},
|
||||
"_gen_time": int(time.time() * 1000),
|
||||
}
|
||||
headers = {
|
||||
"User-Agent": ua,
|
||||
"Content-Type": "application/json; charset=utf-8",
|
||||
"Accept-Encoding": "gzip",
|
||||
"X-SS-REQ-TICKET": str(int(time.time() * 1000)),
|
||||
}
|
||||
|
||||
errors = []
|
||||
for host in REGISTER_HOSTS:
|
||||
try:
|
||||
r = session.post(host + "?" + urlencode(query),
|
||||
data=json.dumps(payload), headers=headers,
|
||||
timeout=timeout)
|
||||
data = r.json()
|
||||
except (requests.RequestException, ValueError) as exc:
|
||||
errors.append(f"{host}: {type(exc).__name__}")
|
||||
continue
|
||||
|
||||
did = str(data.get("device_id_str") or data.get("device_id") or "")
|
||||
iid = str(data.get("install_id_str") or data.get("install_id") or "")
|
||||
if did and did != "0":
|
||||
return cls(did, iid, openudid, cdid, clientudid, profile)
|
||||
errors.append(f"{host}: device_id={did!r}")
|
||||
|
||||
raise DeviceRegisterError("设备注册全部失败: " + "; ".join(errors))
|
||||
|
||||
@classmethod
|
||||
def load_or_register(cls, cache: str | Path,
|
||||
session: Optional[requests.Session] = None) -> "Device":
|
||||
cache = Path(cache)
|
||||
if cache.exists():
|
||||
try:
|
||||
return cls.from_dict(json.loads(cache.read_text(encoding="utf-8")))
|
||||
except (ValueError, KeyError, OSError):
|
||||
pass
|
||||
dev = cls.register(session)
|
||||
cache.parent.mkdir(parents=True, exist_ok=True)
|
||||
cache.write_text(json.dumps(dev.to_dict(), indent=2), encoding="utf-8")
|
||||
return dev
|
||||
|
||||
|
||||
class AppClient:
|
||||
"""按 aweme_id 取详情。不需要签名、Cookie 或登录态。"""
|
||||
|
||||
def __init__(self, device: Optional[Device] = None,
|
||||
cache: str | Path = ".device.json",
|
||||
timeout: float = 25.0) -> None:
|
||||
self.session = requests.Session()
|
||||
self.device = device or Device.load_or_register(cache, self.session)
|
||||
self.timeout = timeout
|
||||
self.session.headers.update({
|
||||
"User-Agent": self.device.user_agent,
|
||||
"Accept-Encoding": "gzip",
|
||||
"Connection": "keep-alive",
|
||||
"sdk-version": "2",
|
||||
})
|
||||
|
||||
def _params(self) -> Dict[str, str]:
|
||||
return {
|
||||
"device_platform": "android", "os": "android", "ssmix": "a",
|
||||
"_rticket": str(int(time.time() * 1000)),
|
||||
"cdid": self.device.cdid, "channel": CHANNEL, "aid": APP_ID,
|
||||
"app_name": "aweme", "version_code": VERSION_CODE,
|
||||
"version_name": APP_VERSION, "manifest_version_code": VERSION_CODE,
|
||||
"update_version_code": UPDATE_VERSION_CODE,
|
||||
"resolution": self.device.resolution, "dpi": self.device.dpi,
|
||||
"device_type": self.device.device_type,
|
||||
"device_brand": self.device.device_brand,
|
||||
"language": "zh", "os_api": self.device.os_api,
|
||||
"os_version": self.device.os_version,
|
||||
"openudid": self.device.openudid,
|
||||
"device_id": self.device.device_id, "iid": self.device.install_id,
|
||||
"ac": "wifi", "ts": str(int(time.time())),
|
||||
}
|
||||
|
||||
def fetch_detail(self, aweme_id: str, retries: int = 3) -> Aweme:
|
||||
last = ""
|
||||
for attempt in range(retries):
|
||||
host = API_HOSTS[attempt % len(API_HOSTS)]
|
||||
params = self._params()
|
||||
params["aweme_id"] = aweme_id
|
||||
url = f"{host}/aweme/v1/aweme/detail/?" + urlencode(params)
|
||||
try:
|
||||
r = self.session.get(url, timeout=self.timeout)
|
||||
except requests.RequestException as exc:
|
||||
last = f"{type(exc).__name__}: {exc}"
|
||||
continue
|
||||
|
||||
if not r.content:
|
||||
last = f"HTTP {r.status_code} 空 body"
|
||||
time.sleep(1 + attempt)
|
||||
continue
|
||||
try:
|
||||
data: Dict[str, Any] = r.json()
|
||||
except ValueError:
|
||||
last = f"非 JSON: {r.text[:120]!r}"
|
||||
continue
|
||||
|
||||
detail = data.get("aweme_detail")
|
||||
if not detail:
|
||||
# 服务端会在 filter_detail 里说明为什么不给正片，
|
||||
# 例如图文作品 filter_reason=images_base
|
||||
f = data.get("filter_detail") or {}
|
||||
reason = f.get("filter_reason") or ""
|
||||
if reason == "images_base":
|
||||
raise NotAVideo(f"{aweme_id} 是图文作品（图集），不是视频")
|
||||
if reason:
|
||||
raise DouyinError(
|
||||
f"{aweme_id} 被服务端过滤：{reason}"
|
||||
+ (f"（{f.get('notice')}）" if f.get("notice") else "")
|
||||
)
|
||||
raise DouyinError(
|
||||
f"无 aweme_detail，status_code={data.get('status_code')}，"
|
||||
"作品可能已删除或为私密"
|
||||
)
|
||||
return DouyinClient.parse(detail)
|
||||
|
||||
raise DouyinError(f"App detail 连续 {retries} 次失败（{last}）")
|
||||
71
douyin/audio.py
Normal file
71
douyin/audio.py
Normal file
@@ -0,0 +1,71 @@
|
||||
"""ffmpeg / ffprobe 的薄封装：找可执行文件、拼抽音命令、探时长。
|
||||
|
||||
批量抽取的调度不在这里 —— `pipeline.py` 自己按条编排（下载→抽音→上传→删），
|
||||
所以这个模块只提供单次调用要用的零件。
|
||||
|
||||
从 mp4 里提取音轨。
|
||||
|
||||
源音频是 AAC 44.1kHz 立体声 128kbps，据此有两种模式：
|
||||
|
||||
mp3 转码成 MP3。有损转有损，必然掉质量。码率给太高只是浪费空间
|
||||
（源就 128k），默认 192k 是留余量避免二次编码劣化，不是"更清晰"。
|
||||
copy 直接把 AAC 流拷进 .m4a 容器。无损、秒完、体积更小。
|
||||
只有在必须是 .mp3 后缀时才用 mp3 模式。
|
||||
|
||||
依赖 ffmpeg，需在 PATH 里。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
|
||||
class FFmpegMissing(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def ensure_ffmpeg(exe: str = "ffmpeg") -> str:
|
||||
path = shutil.which(exe)
|
||||
if not path:
|
||||
raise FFmpegMissing(
|
||||
f"找不到 {exe}。装一个并加进 PATH，或用 --ffmpeg 指定完整路径。")
|
||||
return path
|
||||
|
||||
|
||||
|
||||
def build_cmd(ffmpeg: str, src: Path, dst: Path, mode: str, bitrate: str) -> List[str]:
|
||||
cmd = [ffmpeg, "-hide_banner", "-loglevel", "error", "-y", "-i", str(src), "-vn"]
|
||||
if mode == "copy":
|
||||
# 不重新编码，直接搬运 AAC 流
|
||||
cmd += ["-acodec", "copy"]
|
||||
else:
|
||||
cmd += ["-acodec", "libmp3lame", "-b:a", bitrate]
|
||||
cmd.append(str(dst))
|
||||
return cmd
|
||||
|
||||
|
||||
def probe(path, ffprobe: str = "ffprobe", timeout: float = 60.0):
|
||||
"""探一个媒体文件的容器时长，探不出来返回 None。
|
||||
|
||||
时长取自 mp4（源），不取 mp3 —— mp3 是转码产物，虽然实测两者一致，
|
||||
但源文件才是权威。
|
||||
|
||||
注意这里读的是容器声明的 format=duration，对 mp4 没问题，
|
||||
但**不能拿去判断 mp3 有没有被截断**：LAME 写的 Xing 头声明的是原始
|
||||
总帧数，ffprobe 直接信它而不解码，实测把 mp3 截到 20% 它报的还是原时长。
|
||||
那种场景要用 pipeline.decoded_seconds()（-count_frames 真解码）。
|
||||
"""
|
||||
cmd = [ffprobe, "-v", "error", "-show_entries", "format=duration",
|
||||
"-of", "default=noprint_wrappers=1:nokey=1", str(path)]
|
||||
try:
|
||||
p_ = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
return None
|
||||
try:
|
||||
d = float((p_.stdout or "").strip())
|
||||
except ValueError:
|
||||
return None
|
||||
return d if d > 0 else None
|
||||
197
douyin/batch.py
Normal file
197
douyin/batch.py
Normal file
@@ -0,0 +1,197 @@
|
||||
"""批量下载：并发、可断点续跑、单条失败不影响整批。
|
||||
|
||||
设计要点：
|
||||
- 设备身份只注册一次，各 worker 共用；但每个 worker 有独立 Session
|
||||
（requests.Session 并发下不保证安全）
|
||||
- 落盘文件名就是 aweme_id，所以「是否已下载」直接看文件在不在，
|
||||
不依赖去重记录 —— 记录丢了也能无损续跑
|
||||
- 失败条目写 failed.txt，跑完可直接拿它当输入重试
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Optional, Sequence
|
||||
|
||||
import requests
|
||||
|
||||
from .app import AppClient, Device, NotAVideo
|
||||
from .client import DouyinError
|
||||
from .download import Registry, download
|
||||
|
||||
|
||||
class DevicePool:
|
||||
"""设备身份池。
|
||||
|
||||
实测单个 device_id 在约 575 次 detail 调用后被限流，表现是接口一直
|
||||
回 HTTP 200 空 body；同一 IP 下换个新注册的设备立刻恢复 —— 说明配额
|
||||
是设备级而非 IP 级。注册本身免签名、成本极低，所以按量轮换即可。
|
||||
|
||||
用 generation 标记当前设备：worker 拿到的 gen 和池里的对得上才允许
|
||||
触发轮换，避免多个 worker 同时发现失败、把设备连换好几次。
|
||||
"""
|
||||
|
||||
def __init__(self, max_calls: int = 400, initial: Optional[Device] = None) -> None:
|
||||
self._lock = threading.Lock()
|
||||
self._device = initial
|
||||
self._gen = 0
|
||||
self._calls = 0
|
||||
self.max_calls = max_calls
|
||||
self.rotations = 0
|
||||
|
||||
def current(self) -> tuple[Device, int]:
|
||||
with self._lock:
|
||||
if self._device is None or self._calls >= self.max_calls:
|
||||
self._rotate_locked("配额用尽")
|
||||
self._calls += 1
|
||||
return self._device, self._gen
|
||||
|
||||
def report_dead(self, gen: int) -> None:
|
||||
"""worker 发现当前设备被限流时调用。"""
|
||||
with self._lock:
|
||||
if gen == self._gen:
|
||||
self._rotate_locked("被限流")
|
||||
|
||||
def _rotate_locked(self, why: str) -> None:
|
||||
self._device = Device.register()
|
||||
self._gen += 1
|
||||
self._calls = 0
|
||||
self.rotations += 1
|
||||
print(f" [设备轮换#{self.rotations}] {why} -> "
|
||||
f"device_id={self._device.device_id}", flush=True)
|
||||
|
||||
|
||||
@dataclass
|
||||
class BatchStats:
|
||||
total: int = 0
|
||||
done: int = 0
|
||||
skipped: int = 0
|
||||
not_video: int = 0
|
||||
failed: int = 0
|
||||
bytes_: int = 0
|
||||
lock: threading.Lock = field(default_factory=threading.Lock)
|
||||
|
||||
@property
|
||||
def finished(self) -> int:
|
||||
return self.done + self.skipped + self.not_video + self.failed
|
||||
|
||||
|
||||
def run_batch(aweme_ids: Sequence[str], out_dir: Path, device: Optional[Device] = None,
|
||||
workers: int = 5, delay: float = 0.3,
|
||||
registry: Optional[Registry] = None,
|
||||
log_every: int = 25, max_calls_per_device: int = 400) -> BatchStats:
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
registry = registry or Registry(out_dir / ".downloaded.json")
|
||||
pool = DevicePool(max_calls=max_calls_per_device, initial=device)
|
||||
|
||||
stats = BatchStats(total=len(aweme_ids))
|
||||
failed_path = out_dir / "failed.txt"
|
||||
# 每轮重来，否则上一轮已修复的失败会一直留在文件里造成误导
|
||||
failed_path.write_text("", encoding="utf-8")
|
||||
failed_lock = threading.Lock()
|
||||
local = threading.local()
|
||||
t0 = time.time()
|
||||
|
||||
def client_for(device_: Device, gen: int) -> AppClient:
|
||||
"""每线程每代设备一个 Session（Session 并发不安全）。"""
|
||||
if getattr(local, "gen", None) != gen:
|
||||
local.client = AppClient(device=device_)
|
||||
local.gen = gen
|
||||
return local.client
|
||||
|
||||
def dl_session() -> requests.Session:
|
||||
"""下载用的 Session：CDN 免鉴权，不该占设备配额，也不随设备轮换重建。"""
|
||||
if not hasattr(local, "dl"):
|
||||
local.dl = requests.Session()
|
||||
local.dl.headers["User-Agent"] = (
|
||||
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
|
||||
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36"
|
||||
)
|
||||
return local.dl
|
||||
|
||||
def record_failure(aweme_id: str, why: str) -> None:
|
||||
with failed_lock:
|
||||
with open(failed_path, "a", encoding="utf-8") as f:
|
||||
f.write(f"{aweme_id}\t{why}\n")
|
||||
|
||||
def fetch_with_rotation(aweme_id: str):
|
||||
"""限流表现为空 body。碰到就换设备再试一次，而不是直接判失败。"""
|
||||
last: Optional[Exception] = None
|
||||
for _ in range(2):
|
||||
dev, gen = pool.current()
|
||||
client = client_for(dev, gen)
|
||||
if delay:
|
||||
time.sleep(delay)
|
||||
try:
|
||||
return client.fetch_detail(aweme_id, retries=2)
|
||||
except NotAVideo:
|
||||
raise
|
||||
except DouyinError as exc:
|
||||
last = exc
|
||||
if "空 body" in str(exc):
|
||||
pool.report_dead(gen)
|
||||
continue
|
||||
raise
|
||||
raise last # type: ignore[misc]
|
||||
|
||||
def handle(aweme_id: str) -> None:
|
||||
dest = out_dir / f"{aweme_id}.mp4"
|
||||
# 文件在就算下过，不必碰网络也不依赖记录
|
||||
if dest.exists() and dest.stat().st_size > 0:
|
||||
with stats.lock:
|
||||
stats.skipped += 1
|
||||
return
|
||||
|
||||
try:
|
||||
aweme = fetch_with_rotation(aweme_id)
|
||||
except NotAVideo as exc:
|
||||
with stats.lock:
|
||||
stats.not_video += 1
|
||||
record_failure(aweme_id, f"not_video: {exc}")
|
||||
return
|
||||
except Exception as exc:
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
record_failure(aweme_id, f"detail: {type(exc).__name__}: {exc}")
|
||||
return
|
||||
|
||||
if not aweme.video_urls:
|
||||
with stats.lock:
|
||||
stats.not_video += 1
|
||||
record_failure(aweme_id, "no_video_url")
|
||||
return
|
||||
|
||||
try:
|
||||
result = download(dl_session(), aweme.video_urls, dest, progress=False)
|
||||
except Exception as exc:
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
record_failure(aweme_id, f"download: {type(exc).__name__}: {exc}")
|
||||
return
|
||||
|
||||
registry.add(aweme_id, result.path, result.size,
|
||||
desc=aweme.desc, author=aweme.author, uri=aweme.uri)
|
||||
with stats.lock:
|
||||
stats.done += 1
|
||||
stats.bytes_ += result.size
|
||||
|
||||
with ThreadPoolExecutor(max_workers=workers) as executor:
|
||||
futures = [executor.submit(handle, a) for a in aweme_ids]
|
||||
for n, _ in enumerate(as_completed(futures), 1):
|
||||
if n % log_every == 0 or n == len(futures):
|
||||
el = time.time() - t0
|
||||
rate = n / el if el else 0
|
||||
eta = (len(futures) - n) / rate if rate else 0
|
||||
print(f"[{n}/{len(futures)}] 成功{stats.done} 跳过{stats.skipped} "
|
||||
f"图集{stats.not_video} 失败{stats.failed} "
|
||||
f"| {stats.bytes_ / 1024 ** 3:.2f}GB "
|
||||
f"| {rate * 60:.0f}条/分 | 换设备{pool.rotations}次 "
|
||||
f"| 剩约 {eta / 60:.0f}分",
|
||||
flush=True)
|
||||
|
||||
registry.flush()
|
||||
return stats
|
||||
289
douyin/client.py
Normal file
289
douyin/client.py
Normal file
@@ -0,0 +1,289 @@
|
||||
"""抖音 Web 接口客户端。
|
||||
|
||||
实测要点（都是打真实接口验出来的）：
|
||||
|
||||
1. 空 body 有两种成因，必须分开看：
|
||||
- 新 session 的第一个请求必然空，第二个才通 —— cookie 预热，见 _warm_up()
|
||||
- 服务端主动拦截 —— 响应头 X-Whale-Throughput-Abort-Data 里有 base64 原因
|
||||
2. /aweme/detail/ 对匿名用户返回 abort={"name":"强制登录"}，必须带登录 Cookie。
|
||||
3. /aweme/related/ 匿名可用，返回完整 play_addr，但不含目标视频自身。
|
||||
4. 请求头 UA 必须与签名时的 UA 完全一致，params 里的 browser_version 也要跟着对齐。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import json
|
||||
import random
|
||||
import string
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Any, Dict, List, Optional
|
||||
from urllib.parse import urlencode
|
||||
|
||||
import requests
|
||||
|
||||
from .signer import Signer
|
||||
|
||||
DETAIL_URL = "https://www.douyin.com/aweme/v1/web/aweme/detail/"
|
||||
RELATED_URL = "https://www.douyin.com/aweme/v1/web/aweme/related/"
|
||||
TTWID_URL = "https://ttwid.bytedance.com/ttwid/union/register/"
|
||||
|
||||
BASE_PARAMS: Dict[str, str] = {
|
||||
"device_platform": "webapp",
|
||||
"aid": "6383",
|
||||
"channel": "channel_pc_web",
|
||||
"pc_client_type": "1",
|
||||
"version_code": "190500",
|
||||
"version_name": "19.5.0",
|
||||
"cookie_enabled": "true",
|
||||
"screen_width": "1920",
|
||||
"screen_height": "1080",
|
||||
"browser_language": "zh-CN",
|
||||
"browser_platform": "Win32",
|
||||
"browser_name": "Chrome",
|
||||
"browser_version": "126.0.0.0",
|
||||
"browser_online": "true",
|
||||
"engine_name": "Blink",
|
||||
"engine_version": "126.0.0.0",
|
||||
"os_name": "Windows",
|
||||
"os_version": "10",
|
||||
"cpu_core_num": "8",
|
||||
"device_memory": "8",
|
||||
"platform": "PC",
|
||||
"downlink": "10",
|
||||
"effective_type": "4g",
|
||||
"round_trip_time": "50",
|
||||
}
|
||||
|
||||
|
||||
PLAY_URL = "https://www.douyin.com/aweme/v1/play/"
|
||||
|
||||
|
||||
def play_url_from_uri(uri: str, ratio: str = "1080p", watermark: bool = False) -> str:
|
||||
"""由 video URI 拼出播放直链。
|
||||
|
||||
实测这个接口完全免鉴权：无 a_bogus、无 Cookie、无登录，裸 requests 即可，
|
||||
302 跳到 douyinvod.com 的 CDN 地址。所以只要拿到了 URI，后续下载不再需要
|
||||
任何身份——URI 值得缓存复用。
|
||||
"""
|
||||
path = "playwm" if watermark else "play"
|
||||
return f"https://www.douyin.com/aweme/v1/{path}/?video_id={uri}&ratio={ratio}&line=0"
|
||||
|
||||
|
||||
class DouyinError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class LoginRequired(DouyinError):
|
||||
"""服务端明确要求登录。"""
|
||||
|
||||
|
||||
@dataclass
|
||||
class Aweme:
|
||||
aweme_id: str
|
||||
desc: str
|
||||
author: str
|
||||
create_time: int
|
||||
video_urls: List[str]
|
||||
cover_urls: List[str]
|
||||
music_url: Optional[str]
|
||||
uri: str
|
||||
raw: Dict[str, Any]
|
||||
|
||||
@property
|
||||
def is_video(self) -> bool:
|
||||
return bool(self.video_urls or self.uri)
|
||||
|
||||
def play_url(self, ratio: str = "1080p") -> str:
|
||||
"""免鉴权直链。play_addr 里的地址带时效，URI 换出来的是现算的。"""
|
||||
if not self.uri:
|
||||
raise DouyinError("该作品没有 video URI")
|
||||
return play_url_from_uri(self.uri, ratio)
|
||||
|
||||
|
||||
def _rand(n: int, pool: str) -> str:
|
||||
return "".join(random.choice(pool) for _ in range(n))
|
||||
|
||||
|
||||
def gen_ms_token(length: int = 107) -> str:
|
||||
return _rand(length, string.ascii_letters + string.digits + "=-_")
|
||||
|
||||
|
||||
def gen_s_v_web_id() -> str:
|
||||
body = "".join(
|
||||
"_" if i in (8, 13, 18, 23) else random.choice("0123456789abcdef")
|
||||
for i in range(32)
|
||||
)
|
||||
return f"verify_{format(int(time.time()), 'x')}_{body}"
|
||||
|
||||
|
||||
def decode_abort(resp: requests.Response) -> str:
|
||||
"""解出服务端拦截原因；没有则返回空串。"""
|
||||
raw = resp.headers.get("X-Whale-Throughput-Abort-Data")
|
||||
if not raw:
|
||||
return ""
|
||||
try:
|
||||
info = json.loads(base64.b64decode(raw))
|
||||
return info.get("name") or info.get("content") or ""
|
||||
except (ValueError, TypeError):
|
||||
return "unknown"
|
||||
|
||||
|
||||
class DouyinClient:
|
||||
def __init__(self, signer: Signer, cookie: str = "", timeout: float = 25.0,
|
||||
warm_up: bool = True) -> None:
|
||||
self.signer = signer
|
||||
self.timeout = timeout
|
||||
self.user_agent = signer.user_agent
|
||||
self.fp = gen_s_v_web_id()
|
||||
|
||||
# params 里的浏览器版本必须跟签名用的 UA 对上，否则服务端会判不一致
|
||||
self.params = dict(BASE_PARAMS)
|
||||
if "Chrome/" in self.user_agent:
|
||||
ver = self.user_agent.split("Chrome/")[1].split(" ")[0]
|
||||
self.params["browser_version"] = ver
|
||||
self.params["engine_version"] = ver
|
||||
|
||||
self.session = requests.Session()
|
||||
self.session.headers.update({
|
||||
"User-Agent": self.user_agent,
|
||||
"Accept": "application/json, text/plain, */*",
|
||||
"Accept-Language": "zh-CN,zh;q=0.9",
|
||||
"Referer": "https://www.douyin.com/",
|
||||
})
|
||||
|
||||
self._logged_in = bool(cookie)
|
||||
if cookie:
|
||||
self.session.headers["Cookie"] = cookie
|
||||
else:
|
||||
self._bootstrap_cookies()
|
||||
|
||||
self._warmed = False
|
||||
if warm_up:
|
||||
self._warm_up()
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def _bootstrap_cookies(self) -> None:
|
||||
try:
|
||||
self.session.post(TTWID_URL, timeout=self.timeout, json={
|
||||
"region": "cn", "aid": 1768, "needFid": False,
|
||||
"service": "www.ixigua.com",
|
||||
"migrate_info": {"ticket": "", "source": "node"},
|
||||
"cbUrlProtocol": "https", "union": True,
|
||||
})
|
||||
except requests.RequestException:
|
||||
pass
|
||||
self.session.cookies.set("s_v_web_id", self.fp, domain=".douyin.com")
|
||||
self.session.cookies.set("odin_tt", _rand(160, "0123456789abcdef"),
|
||||
domain=".douyin.com")
|
||||
|
||||
def _warm_up(self) -> None:
|
||||
"""新 session 的第一个接口请求必然返回空，先打一发把它消耗掉。"""
|
||||
if self._warmed:
|
||||
return
|
||||
try:
|
||||
self._get(RELATED_URL, {"aweme_id": "1", "count": "1", "filterGids": "1"})
|
||||
except (DouyinError, requests.RequestException):
|
||||
pass
|
||||
self._warmed = True
|
||||
|
||||
def _build_url(self, base: str, extra: Dict[str, str]) -> str:
|
||||
p = dict(self.params)
|
||||
p.update(extra)
|
||||
p["msToken"] = gen_ms_token()
|
||||
return f"{base}?{urlencode(p)}"
|
||||
|
||||
def _get(self, base: str, extra: Dict[str, str],
|
||||
retries: int = 3) -> Dict[str, Any]:
|
||||
"""签名 + 请求 + 解析。
|
||||
|
||||
「空 body 且无拦截原因」是可重试的瞬时态（session 还没热起来），
|
||||
「空 body 且原因含登录」是硬门槛，立即抛错不重试。
|
||||
"""
|
||||
last = ""
|
||||
for attempt in range(1, retries + 1):
|
||||
url = self.signer.sign_url(self._build_url(base, extra), self.fp)
|
||||
resp = self.session.get(url, timeout=self.timeout)
|
||||
|
||||
if resp.content:
|
||||
try:
|
||||
return resp.json()
|
||||
except ValueError as exc:
|
||||
raise DouyinError(f"响应不是 JSON: {resp.text[:200]!r}") from exc
|
||||
|
||||
reason = decode_abort(resp)
|
||||
if "登录" in reason:
|
||||
raise LoginRequired(
|
||||
f"接口要求登录（服务端返回 {reason!r}）。"
|
||||
"请用 --cookie 传入已登录的抖音 Cookie。"
|
||||
)
|
||||
last = (f"HTTP {resp.status_code}"
|
||||
+ (f"，拦截原因 {reason!r}" if reason else "，无拦截原因"))
|
||||
if attempt < retries:
|
||||
time.sleep(2 * attempt)
|
||||
|
||||
raise DouyinError(f"接口连续 {retries} 次返回空 body（{last}）")
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
def fetch_detail(self, aweme_id: str) -> Aweme:
|
||||
"""按 aweme_id 取作品详情。匿名调用会抛 LoginRequired。"""
|
||||
self.session.headers["Referer"] = f"https://www.douyin.com/video/{aweme_id}"
|
||||
data = self._get(DETAIL_URL, {"aweme_id": aweme_id})
|
||||
|
||||
detail = data.get("aweme_detail")
|
||||
if not detail:
|
||||
raise DouyinError(
|
||||
f"响应无 aweme_detail，status_code={data.get('status_code')}，"
|
||||
"作品可能已删除或为私密"
|
||||
)
|
||||
return self.parse(detail)
|
||||
|
||||
def fetch_related(self, aweme_id: str, count: int = 10) -> List[Aweme]:
|
||||
"""相关推荐。匿名可用，但不含 aweme_id 自身。"""
|
||||
data = self._get(RELATED_URL, {
|
||||
"aweme_id": aweme_id, "count": str(count), "filterGids": aweme_id,
|
||||
})
|
||||
return [self.parse(a) for a in (data.get("aweme_list") or [])]
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
|
||||
@staticmethod
|
||||
def parse(detail: Dict[str, Any]) -> Aweme:
|
||||
video = detail.get("video") or {}
|
||||
|
||||
# play_addr 带水印，playwm -> play 即无水印源。
|
||||
# bit_rate 里有多档清晰度，优先挑码率最高的。
|
||||
urls: List[str] = []
|
||||
gears = sorted(
|
||||
(b for b in (video.get("bit_rate") or []) if b.get("play_addr")),
|
||||
key=lambda b: b.get("bit_rate") or 0, reverse=True,
|
||||
)
|
||||
for gear in gears:
|
||||
urls.extend((gear["play_addr"].get("url_list") or []))
|
||||
urls.extend((video.get("play_addr") or {}).get("url_list") or [])
|
||||
|
||||
seen: set[str] = set()
|
||||
video_urls = []
|
||||
for u in urls:
|
||||
u = u.replace("playwm", "play")
|
||||
if u not in seen:
|
||||
seen.add(u)
|
||||
video_urls.append(u)
|
||||
|
||||
cover = video.get("origin_cover") or video.get("cover") or {}
|
||||
music_list = ((detail.get("music") or {}).get("play_url") or {}).get("url_list") or []
|
||||
|
||||
return Aweme(
|
||||
aweme_id=str(detail.get("aweme_id", "")),
|
||||
desc=(detail.get("desc") or "").strip(),
|
||||
author=((detail.get("author") or {}).get("nickname") or "").strip(),
|
||||
create_time=int(detail.get("create_time") or 0),
|
||||
video_urls=video_urls,
|
||||
cover_urls=list(cover.get("url_list") or []),
|
||||
music_url=music_list[0] if music_list else None,
|
||||
uri=(video.get("play_addr") or {}).get("uri") or "",
|
||||
raw=detail,
|
||||
)
|
||||
94
douyin/cos.py
Normal file
94
douyin/cos.py
Normal file
@@ -0,0 +1,94 @@
|
||||
"""腾讯云 COS 上传，带上传后校验。
|
||||
|
||||
流水线传完就删本地文件，所以「传成功了吗」这个判断必须硬。
|
||||
`upload_file` 不抛异常只说明请求没报错，不说明对象真的落到桶里、
|
||||
大小也对。所以 put() 一律在上传后 head_object 取回 Content-Length
|
||||
跟本地字节数比对，对不上就当失败——删除闸门卡在这个返回值上。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import threading
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
DEFAULT_BUCKET = "asr-dig-data-1253181912"
|
||||
DEFAULT_PREFIX = "0005/zh/"
|
||||
|
||||
|
||||
class CosError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class SizeMismatch(CosError):
|
||||
"""对象传上去了，但远端大小和本地对不上。"""
|
||||
|
||||
|
||||
def object_url(key: str, bucket: str, region: str, style: str = "cos") -> str:
|
||||
if style == "cos":
|
||||
return f"cos://{bucket}/{key}"
|
||||
if style == "https":
|
||||
return f"https://{bucket}.cos.{region}.myqcloud.com/{key}"
|
||||
return f"/{bucket}/{key}"
|
||||
|
||||
|
||||
class CosUploader:
|
||||
"""线程安全的上传器。同名对象一律覆盖，不做存在性检查。
|
||||
|
||||
重复劳动由上层的 records.jsonl 断点续跑拦掉（已完成的 id 整条跳过，
|
||||
连下载都不会发生），所以这里不需要再靠 object_exists 去省事——
|
||||
少一个分支，行为也更好预测。
|
||||
"""
|
||||
|
||||
def __init__(self, region: str, bucket: str = DEFAULT_BUCKET,
|
||||
prefix: str = DEFAULT_PREFIX,
|
||||
secret_id: str = "", secret_key: str = "",
|
||||
url_style: str = "cos", scheme: str = "https") -> None:
|
||||
secret_id = secret_id or os.environ.get("COS_SECRET_ID", "")
|
||||
secret_key = secret_key or os.environ.get("COS_SECRET_KEY", "")
|
||||
if not secret_id or not secret_key:
|
||||
raise CosError(
|
||||
"缺 COS 凭证。设环境变量 COS_SECRET_ID / COS_SECRET_KEY，"
|
||||
"别写在命令行里（会留在 shell history）。")
|
||||
|
||||
from qcloud_cos import CosConfig, CosS3Client
|
||||
self.bucket = bucket
|
||||
self.region = region
|
||||
self.prefix = prefix.strip("/") + "/" if prefix.strip("/") else ""
|
||||
self.url_style = url_style
|
||||
self._client = CosS3Client(CosConfig(
|
||||
Region=region, SecretId=secret_id, SecretKey=secret_key, Scheme=scheme))
|
||||
self._lock = threading.Lock()
|
||||
|
||||
def key_for(self, name: str) -> str:
|
||||
return f"{self.prefix}{name}"
|
||||
|
||||
def url_for(self, key: str) -> str:
|
||||
return object_url(key, self.bucket, self.region, self.url_style)
|
||||
|
||||
def remote_size(self, key: str) -> Optional[int]:
|
||||
try:
|
||||
return int(self._client.head_object(Bucket=self.bucket, Key=key)["Content-Length"])
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def put(self, local: Path, key: str) -> tuple[str, int]:
|
||||
"""上传并校验。返回 (url, 字节数)。校验不过抛 SizeMismatch。"""
|
||||
local = Path(local)
|
||||
size = local.stat().st_size
|
||||
if size == 0:
|
||||
raise CosError(f"{local.name} 是空文件，拒绝上传")
|
||||
|
||||
# EnableMD5 关掉是为了速度；正确性靠上传后的 head_object 校验兜底
|
||||
self._client.upload_file(
|
||||
Bucket=self.bucket, Key=key, LocalFilePath=str(local),
|
||||
EnableMD5=False, PartSize=10, MAXThread=2,
|
||||
)
|
||||
|
||||
remote = self.remote_size(key)
|
||||
if remote is None:
|
||||
raise SizeMismatch(f"{key} 上传后 head_object 拿不到，判定失败")
|
||||
if remote != size:
|
||||
raise SizeMismatch(f"{key} 大小不符：COS={remote} 本地={size}")
|
||||
return self.url_for(key), size
|
||||
179
douyin/crypto.py
Normal file
179
douyin/crypto.py
Normal file
@@ -0,0 +1,179 @@
|
||||
"""a_bogus 依赖的底层密码学原语：SM3、RC4、自定义 Base64。
|
||||
|
||||
这三样都是公开算法，与抖音无关，可以独立验证正确性：
|
||||
- SM3 见 GB/T 32905-2016，本文件底部带官方测试向量自检
|
||||
- RC4 标准流密码
|
||||
- 自定义 Base64 只是把标准码表换成抖音自己那套字符顺序
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Iterable, List, Sequence
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# SM3
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
_IV = [
|
||||
0x7380166F, 0x4914B2B9, 0x172442D7, 0xDA8A0600,
|
||||
0xA96F30BC, 0x163138AA, 0xE38DEE4D, 0xB0FB0E4E,
|
||||
]
|
||||
_MASK = 0xFFFFFFFF
|
||||
|
||||
|
||||
def _rotl(x: int, n: int) -> int:
|
||||
n &= 31
|
||||
return ((x << n) | (x >> (32 - n))) & _MASK
|
||||
|
||||
|
||||
def _ff(j: int, x: int, y: int, z: int) -> int:
|
||||
return (x ^ y ^ z) if j < 16 else ((x & y) | (x & z) | (y & z))
|
||||
|
||||
|
||||
def _gg(j: int, x: int, y: int, z: int) -> int:
|
||||
return (x ^ y ^ z) if j < 16 else ((x & y) | ((~x & _MASK) & z))
|
||||
|
||||
|
||||
def _p0(x: int) -> int:
|
||||
return x ^ _rotl(x, 9) ^ _rotl(x, 17)
|
||||
|
||||
|
||||
def _p1(x: int) -> int:
|
||||
return x ^ _rotl(x, 15) ^ _rotl(x, 23)
|
||||
|
||||
|
||||
def _cf(v: Sequence[int], block: bytes) -> List[int]:
|
||||
"""压缩函数 CF(V, B)。"""
|
||||
w = [int.from_bytes(block[i * 4:i * 4 + 4], "big") for i in range(16)]
|
||||
for j in range(16, 68):
|
||||
w.append(
|
||||
_p1(w[j - 16] ^ w[j - 9] ^ _rotl(w[j - 3], 15))
|
||||
^ _rotl(w[j - 13], 7)
|
||||
^ w[j - 6]
|
||||
)
|
||||
w1 = [w[j] ^ w[j + 4] for j in range(64)]
|
||||
|
||||
a, b, c, d, e, f, g, h = v
|
||||
for j in range(64):
|
||||
t = 0x79CC4519 if j < 16 else 0x7A879D8A
|
||||
ss1 = _rotl((_rotl(a, 12) + e + _rotl(t, j)) & _MASK, 7)
|
||||
ss2 = ss1 ^ _rotl(a, 12)
|
||||
tt1 = (_ff(j, a, b, c) + d + ss2 + w1[j]) & _MASK
|
||||
tt2 = (_gg(j, e, f, g) + h + ss1 + w[j]) & _MASK
|
||||
d = c
|
||||
c = _rotl(b, 9)
|
||||
b = a
|
||||
a = tt1
|
||||
h = g
|
||||
g = _rotl(f, 19)
|
||||
f = e
|
||||
e = _p0(tt2)
|
||||
|
||||
return [x ^ y for x, y in zip(v, (a, b, c, d, e, f, g, h))]
|
||||
|
||||
|
||||
def sm3(data: bytes) -> bytes:
|
||||
"""SM3 摘要，返回 32 字节。"""
|
||||
bit_len = len(data) * 8
|
||||
padded = data + b"\x80"
|
||||
padded += b"\x00" * ((56 - len(padded) % 64) % 64)
|
||||
padded += bit_len.to_bytes(8, "big")
|
||||
|
||||
v = list(_IV)
|
||||
for i in range(0, len(padded), 64):
|
||||
v = _cf(v, padded[i:i + 64])
|
||||
return b"".join(x.to_bytes(4, "big") for x in v)
|
||||
|
||||
|
||||
def sm3_to_list(data: bytes | Iterable[int]) -> List[int]:
|
||||
"""SM3 摘要，返回 32 个 int（a_bogus 内部按字节数组传递）。"""
|
||||
if not isinstance(data, (bytes, bytearray)):
|
||||
data = bytes(bytearray(data))
|
||||
return list(sm3(bytes(data)))
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# RC4
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
def rc4(key: bytes, data: bytes) -> bytes:
|
||||
"""标准 RC4。加解密同一函数。"""
|
||||
s = list(range(256))
|
||||
j = 0
|
||||
klen = len(key)
|
||||
for i in range(256):
|
||||
j = (j + s[i] + key[i % klen]) & 0xFF
|
||||
s[i], s[j] = s[j], s[i]
|
||||
|
||||
out = bytearray(len(data))
|
||||
i = j = 0
|
||||
for n, ch in enumerate(data):
|
||||
i = (i + 1) & 0xFF
|
||||
j = (j + s[i]) & 0xFF
|
||||
s[i], s[j] = s[j], s[i]
|
||||
out[n] = ch ^ s[(s[i] + s[j]) & 0xFF]
|
||||
return bytes(out)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 自定义码表 Base64
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
STD_ALPHABET = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/="
|
||||
|
||||
|
||||
def b64_encode(data: bytes, alphabet: str, pad: bool = True) -> str:
|
||||
"""按给定 64(+1) 字符码表做 Base64 编码。
|
||||
|
||||
alphabet 长度为 64 时不补位；为 65 时最后一位作为填充字符。
|
||||
"""
|
||||
if len(alphabet) not in (64, 65):
|
||||
raise ValueError(f"码表长度必须是 64 或 65，收到 {len(alphabet)}")
|
||||
fill = alphabet[64] if len(alphabet) == 65 else ""
|
||||
|
||||
out: List[str] = []
|
||||
for i in range(0, len(data), 3):
|
||||
chunk = data[i:i + 3]
|
||||
n = int.from_bytes(chunk + b"\x00" * (3 - len(chunk)), "big")
|
||||
quad = [alphabet[(n >> shift) & 0x3F] for shift in (18, 12, 6, 0)]
|
||||
keep = len(chunk) + 1 # 1字节->2字符, 2字节->3字符, 3字节->4字符
|
||||
out.extend(quad[:keep])
|
||||
if pad and fill:
|
||||
out.extend(fill * (4 - keep))
|
||||
return "".join(out)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 自检
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
# GB/T 32905-2016 附录 A 给出的两组标准测试向量
|
||||
_SM3_VECTORS = [
|
||||
(b"abc",
|
||||
"66c7f0f462eeedd9d1f2d46bdc10e4e24167c4875cf2f7a2297da02b8f4ba8e0"),
|
||||
(b"abcd" * 16,
|
||||
"debe9ff92275b8a138604889c18e5a4d6fdb70e5387e5765293dcba39c0c5732"),
|
||||
]
|
||||
|
||||
|
||||
def self_test() -> None:
|
||||
"""跑通即说明 SM3/RC4/Base64 实现无误。"""
|
||||
for msg, want in _SM3_VECTORS:
|
||||
got = sm3(msg).hex()
|
||||
assert got == want, f"SM3({msg[:8]!r}...) = {got}, 期望 {want}"
|
||||
|
||||
# RC4 官方测试向量：key="Key", plaintext="Plaintext" -> BBF316E8D940AF0AD3
|
||||
assert rc4(b"Key", b"Plaintext").hex().upper() == "BBF316E8D940AF0AD3"
|
||||
|
||||
# 用标准码表时，自定义 Base64 应与 base64 模块完全一致
|
||||
import base64
|
||||
import os
|
||||
for n in range(1, 40):
|
||||
raw = os.urandom(n)
|
||||
assert b64_encode(raw, STD_ALPHABET) == base64.b64encode(raw).decode()
|
||||
|
||||
print("crypto self-test OK: SM3 / RC4 / custom-base64 均通过")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
self_test()
|
||||
174
douyin/download.py
Normal file
174
douyin/download.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""断点续传下载 + 已下载去重。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Dict, Iterable, Optional
|
||||
|
||||
import requests
|
||||
|
||||
CHUNK = 256 * 1024
|
||||
|
||||
# Windows 文件名非法字符，外加控制字符
|
||||
_ILLEGAL = re.compile(r'[\\/:*?"<>|\x00-\x1f]')
|
||||
|
||||
|
||||
def safe_filename(name: str, max_len: int = 80) -> str:
|
||||
name = _ILLEGAL.sub("_", name).strip(" .")
|
||||
name = re.sub(r"\s+", " ", name)
|
||||
if len(name) > max_len:
|
||||
name = name[:max_len].rstrip()
|
||||
return name or "untitled"
|
||||
|
||||
|
||||
class Registry:
|
||||
"""记录已下载的 aweme_id，避免重复下载。
|
||||
|
||||
批量场景下会被多线程写，所以加锁；并且不再每条都落盘（5000 条时
|
||||
每次全量 dump 是 O(n^2) 的 IO），改成攒够 save_every 条再写，
|
||||
结束时用 flush() 收尾。文件名本身就是 aweme_id，记录丢了也能靠
|
||||
目录重建，所以这个折中是安全的。
|
||||
"""
|
||||
|
||||
def __init__(self, path: Path, save_every: int = 1) -> None:
|
||||
self.path = path
|
||||
self.save_every = save_every
|
||||
self._lock = threading.Lock()
|
||||
self._pending = 0
|
||||
self._data: Dict[str, dict] = {}
|
||||
if path.exists():
|
||||
try:
|
||||
self._data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (ValueError, OSError):
|
||||
self._data = {}
|
||||
|
||||
def __contains__(self, aweme_id: str) -> bool:
|
||||
entry = self._data.get(aweme_id)
|
||||
if not entry:
|
||||
return False
|
||||
# 记录在但文件被手动删了，就当没下过
|
||||
return Path(entry.get("file", "")).exists()
|
||||
|
||||
def get(self, aweme_id: str) -> Optional[dict]:
|
||||
return self._data.get(aweme_id)
|
||||
|
||||
def add(self, aweme_id: str, file: Path, size: int, desc: str = "",
|
||||
author: str = "", uri: str = "") -> None:
|
||||
"""记一条。文件名只是 aweme_id，作者/标题这些就全靠这里留存。"""
|
||||
with self._lock:
|
||||
self._data[aweme_id] = {
|
||||
"file": str(file),
|
||||
"size": size,
|
||||
"author": author,
|
||||
"desc": desc,
|
||||
"uri": uri,
|
||||
"time": int(time.time()),
|
||||
}
|
||||
self._pending += 1
|
||||
if self._pending >= self.save_every:
|
||||
self._save_locked()
|
||||
|
||||
def flush(self) -> None:
|
||||
"""强制落盘。批量结束时调用。"""
|
||||
with self._lock:
|
||||
if self._pending:
|
||||
self._save_locked()
|
||||
|
||||
def save(self) -> None:
|
||||
with self._lock:
|
||||
self._save_locked()
|
||||
|
||||
def _save_locked(self) -> None:
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
tmp = self.path.with_suffix(self.path.suffix + ".tmp")
|
||||
tmp.write_text(json.dumps(self._data, ensure_ascii=False, indent=2),
|
||||
encoding="utf-8")
|
||||
tmp.replace(self.path)
|
||||
self._pending = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
class DownloadResult:
|
||||
path: Path
|
||||
size: int
|
||||
resumed: bool
|
||||
skipped: bool = False
|
||||
|
||||
|
||||
def _human(n: float) -> str:
|
||||
for unit in ("B", "KB", "MB", "GB"):
|
||||
if n < 1024:
|
||||
return f"{n:.1f}{unit}"
|
||||
n /= 1024
|
||||
return f"{n:.1f}TB"
|
||||
|
||||
|
||||
def download(session: requests.Session, urls: Iterable[str], dest: Path,
|
||||
timeout: float = 30.0, retries: int = 4,
|
||||
progress: bool = True) -> DownloadResult:
|
||||
"""下载到 dest，支持 Range 续传；urls 为多个备用直链，逐个尝试。
|
||||
|
||||
抖音的视频直链会 302 到 CDN，且校验 Referer/UA，所以必须带上原 session 的头。
|
||||
"""
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
part = dest.with_suffix(dest.suffix + ".part")
|
||||
|
||||
if dest.exists() and dest.stat().st_size > 0:
|
||||
return DownloadResult(dest, dest.stat().st_size, resumed=False, skipped=True)
|
||||
|
||||
url_list = [u for u in urls if u]
|
||||
if not url_list:
|
||||
raise ValueError("没有可用的下载直链")
|
||||
|
||||
last_err: Optional[Exception] = None
|
||||
for url in url_list:
|
||||
for attempt in range(1, retries + 1):
|
||||
done = part.stat().st_size if part.exists() else 0
|
||||
headers = {"Range": f"bytes={done}-"} if done else {}
|
||||
try:
|
||||
with session.get(url, headers=headers, stream=True,
|
||||
timeout=timeout, allow_redirects=True) as r:
|
||||
if done and r.status_code == 200:
|
||||
# 服务端不认 Range，只能从头来
|
||||
done = 0
|
||||
part.unlink(missing_ok=True)
|
||||
elif done and r.status_code != 206:
|
||||
r.raise_for_status()
|
||||
else:
|
||||
r.raise_for_status()
|
||||
|
||||
total = int(r.headers.get("Content-Length") or 0) + done
|
||||
mode = "ab" if done else "wb"
|
||||
with open(part, mode) as f:
|
||||
for chunk in r.iter_content(CHUNK):
|
||||
if not chunk:
|
||||
continue
|
||||
f.write(chunk)
|
||||
done += len(chunk)
|
||||
if progress:
|
||||
pct = f"{done / total * 100:5.1f}%" if total else " ? "
|
||||
print(f"\r {pct} {_human(done)}"
|
||||
+ (f" / {_human(total)}" if total else ""),
|
||||
end="", flush=True)
|
||||
if progress:
|
||||
print()
|
||||
|
||||
if total and done < total:
|
||||
raise IOError(f"下载不完整: {done}/{total}")
|
||||
|
||||
part.replace(dest)
|
||||
return DownloadResult(dest, done, resumed=bool(headers))
|
||||
|
||||
except (requests.RequestException, IOError) as exc:
|
||||
last_err = exc
|
||||
if progress:
|
||||
print(f"\r 第 {attempt}/{retries} 次失败: {exc}")
|
||||
time.sleep(min(2 ** attempt, 8))
|
||||
|
||||
raise RuntimeError(f"所有直链均下载失败，最后一个错误: {last_err}")
|
||||
252
douyin/meta.py
Normal file
252
douyin/meta.py
Normal file
@@ -0,0 +1,252 @@
|
||||
"""元信息：标题/标签拆分，以及 records.jsonl 的追加写。
|
||||
|
||||
records.jsonl 是流水线唯一的落点，一条视频一行，raw 也内嵌在里面，
|
||||
所以单行可能几十 KB、整个文件几百 MB。由此有两条硬性约定：
|
||||
|
||||
1. 读取一律逐行流式，任何地方都不许把整个文件读进内存。
|
||||
2. `aweme_id` 必须是每行的第一个字段。续跑扫描靠 _ID_RE 从行首直接
|
||||
抠 id，不解析整行 JSON —— 400 MB 的差别是 ~1 秒 对 十几秒。
|
||||
重排字段不会报错，只会让启动悄悄变慢，所以这是契约不是排版。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import re
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Sequence, Tuple
|
||||
|
||||
# 行首抠 id，不解析整行
|
||||
_ID_RE = re.compile(r'^\{"aweme_id":\s*"(\d+)"')
|
||||
|
||||
# 一个标签：# 后面跟到下一个空白或下一个 # 为止。
|
||||
# 抖音有 "#四川话#方言" 这种连写，所以 # 也要作为终止符。
|
||||
_TAG_RE = re.compile(r"#([^\s#]+)")
|
||||
|
||||
|
||||
# 标签两端常粘着括号、引号、逗号之类的标点，剥掉
|
||||
_EDGE_PUNCT = "（）()[]【】{}「」<>《》\"'“”‘’,，.。!！?？:：;；、~～-—_/\\|"
|
||||
|
||||
|
||||
def _clean_tag(name: str) -> str:
|
||||
"""归一化一个标签候选，判定不是标签就返回空串。"""
|
||||
name = (name or "").strip().strip(_EDGE_PUNCT).strip()
|
||||
# 带 @ 的是提及，不是标签。实测 desc 里有作者手打的
|
||||
# "#(提示词出自@青木"，抖音自己也没把它收进 text_extra。
|
||||
if not name or "@" in name:
|
||||
return ""
|
||||
return name
|
||||
|
||||
|
||||
def _dedupe(items) -> List[str]:
|
||||
"""按大小写不敏感去重，保留首次出现的写法。
|
||||
|
||||
抖音把 text_extra 里的标签统一转成了小写，desc 里却是作者的原始大小写。
|
||||
实测同一条视频拿到 ['一站式AI创作工作台', ..., '一站式ai创作工作台', ...]，
|
||||
同一个标签存了两份。desc 排在前面，所以保留的是原始大小写。
|
||||
"""
|
||||
seen, out = set(), []
|
||||
for x in items:
|
||||
x = _clean_tag(x)
|
||||
if not x:
|
||||
continue
|
||||
k = x.casefold()
|
||||
if k not in seen:
|
||||
seen.add(k)
|
||||
out.append(x)
|
||||
return out
|
||||
|
||||
|
||||
def split_desc(desc: Optional[str],
|
||||
text_extra: Optional[Sequence[Dict[str, Any]]] = None,
|
||||
) -> Tuple[str, List[str]]:
|
||||
"""把 desc 拆成 (纯标题, 标签列表)。
|
||||
|
||||
标签以 desc 为准（正则取全集），text_extra 只用来补充。实测 12 条样本
|
||||
里有 2 条的标签在 desc 里有、text_extra 里没有，以 text_extra 为准会
|
||||
静默丢标签。
|
||||
|
||||
剥标题时优先用 text_extra 给的 start/end 偏移——它能认出 "C# 教程"
|
||||
里的 # 不是标签。但偏移不一定可信（越界、错位都见过），所以每个都先
|
||||
验证 desc[start:end] 确实是 "#" + hashtag_name，验不过就退回正则。
|
||||
|
||||
兜底删的是 "#" + 已知标签名 这个精确串，不是任意 #，所以 "C# 教程"
|
||||
是安全的。唯一的有损场景：标签名以 "#名字" 的形式也出现在正文里，
|
||||
那两处会一起被删掉（见 test_split_repeated_tagname_in_body_is_lossy）。
|
||||
"""
|
||||
desc = (desc or "").strip()
|
||||
entries = list(text_extra or [])
|
||||
|
||||
# 两件事要分开：
|
||||
# raw_hits —— desc 里 # 后面的原文，用来剥标题（必须能字面匹配上）
|
||||
# tags —— 清洗+去重后的结果，是对外的字段
|
||||
# 混用会导致 "#（括号标签）" 剥不掉：清洗后的名字没有那对括号，
|
||||
# replace 匹配不上，标签就留在标题里了。
|
||||
raw_hits = _TAG_RE.findall(desc)
|
||||
from_te = [e.get("hashtag_name") for e in entries if e.get("hashtag_name")]
|
||||
tags = _dedupe(raw_hits + from_te)
|
||||
|
||||
if not desc:
|
||||
return "", tags
|
||||
|
||||
# 能用偏移剥掉的标签，逐个记下要挖掉的区间
|
||||
spans: List[Tuple[int, int]] = []
|
||||
covered = set()
|
||||
for e in entries:
|
||||
name = e.get("hashtag_name")
|
||||
start, end = e.get("start"), e.get("end")
|
||||
if not name or not isinstance(start, int) or not isinstance(end, int):
|
||||
continue
|
||||
if not (0 <= start < end <= len(desc)):
|
||||
continue
|
||||
if desc[start:end] != "#" + name:
|
||||
continue
|
||||
spans.append((start, end))
|
||||
covered.add(name)
|
||||
|
||||
title = desc
|
||||
for start, end in sorted(spans, reverse=True): # 从后往前挖，偏移不会串位
|
||||
title = title[:start] + title[end:]
|
||||
|
||||
# 偏移没盖到的，用 desc 里的原文删（不是清洗过的 tags）
|
||||
for hit in raw_hits:
|
||||
if hit not in covered:
|
||||
title = title.replace("#" + hit, "")
|
||||
|
||||
return re.sub(r"\s+", " ", title).strip(), tags
|
||||
|
||||
|
||||
def build_record(aweme, *, duration_sec: float, cos_mp4: str, cos_mp3: str,
|
||||
size_mp4: int, size_mp3: int,
|
||||
source_url: str = "") -> Dict[str, Any]:
|
||||
"""组装一条 record。字段顺序是有意的，aweme_id 必须在最前。"""
|
||||
raw = aweme.raw or {}
|
||||
title, tags = split_desc(aweme.desc, raw.get("text_extra"))
|
||||
author = raw.get("author") or {}
|
||||
stat = raw.get("statistics") or {}
|
||||
music = raw.get("music") or {}
|
||||
video = raw.get("video") or {}
|
||||
ct = int(aweme.create_time or 0)
|
||||
|
||||
return {
|
||||
"aweme_id": str(aweme.aweme_id),
|
||||
"title": title,
|
||||
"desc": aweme.desc,
|
||||
"tags": tags,
|
||||
"author": {
|
||||
"nickname": aweme.author,
|
||||
"uid": str(author.get("uid") or ""),
|
||||
"sec_uid": str(author.get("sec_uid") or ""),
|
||||
},
|
||||
"create_time": ct,
|
||||
"create_time_str": (time.strftime("%Y-%m-%d %H:%M:%S", time.localtime(ct))
|
||||
if ct else ""),
|
||||
"duration_sec": duration_sec,
|
||||
# 服务端声明值，与 ffprobe 解出来的不等（实测 41750 vs 41703 ms）。
|
||||
# 两个都留着，切分要用 duration_sec。
|
||||
"duration_ms_api": video.get("duration") or raw.get("duration") or 0,
|
||||
"stats": {
|
||||
"digg": stat.get("digg_count", 0),
|
||||
"comment": stat.get("comment_count", 0),
|
||||
"collect": stat.get("collect_count", 0),
|
||||
"share": stat.get("share_count", 0),
|
||||
"play": stat.get("play_count", 0),
|
||||
},
|
||||
"music": {
|
||||
"title": (music.get("title") or "").strip(),
|
||||
"author": (music.get("author") or "").strip(),
|
||||
"url": aweme.music_url or "",
|
||||
},
|
||||
"cover_url": aweme.cover_urls[0] if aweme.cover_urls else "",
|
||||
"video": {
|
||||
"uri": aweme.uri,
|
||||
"width": video.get("width") or 0,
|
||||
"height": video.get("height") or 0,
|
||||
},
|
||||
"cos_mp4": cos_mp4,
|
||||
"cos_mp3": cos_mp3,
|
||||
"size_mp4": size_mp4,
|
||||
"size_mp3": size_mp3,
|
||||
"source_url": source_url or f"https://www.douyin.com/video/{aweme.aweme_id}",
|
||||
"fetched_at": int(time.time()),
|
||||
"raw": raw,
|
||||
}
|
||||
|
||||
|
||||
class RecordLog:
|
||||
"""records.jsonl 的追加写 + 已完成 id 集合。多线程安全。"""
|
||||
|
||||
def __init__(self, path: Path) -> None:
|
||||
self.path = Path(path)
|
||||
self._lock = threading.Lock()
|
||||
self._ids = self._scan()
|
||||
|
||||
def _scan(self) -> set:
|
||||
"""流式扫已有 id。只看行首，不解析整行。"""
|
||||
ids = set()
|
||||
if not self.path.exists():
|
||||
return ids
|
||||
with open(self.path, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
m = _ID_RE.match(line)
|
||||
if m:
|
||||
ids.add(m.group(1))
|
||||
return ids
|
||||
|
||||
def __contains__(self, aweme_id: str) -> bool:
|
||||
return str(aweme_id) in self._ids
|
||||
|
||||
def __len__(self) -> int:
|
||||
return len(self._ids)
|
||||
|
||||
@property
|
||||
def ids(self) -> set:
|
||||
return set(self._ids)
|
||||
|
||||
def _needs_newline(self) -> bool:
|
||||
"""上一次可能被 Ctrl-C 打断，留下没换行的半行。"""
|
||||
try:
|
||||
if self.path.stat().st_size == 0:
|
||||
return False
|
||||
except OSError:
|
||||
return False
|
||||
with open(self.path, "rb") as f:
|
||||
f.seek(-1, 2)
|
||||
return f.read(1) != b"\n"
|
||||
|
||||
def write(self, record: Dict[str, Any]) -> bool:
|
||||
"""追加一条。已存在同 id 则跳过，返回 False。"""
|
||||
aweme_id = str(record.get("aweme_id") or "")
|
||||
if not aweme_id:
|
||||
raise ValueError("record 缺 aweme_id")
|
||||
|
||||
with self._lock:
|
||||
if aweme_id in self._ids:
|
||||
return False
|
||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
||||
prefix = "\n" if self._needs_newline() else ""
|
||||
line = json.dumps(record, ensure_ascii=False, separators=(",", ":"))
|
||||
if not line.startswith('{"aweme_id":'):
|
||||
raise ValueError("aweme_id 必须是第一个字段，见本模块开头的约定")
|
||||
with open(self.path, "a", encoding="utf-8") as f:
|
||||
f.write(prefix + line + "\n")
|
||||
self._ids.add(aweme_id)
|
||||
return True
|
||||
|
||||
|
||||
def iter_records(path: Path):
|
||||
"""逐行读 records.jsonl，跳过坏行（可能有被打断留下的半行）。"""
|
||||
path = Path(path)
|
||||
if not path.exists():
|
||||
return
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
yield json.loads(line)
|
||||
except ValueError:
|
||||
continue
|
||||
156
douyin/signer.py
Normal file
156
douyin/signer.py
Normal file
@@ -0,0 +1,156 @@
|
||||
"""a_bogus 签名器。
|
||||
|
||||
两种后端，接口相同、可互换：
|
||||
|
||||
NodeSigner 调本地 node_signer（真实 webmssdk 跑在伪造浏览器环境里）。
|
||||
当前唯一可用的实现，已实测能被抖音接受。
|
||||
ABogusSigner 纯 Python 实现。底层 SM3/RC4/自定义 Base64 已在 crypto.py
|
||||
验证通过，剩字节拼装顺序待从 ab.js 逆出来。
|
||||
|
||||
签名器同时决定 User-Agent：a_bogus 把 UA 编进签名，服务端会交叉校验，
|
||||
所以请求头的 UA 必须与签名时用的完全一致。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Protocol, Sequence
|
||||
|
||||
|
||||
class SignError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
class SignerNotConfigured(SignError):
|
||||
pass
|
||||
|
||||
|
||||
class Signer(Protocol):
|
||||
"""client 只依赖这两个成员。"""
|
||||
|
||||
@property
|
||||
def user_agent(self) -> str:
|
||||
"""签名时使用的 UA，请求头必须用同一个。"""
|
||||
...
|
||||
|
||||
def sign_url(self, url: str, fp: str = "") -> str:
|
||||
"""给一条裸 URL，返回带 a_bogus 的完整 URL。"""
|
||||
...
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Node 后端
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
class NodeSigner:
|
||||
"""通过 node_signer/sign.js 签名。
|
||||
|
||||
node 启动 + 加载 347KB 的 SDK 大约 0.7s，所以支持 batch 一次签多条，
|
||||
把这份固定开销摊掉。
|
||||
"""
|
||||
|
||||
def __init__(self, signer_dir: str | Path, node: str = "node",
|
||||
timeout: float = 120.0) -> None:
|
||||
self.dir = Path(signer_dir)
|
||||
self.node = node
|
||||
self.timeout = timeout
|
||||
self._ua: str | None = None
|
||||
|
||||
script = self.dir / "sign.js"
|
||||
if not script.exists():
|
||||
raise SignerNotConfigured(f"找不到签名脚本: {script}")
|
||||
|
||||
def _run(self, args: Sequence[str], stdin: str | None = None) -> dict:
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[self.node, "sign.js", *args], cwd=self.dir, input=stdin,
|
||||
capture_output=True, text=True, encoding="utf-8",
|
||||
timeout=self.timeout,
|
||||
)
|
||||
except FileNotFoundError as exc:
|
||||
raise SignerNotConfigured(f"找不到 node 可执行文件: {self.node}") from exc
|
||||
except subprocess.TimeoutExpired as exc:
|
||||
raise SignError(f"签名超时（>{self.timeout}s）") from exc
|
||||
|
||||
out = (proc.stdout or "").strip()
|
||||
if not out:
|
||||
raise SignError(f"签名器无输出，rc={proc.returncode}，stderr={proc.stderr[:400]}")
|
||||
try:
|
||||
# sign.js 只在最后一行吐 JSON，前面可能有 SDK 自己的噪声
|
||||
return json.loads(out.splitlines()[-1])
|
||||
except ValueError as exc:
|
||||
raise SignError(f"签名器输出不是 JSON: {out[:300]!r}") from exc
|
||||
|
||||
@property
|
||||
def user_agent(self) -> str:
|
||||
if self._ua is None:
|
||||
probe = self._run(["https://www.douyin.com/aweme/v1/web/aweme/detail/?aweme_id=1", ""])
|
||||
self._ua = probe.get("userAgent") or ""
|
||||
if not self._ua:
|
||||
raise SignError("签名器没有返回 userAgent")
|
||||
return self._ua
|
||||
|
||||
def sign_url(self, url: str, fp: str = "") -> str:
|
||||
res = self._run([url, fp])
|
||||
if not res.get("ok"):
|
||||
raise SignError(f"签名失败: {res.get('error')}")
|
||||
self._ua = res.get("userAgent") or self._ua
|
||||
return res["signedUrl"]
|
||||
|
||||
def sign_many(self, urls: List[str], fp: str = "") -> List[str]:
|
||||
"""一次 node 启动签完多条。"""
|
||||
if not urls:
|
||||
return []
|
||||
res = self._run(["--batch", fp], stdin=json.dumps(urls))
|
||||
if not res.get("ok"):
|
||||
raise SignError(f"批量签名失败: {res.get('error')}")
|
||||
self._ua = res.get("userAgent") or self._ua
|
||||
signed = []
|
||||
for item in res["results"]:
|
||||
if not item.get("ok"):
|
||||
raise SignError(f"某条签名失败: {item.get('error')} url={item.get('rawUrl')}")
|
||||
signed.append(item["signedUrl"])
|
||||
return signed
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 纯 Python 后端（待补）
|
||||
# --------------------------------------------------------------------------- #
|
||||
|
||||
@dataclass
|
||||
class ABogusConfig:
|
||||
"""a_bogus 算法常量。必须与线上 webmssdk.js 当前版本一致。"""
|
||||
|
||||
alphabets: Dict[str, str] = field(default_factory=dict)
|
||||
ua_key: bytes = b""
|
||||
arguments: List[int] = field(default_factory=list)
|
||||
browser_fp: str = ""
|
||||
end_string: str = ""
|
||||
user_agent: str = ""
|
||||
|
||||
def validate(self) -> None:
|
||||
missing = [n for n in ("alphabets", "ua_key", "arguments",
|
||||
"browser_fp", "end_string", "user_agent")
|
||||
if not getattr(self, n)]
|
||||
if missing:
|
||||
raise SignerNotConfigured("a_bogus 常量未配置，缺少: " + ", ".join(missing))
|
||||
|
||||
|
||||
class ABogusSigner:
|
||||
"""纯 Python a_bogus。crypto.py 里的 SM3/RC4/Base64 已验证可用。"""
|
||||
|
||||
def __init__(self, config: ABogusConfig) -> None:
|
||||
config.validate()
|
||||
self.config = config
|
||||
|
||||
@property
|
||||
def user_agent(self) -> str:
|
||||
return self.config.user_agent
|
||||
|
||||
def sign_url(self, url: str, fp: str = "") -> str:
|
||||
raise NotImplementedError(
|
||||
"a_bogus 字节拼装顺序待从 ab.js 逆出。目前请用 NodeSigner。"
|
||||
)
|
||||
85
douyin/urls.py
Normal file
85
douyin/urls.py
Normal file
@@ -0,0 +1,85 @@
|
||||
"""链接解析：把用户随手粘的东西还原成 aweme_id。
|
||||
|
||||
支持的输入形态：
|
||||
7xxxxxxxxxxxxxxxxxx 裸 id
|
||||
https://v.douyin.com/xxxxxxx/ 分享短链（需要一次网络跳转）
|
||||
4.53 复制打开抖音... https://v.douyin.com/xxx/ ... 整段分享口令
|
||||
https://www.douyin.com/video/7xxxxxx
|
||||
https://www.douyin.com/note/7xxxxxx 图文
|
||||
https://www.douyin.com/?modal_id=7xxxxxx
|
||||
https://www.iesdouyin.com/share/video/7xxxxxx/
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
|
||||
# 从一段中文分享口令里把 url 抠出来
|
||||
URL_RE = re.compile(r"https?://[-\w./?=&%#:]+")
|
||||
|
||||
SHORT_HOSTS = ("v.douyin.com", "v.iesdouyin.com")
|
||||
|
||||
# 按优先级尝试的 id 提取规则
|
||||
ID_PATTERNS = (
|
||||
re.compile(r"/(?:video|note|slides|share/video|share/note)/(\d{15,25})"),
|
||||
re.compile(r"[?&](?:modal_id|aweme_id|item_ids|vid)=(\d{15,25})"),
|
||||
)
|
||||
|
||||
BARE_ID_RE = re.compile(r"^\d{15,25}$")
|
||||
|
||||
|
||||
class ParseError(ValueError):
|
||||
"""输入里找不到可用的 aweme_id。"""
|
||||
|
||||
|
||||
def _first_url(text: str) -> Optional[str]:
|
||||
m = URL_RE.search(text)
|
||||
return m.group(0) if m else None
|
||||
|
||||
|
||||
def _extract_id(url: str) -> Optional[str]:
|
||||
for pat in ID_PATTERNS:
|
||||
m = pat.search(url)
|
||||
if m:
|
||||
return m.group(1)
|
||||
return None
|
||||
|
||||
|
||||
def resolve_short_link(url: str, session: requests.Session, timeout: float = 15.0) -> str:
|
||||
"""跟随 v.douyin.com 短链跳转，返回落地长链。"""
|
||||
resp = session.get(url, timeout=timeout, allow_redirects=True)
|
||||
return resp.url
|
||||
|
||||
|
||||
def extract_aweme_id(raw: str, session: Optional[requests.Session] = None) -> str:
|
||||
"""从任意输入中解析出 aweme_id，必要时联网还原短链。"""
|
||||
raw = raw.strip()
|
||||
if BARE_ID_RE.match(raw):
|
||||
return raw
|
||||
|
||||
url = _first_url(raw)
|
||||
if not url:
|
||||
raise ParseError(f"输入里没找到链接也不是纯数字 id: {raw[:80]!r}")
|
||||
|
||||
aid = _extract_id(url)
|
||||
if aid:
|
||||
return aid
|
||||
|
||||
if any(h in url for h in SHORT_HOSTS):
|
||||
if session is None:
|
||||
session = requests.Session()
|
||||
session.headers["User-Agent"] = (
|
||||
"Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) "
|
||||
"AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 "
|
||||
"Mobile/15E148 Safari/604.1"
|
||||
)
|
||||
final = resolve_short_link(url, session)
|
||||
aid = _extract_id(final)
|
||||
if aid:
|
||||
return aid
|
||||
raise ParseError(f"短链 {url} 跳转到 {final}，但里面没有 aweme_id")
|
||||
|
||||
raise ParseError(f"无法从 {url} 中解析出 aweme_id")
|
||||
106447
lists/ai_118469.txt
Normal file
106447
lists/ai_118469.txt
Normal file
File diff suppressed because it is too large
Load Diff
37703
lists/ai_42391.txt
Normal file
37703
lists/ai_42391.txt
Normal file
File diff suppressed because it is too large
Load Diff
776
lists/ai_57261.txt
Normal file
776
lists/ai_57261.txt
Normal file
@@ -0,0 +1,776 @@
|
||||
https://www.douyin.com/video/6943122417938173219
|
||||
https://www.douyin.com/video/7519393200386723091
|
||||
https://www.douyin.com/video/7520105860204596492
|
||||
https://www.douyin.com/video/7553798797877366062
|
||||
https://www.douyin.com/video/7553888041010548014
|
||||
https://www.douyin.com/video/7554172873657699610
|
||||
https://www.douyin.com/video/7554258413786615049
|
||||
https://www.douyin.com/video/7554547768417586478
|
||||
https://www.douyin.com/video/7554715238831033609
|
||||
https://www.douyin.com/video/7554912956857797939
|
||||
https://www.douyin.com/video/7554985407327243546
|
||||
https://www.douyin.com/video/7555476793818500378
|
||||
https://www.douyin.com/video/7556159053320293658
|
||||
https://www.douyin.com/video/7557202537238056218
|
||||
https://www.douyin.com/video/7557575002803932426
|
||||
https://www.douyin.com/video/7558027934854892809
|
||||
https://www.douyin.com/video/7558340211575409966
|
||||
https://www.douyin.com/video/7560697691928726803
|
||||
https://www.douyin.com/video/7562089250788674870
|
||||
https://www.douyin.com/video/7564399768768630067
|
||||
https://www.douyin.com/video/7564639934120791334
|
||||
https://www.douyin.com/video/7564751054412287258
|
||||
https://www.douyin.com/video/7564963930150374682
|
||||
https://www.douyin.com/video/7565127231735893298
|
||||
https://www.douyin.com/video/7565377771719970094
|
||||
https://www.douyin.com/video/7565702956490738970
|
||||
https://www.douyin.com/video/7565853315041414410
|
||||
https://www.douyin.com/video/7566154309616831782
|
||||
https://www.douyin.com/video/7566232008305757486
|
||||
https://www.douyin.com/video/7566587563641785641
|
||||
https://www.douyin.com/video/7568905480349764902
|
||||
https://www.douyin.com/video/7569632988347026738
|
||||
https://www.douyin.com/video/7570184180492012827
|
||||
https://www.douyin.com/video/7572001072161993427
|
||||
https://www.douyin.com/video/7572375979444751642
|
||||
https://www.douyin.com/video/7572546058958474546
|
||||
https://www.douyin.com/video/7573139707496811822
|
||||
https://www.douyin.com/video/7573235518420672511
|
||||
https://www.douyin.com/video/7573494590506798345
|
||||
https://www.douyin.com/video/7573934793147092233
|
||||
https://www.douyin.com/video/7574237778314414335
|
||||
https://www.douyin.com/video/7574249747658083611
|
||||
https://www.douyin.com/video/7574604143612988723
|
||||
https://www.douyin.com/video/7575569253295855862
|
||||
https://www.douyin.com/video/7576562406902533426
|
||||
https://www.douyin.com/video/7577274848418762030
|
||||
https://www.douyin.com/video/7578033313944718602
|
||||
https://www.douyin.com/video/7578684709299965210
|
||||
https://www.douyin.com/video/7579632942910147894
|
||||
https://www.douyin.com/video/7579885234519428371
|
||||
https://www.douyin.com/video/7580211967593349254
|
||||
https://www.douyin.com/video/7580230312265878799
|
||||
https://www.douyin.com/video/7580552466681367842
|
||||
https://www.douyin.com/video/7580630563740978467
|
||||
https://www.douyin.com/video/7580638698766970218
|
||||
https://www.douyin.com/video/7580998521541758854
|
||||
https://www.douyin.com/video/7581546931097488683
|
||||
https://www.douyin.com/video/7582021145629576494
|
||||
https://www.douyin.com/video/7582089479017876772
|
||||
https://www.douyin.com/video/7582410806042627328
|
||||
https://www.douyin.com/video/7582619119854423339
|
||||
https://www.douyin.com/video/7582831090910383398
|
||||
https://www.douyin.com/video/7583258850694810943
|
||||
https://www.douyin.com/video/7583747521584074020
|
||||
https://www.douyin.com/video/7584028010996600106
|
||||
https://www.douyin.com/video/7585002696411778313
|
||||
https://www.douyin.com/video/7585245325321850147
|
||||
https://www.douyin.com/video/7585250568013171987
|
||||
https://www.douyin.com/video/7585376145323986186
|
||||
https://www.douyin.com/video/7585867986922360070
|
||||
https://www.douyin.com/video/7586983813100703019
|
||||
https://www.douyin.com/video/7587226343230786843
|
||||
https://www.douyin.com/video/7587605794049314099
|
||||
https://www.douyin.com/video/7587646301697871158
|
||||
https://www.douyin.com/video/7587704115791252776
|
||||
https://www.douyin.com/video/7587764268461886756
|
||||
https://www.douyin.com/video/7588206930322311635
|
||||
https://www.douyin.com/video/7588395674413075758
|
||||
https://www.douyin.com/video/7588741008712666419
|
||||
https://www.douyin.com/video/7589089208669605171
|
||||
https://www.douyin.com/video/7589501379968185252
|
||||
https://www.douyin.com/video/7589641625794530602
|
||||
https://www.douyin.com/video/7590229657761402138
|
||||
https://www.douyin.com/video/7590966760849804570
|
||||
https://www.douyin.com/video/7591301498874155236
|
||||
https://www.douyin.com/video/7591398292845238633
|
||||
https://www.douyin.com/video/7592202513383270649
|
||||
https://www.douyin.com/video/7593191283851579497
|
||||
https://www.douyin.com/video/7593229928586366259
|
||||
https://www.douyin.com/video/7594116662737052671
|
||||
https://www.douyin.com/video/7594440381572435209
|
||||
https://www.douyin.com/video/7594775095534162422
|
||||
https://www.douyin.com/video/7594816345863114010
|
||||
https://www.douyin.com/video/7594819503284817215
|
||||
https://www.douyin.com/video/7595153279307807601
|
||||
https://www.douyin.com/video/7595473187765552454
|
||||
https://www.douyin.com/video/7595854224455231849
|
||||
https://www.douyin.com/video/7595893672957300022
|
||||
https://www.douyin.com/video/7596256747297705094
|
||||
https://www.douyin.com/video/7596635881218983914
|
||||
https://www.douyin.com/video/7596998473229052081
|
||||
https://www.douyin.com/video/7597056616344659347
|
||||
https://www.douyin.com/video/7597334429228453147
|
||||
https://www.douyin.com/video/7597361008847818161
|
||||
https://www.douyin.com/video/7597406545911008538
|
||||
https://www.douyin.com/video/7597422956876269482
|
||||
https://www.douyin.com/video/7597427547168656694
|
||||
https://www.douyin.com/video/7597621989137829147
|
||||
https://www.douyin.com/video/7597691941626875323
|
||||
https://www.douyin.com/video/7597817277886596964
|
||||
https://www.douyin.com/video/7597989218547289394
|
||||
https://www.douyin.com/video/7598122640762621219
|
||||
https://www.douyin.com/video/7598134393957299633
|
||||
https://www.douyin.com/video/7598399984580578603
|
||||
https://www.douyin.com/video/7598402730087959834
|
||||
https://www.douyin.com/video/7598445953053461787
|
||||
https://www.douyin.com/video/7598771708765506862
|
||||
https://www.douyin.com/video/7599474626430766382
|
||||
https://www.douyin.com/video/7599911866143395118
|
||||
https://www.douyin.com/video/7600690419866090235
|
||||
https://www.douyin.com/video/7601020122957877733
|
||||
https://www.douyin.com/video/7602102264887523045
|
||||
https://www.douyin.com/video/7602499280254946569
|
||||
https://www.douyin.com/video/7602522606725500202
|
||||
https://www.douyin.com/video/7602836692906560778
|
||||
https://www.douyin.com/video/7603211416799120650
|
||||
https://www.douyin.com/video/7603323120782445862
|
||||
https://www.douyin.com/video/7603360772999122214
|
||||
https://www.douyin.com/video/7603630329710038318
|
||||
https://www.douyin.com/video/7603932736688065802
|
||||
https://www.douyin.com/video/7604054141173828905
|
||||
https://www.douyin.com/video/7604108812122918163
|
||||
https://www.douyin.com/video/7604380466937875738
|
||||
https://www.douyin.com/video/7604727626364339441
|
||||
https://www.douyin.com/video/7604904484086549796
|
||||
https://www.douyin.com/video/7605118039188704522
|
||||
https://www.douyin.com/video/7606182228908920090
|
||||
https://www.douyin.com/video/7606552755691408650
|
||||
https://www.douyin.com/video/7606605160692993314
|
||||
https://www.douyin.com/video/7606625462822931727
|
||||
https://www.douyin.com/video/7606942633830649134
|
||||
https://www.douyin.com/video/7606980034263321892
|
||||
https://www.douyin.com/video/7606995703236444239
|
||||
https://www.douyin.com/video/7607320526113049906
|
||||
https://www.douyin.com/video/7607400979381767443
|
||||
https://www.douyin.com/video/7608043185789741947
|
||||
https://www.douyin.com/video/7608413074671848730
|
||||
https://www.douyin.com/video/7608542392756572358
|
||||
https://www.douyin.com/video/7609180667515063592
|
||||
https://www.douyin.com/video/7609867642543948282
|
||||
https://www.douyin.com/video/7609984557378538790
|
||||
https://www.douyin.com/video/7610146429398502690
|
||||
https://www.douyin.com/video/7611830655957451177
|
||||
https://www.douyin.com/video/7612620405915967999
|
||||
https://www.douyin.com/video/7612626386531978547
|
||||
https://www.douyin.com/video/7612851474276222066
|
||||
https://www.douyin.com/video/7612879197770000370
|
||||
https://www.douyin.com/video/7612917384076067199
|
||||
https://www.douyin.com/video/7613574269888632433
|
||||
https://www.douyin.com/video/7613668005389569306
|
||||
https://www.douyin.com/video/7613778328596839926
|
||||
https://www.douyin.com/video/7614082236790423466
|
||||
https://www.douyin.com/video/7614439185005879659
|
||||
https://www.douyin.com/video/7614482669908380937
|
||||
https://www.douyin.com/video/7614742399537949540
|
||||
https://www.douyin.com/video/7615276283405424298
|
||||
https://www.douyin.com/video/7615549848285079921
|
||||
https://www.douyin.com/video/7615638874379732964
|
||||
https://www.douyin.com/video/7615807141757114873
|
||||
https://www.douyin.com/video/7615816260149053289
|
||||
https://www.douyin.com/video/7615845470321052974
|
||||
https://www.douyin.com/video/7615930582690984617
|
||||
https://www.douyin.com/video/7616185378906643301
|
||||
https://www.douyin.com/video/7616312371875733958
|
||||
https://www.douyin.com/video/7616395708984272767
|
||||
https://www.douyin.com/video/7616413276741586203
|
||||
https://www.douyin.com/video/7616713139165482278
|
||||
https://www.douyin.com/video/7616766668118914303
|
||||
https://www.douyin.com/video/7617325809296002559
|
||||
https://www.douyin.com/video/7617460880041635001
|
||||
https://www.douyin.com/video/7617514945508298155
|
||||
https://www.douyin.com/video/7617709513994933881
|
||||
https://www.douyin.com/video/7617842610997153893
|
||||
https://www.douyin.com/video/7617844922620023286
|
||||
https://www.douyin.com/video/7617905644313595174
|
||||
https://www.douyin.com/video/7617906023080152366
|
||||
https://www.douyin.com/video/7617906321320430874
|
||||
https://www.douyin.com/video/7618055262938794986
|
||||
https://www.douyin.com/video/7618239783752856191
|
||||
https://www.douyin.com/video/7618431510190082438
|
||||
https://www.douyin.com/video/7618686925839520083
|
||||
https://www.douyin.com/video/7618781384173244581
|
||||
https://www.douyin.com/video/7618917413900036934
|
||||
https://www.douyin.com/video/7619123455581080883
|
||||
https://www.douyin.com/video/7619142462464491195
|
||||
https://www.douyin.com/video/7619175919709727665
|
||||
https://www.douyin.com/video/7619313117805251903
|
||||
https://www.douyin.com/video/7619320302955442003
|
||||
https://www.douyin.com/video/7619327283395956009
|
||||
https://www.douyin.com/video/7619366626797833482
|
||||
https://www.douyin.com/video/7619642803363009478
|
||||
https://www.douyin.com/video/7619655047777019633
|
||||
https://www.douyin.com/video/7619739298111261958
|
||||
https://www.douyin.com/video/7619855484144684294
|
||||
https://www.douyin.com/video/7620573913735132914
|
||||
https://www.douyin.com/video/7620760577976533945
|
||||
https://www.douyin.com/video/7621408706206469426
|
||||
https://www.douyin.com/video/7621463684434414883
|
||||
https://www.douyin.com/video/7621824306896440576
|
||||
https://www.douyin.com/video/7621870175855463723
|
||||
https://www.douyin.com/video/7622348326750440710
|
||||
https://www.douyin.com/video/7622551801807654180
|
||||
https://www.douyin.com/video/7622551849857551652
|
||||
https://www.douyin.com/video/7622551867939228970
|
||||
https://www.douyin.com/video/7622916648307019043
|
||||
https://www.douyin.com/video/7622926694719128037
|
||||
https://www.douyin.com/video/7623037925130833186
|
||||
https://www.douyin.com/video/7623393918260836709
|
||||
https://www.douyin.com/video/7623445134398524714
|
||||
https://www.douyin.com/video/7623634756546071846
|
||||
https://www.douyin.com/video/7623830379681647878
|
||||
https://www.douyin.com/video/7624392177145105679
|
||||
https://www.douyin.com/video/7624529056972803338
|
||||
https://www.douyin.com/video/7624826104108338432
|
||||
https://www.douyin.com/video/7624830305488080168
|
||||
https://www.douyin.com/video/7625810880374082826
|
||||
https://www.douyin.com/video/7625878643599756595
|
||||
https://www.douyin.com/video/7625996760858594603
|
||||
https://www.douyin.com/video/7626338305965182258
|
||||
https://www.douyin.com/video/7626697056581192179
|
||||
https://www.douyin.com/video/7626715245750406435
|
||||
https://www.douyin.com/video/7626725992530332978
|
||||
https://www.douyin.com/video/7627113956481584435
|
||||
https://www.douyin.com/video/7627116296735444260
|
||||
https://www.douyin.com/video/7627404581693181238
|
||||
https://www.douyin.com/video/7627424782224559398
|
||||
https://www.douyin.com/video/7627433230710492470
|
||||
https://www.douyin.com/video/7627827715109965083
|
||||
https://www.douyin.com/video/7627839718314495274
|
||||
https://www.douyin.com/video/7628147653880089896
|
||||
https://www.douyin.com/video/7628486944548244763
|
||||
https://www.douyin.com/video/7628550586404703514
|
||||
https://www.douyin.com/video/7628907808217451827
|
||||
https://www.douyin.com/video/7629016832065768754
|
||||
https://www.douyin.com/video/7629023262042787107
|
||||
https://www.douyin.com/video/7629364084495944987
|
||||
https://www.douyin.com/video/7629674904291495206
|
||||
https://www.douyin.com/video/7629715248747957547
|
||||
https://www.douyin.com/video/7630002143101046026
|
||||
https://www.douyin.com/video/7630035011151990035
|
||||
https://www.douyin.com/video/7630435755621453097
|
||||
https://www.douyin.com/video/7630800798733192490
|
||||
https://www.douyin.com/video/7631132552295746835
|
||||
https://www.douyin.com/video/7631152747122855174
|
||||
https://www.douyin.com/video/7631513007029325075
|
||||
https://www.douyin.com/video/7631888456557350170
|
||||
https://www.douyin.com/video/7632002655170153754
|
||||
https://www.douyin.com/video/7632128546911112499
|
||||
https://www.douyin.com/video/7632250717218131234
|
||||
https://www.douyin.com/video/7632317146051169582
|
||||
https://www.douyin.com/video/7632677096627735842
|
||||
https://www.douyin.com/video/7632891709134736649
|
||||
https://www.douyin.com/video/7633051373524847899
|
||||
https://www.douyin.com/video/7633337273710333230
|
||||
https://www.douyin.com/video/7633346756327050522
|
||||
https://www.douyin.com/video/7633682062968458534
|
||||
https://www.douyin.com/video/7634022529052593443
|
||||
https://www.douyin.com/video/7634050693557294586
|
||||
https://www.douyin.com/video/7634125301194313011
|
||||
https://www.douyin.com/video/7634451794004888842
|
||||
https://www.douyin.com/video/7634496875764272390
|
||||
https://www.douyin.com/video/7634522194848124214
|
||||
https://www.douyin.com/video/7634600021383499059
|
||||
https://www.douyin.com/video/7634640655255489844
|
||||
https://www.douyin.com/video/7634857312013077605
|
||||
https://www.douyin.com/video/7634895725691817258
|
||||
https://www.douyin.com/video/7635223823972830500
|
||||
https://www.douyin.com/video/7635236908355865898
|
||||
https://www.douyin.com/video/7635260096217468187
|
||||
https://www.douyin.com/video/7635263813381672202
|
||||
https://www.douyin.com/video/7635558808021929257
|
||||
https://www.douyin.com/video/7635754732098768174
|
||||
https://www.douyin.com/video/7635919581701344553
|
||||
https://www.douyin.com/video/7635928586301574426
|
||||
https://www.douyin.com/video/7636050065580248347
|
||||
https://www.douyin.com/video/7636280897676016058
|
||||
https://www.douyin.com/video/7636327999012441371
|
||||
https://www.douyin.com/video/7637323468807079781
|
||||
https://www.douyin.com/video/7637389704832371987
|
||||
https://www.douyin.com/video/7637516612030958883
|
||||
https://www.douyin.com/video/7638114648118203877
|
||||
https://www.douyin.com/video/7638159617234603270
|
||||
https://www.douyin.com/video/7638174898539498788
|
||||
https://www.douyin.com/video/7638600102835146020
|
||||
https://www.douyin.com/video/7638933360575892778
|
||||
https://www.douyin.com/video/7638968360796982579
|
||||
https://www.douyin.com/video/7639319248816524571
|
||||
https://www.douyin.com/video/7639367842818937265
|
||||
https://www.douyin.com/video/7639391448319724835
|
||||
https://www.douyin.com/video/7639723207913704710
|
||||
https://www.douyin.com/video/7640189751533899048
|
||||
https://www.douyin.com/video/7640756535873883401
|
||||
https://www.douyin.com/video/7640869909517307171
|
||||
https://www.douyin.com/video/7641120721636265270
|
||||
https://www.douyin.com/video/7641589532265123122
|
||||
https://www.douyin.com/video/7641968344052108594
|
||||
https://www.douyin.com/video/7642269444743367936
|
||||
https://www.douyin.com/video/7642325261748342042
|
||||
https://www.douyin.com/video/7643079455417601306
|
||||
https://www.douyin.com/video/7643099561996586280
|
||||
https://www.douyin.com/video/7643330715563724083
|
||||
https://www.douyin.com/video/7643747314741798898
|
||||
https://www.douyin.com/video/7643820601257299251
|
||||
https://www.douyin.com/video/7644129791554469745
|
||||
https://www.douyin.com/video/7644189304972987702
|
||||
https://www.douyin.com/video/7644398135640548859
|
||||
https://www.douyin.com/video/7644454594528939314
|
||||
https://www.douyin.com/video/7644485500253785407
|
||||
https://www.douyin.com/video/7644565490399989055
|
||||
https://www.douyin.com/video/7645037263830699283
|
||||
https://www.douyin.com/video/7645153715855112633
|
||||
https://www.douyin.com/video/7645237321650031878
|
||||
https://www.douyin.com/video/7645319910847040787
|
||||
https://www.douyin.com/video/7645601447903530240
|
||||
https://www.douyin.com/video/7645967951392247091
|
||||
https://www.douyin.com/video/7646052528894774591
|
||||
https://www.douyin.com/video/7646307038016316133
|
||||
https://www.douyin.com/video/7646388274046537002
|
||||
https://www.douyin.com/video/7646649948490026290
|
||||
https://www.douyin.com/video/7646776690709957951
|
||||
https://www.douyin.com/video/7647147109493247238
|
||||
https://www.douyin.com/video/7647535640694836543
|
||||
https://www.douyin.com/video/7647723480591343601
|
||||
https://www.douyin.com/video/7647871582534241551
|
||||
https://www.douyin.com/video/7648180273783344422
|
||||
https://www.douyin.com/video/7648237241789074751
|
||||
https://www.douyin.com/video/7648532239407516974
|
||||
https://www.douyin.com/video/7648555100415298835
|
||||
https://www.douyin.com/video/7648581346801044774
|
||||
https://www.douyin.com/video/7648604334774684947
|
||||
https://www.douyin.com/video/7648892407303507263
|
||||
https://www.douyin.com/video/7648981178782960942
|
||||
https://www.douyin.com/video/7649290429682208052
|
||||
https://www.douyin.com/video/7649322286621472441
|
||||
https://www.douyin.com/video/7649382734304972075
|
||||
https://www.douyin.com/video/7649472549520198962
|
||||
https://www.douyin.com/video/7649679394864172298
|
||||
https://www.douyin.com/video/7649733463527918889
|
||||
https://www.douyin.com/video/7649943483725171987
|
||||
https://www.douyin.com/video/7650038680744250675
|
||||
https://www.douyin.com/video/7650072092766915880
|
||||
https://www.douyin.com/video/7650110217032307045
|
||||
https://www.douyin.com/video/7650313961908974523
|
||||
https://www.douyin.com/video/7650477456188509486
|
||||
https://www.douyin.com/video/7650501431241329935
|
||||
https://www.douyin.com/video/7650769199626865947
|
||||
https://www.douyin.com/video/7650836608844877065
|
||||
https://www.douyin.com/video/7650837877999291702
|
||||
https://www.douyin.com/video/7650997338525158708
|
||||
https://www.douyin.com/video/7651270613679557894
|
||||
https://www.douyin.com/video/7651469973809876276
|
||||
https://www.douyin.com/video/7651511047378980130
|
||||
https://www.douyin.com/video/7651620162990329140
|
||||
https://www.douyin.com/video/7651783213036164025
|
||||
https://www.douyin.com/video/7651983818441772351
|
||||
https://www.douyin.com/video/7652159495296044324
|
||||
https://www.douyin.com/video/7652332368618966326
|
||||
https://www.douyin.com/video/7652337263422098707
|
||||
https://www.douyin.com/video/7652690556325301531
|
||||
https://www.douyin.com/video/7652760586555624755
|
||||
https://www.douyin.com/video/7653037217077579058
|
||||
https://www.douyin.com/video/7653044535480802041
|
||||
https://www.douyin.com/video/7653272029650505006
|
||||
https://www.douyin.com/video/7653319960801889542
|
||||
https://www.douyin.com/video/7653342880093850920
|
||||
https://www.douyin.com/video/7653384213898530048
|
||||
https://www.douyin.com/video/7653395254703361330
|
||||
https://www.douyin.com/video/7653462607520648490
|
||||
https://www.douyin.com/video/7653742612871990538
|
||||
https://www.douyin.com/video/7653780737778273587
|
||||
https://www.douyin.com/video/7653785256289471786
|
||||
https://www.douyin.com/video/7654081151534910746
|
||||
https://www.douyin.com/video/7654509285417798963
|
||||
https://www.douyin.com/video/7654582738226631141
|
||||
https://www.douyin.com/video/7654695117399346451
|
||||
https://www.douyin.com/video/7654841525762919726
|
||||
https://www.douyin.com/video/7654877200253023857
|
||||
https://www.douyin.com/video/7654938141274454641
|
||||
https://www.douyin.com/video/7655274807872867635
|
||||
https://www.douyin.com/video/7655294658511146250
|
||||
https://www.douyin.com/video/7655578760405732644
|
||||
https://www.douyin.com/video/7655634255539703040
|
||||
https://www.douyin.com/video/7655641314603666730
|
||||
https://www.douyin.com/video/7655649609087323446
|
||||
https://www.douyin.com/video/7655966421515672689
|
||||
https://www.douyin.com/video/7656044126965681418
|
||||
https://www.douyin.com/video/7656047318969437503
|
||||
https://www.douyin.com/video/7656049583902051620
|
||||
https://www.douyin.com/video/7656071302272273715
|
||||
https://www.douyin.com/video/7656203379277860106
|
||||
https://www.douyin.com/video/7656282438393482548
|
||||
https://www.douyin.com/video/7656344529238265138
|
||||
https://www.douyin.com/video/7656459414970420521
|
||||
https://www.douyin.com/video/7656713475099561250
|
||||
https://www.douyin.com/video/7656749023751081254
|
||||
https://www.douyin.com/video/7656753234333158699
|
||||
https://www.douyin.com/video/7657145292726258986
|
||||
https://www.douyin.com/video/7657175721030323497
|
||||
https://www.douyin.com/video/7657180863557340454
|
||||
https://www.douyin.com/video/7657193988009417081
|
||||
https://www.douyin.com/video/7657219879371672841
|
||||
https://www.douyin.com/video/7657484474833292585
|
||||
https://www.douyin.com/video/7657555170498284826
|
||||
https://www.douyin.com/video/7658148334569360650
|
||||
https://www.douyin.com/video/7658231417406491919
|
||||
https://www.douyin.com/video/7658241711407140131
|
||||
https://www.douyin.com/video/7658256433359736070
|
||||
https://www.douyin.com/video/7658513811904695592
|
||||
https://www.douyin.com/video/7658558423587474731
|
||||
https://www.douyin.com/video/7658627840988381435
|
||||
https://www.douyin.com/video/7658660005062675763
|
||||
https://www.douyin.com/video/7658936566362034682
|
||||
https://www.douyin.com/video/7659004095709089078
|
||||
https://www.douyin.com/video/7659009919047241012
|
||||
https://www.douyin.com/video/7659265009436922298
|
||||
https://www.douyin.com/video/7659329841984207717
|
||||
https://www.douyin.com/video/7659365936067120403
|
||||
https://www.douyin.com/video/7659387012948020514
|
||||
https://www.douyin.com/video/7659677959305400305
|
||||
https://www.douyin.com/video/7659771186154376483
|
||||
https://www.douyin.com/video/7659812552586628390
|
||||
https://www.douyin.com/video/7660043887762853158
|
||||
https://www.douyin.com/video/7660080456532100325
|
||||
https://www.douyin.com/video/7660378622334668051
|
||||
https://www.douyin.com/video/7660452842536176911
|
||||
https://www.douyin.com/video/7660461857463833445
|
||||
https://www.douyin.com/video/7660746995748883766
|
||||
https://www.douyin.com/video/7660829348592705957
|
||||
https://www.douyin.com/video/7660882419880332559
|
||||
https://www.douyin.com/video/7661194952365444403
|
||||
https://www.douyin.com/video/7661198419789198769
|
||||
https://www.douyin.com/video/7661240245349813554
|
||||
https://www.douyin.com/video/7661262754998182373
|
||||
https://www.douyin.com/video/7661325177510169891
|
||||
https://www.douyin.com/video/7661447694502717925
|
||||
https://www.douyin.com/video/7661626304455136997
|
||||
https://www.douyin.com/video/7661917157235739950
|
||||
https://www.douyin.com/video/7661934792333708584
|
||||
https://www.douyin.com/video/7661950772778683700
|
||||
https://www.douyin.com/video/7662005958176180197
|
||||
https://www.douyin.com/video/7662207301134155953
|
||||
https://www.douyin.com/video/7662216114093321125
|
||||
https://www.douyin.com/video/7662342717439036707
|
||||
https://www.douyin.com/video/7662409809232284928
|
||||
https://www.douyin.com/video/7662436209800514850
|
||||
https://www.douyin.com/video/7662655730943593791
|
||||
https://www.douyin.com/video/7662675520517066027
|
||||
https://www.douyin.com/video/7662726086103140261
|
||||
https://www.douyin.com/video/7662726933427260714
|
||||
https://www.douyin.com/video/7663349110917863781
|
||||
https://www.douyin.com/video/7663437082131500339
|
||||
https://www.douyin.com/video/7663528696544195441
|
||||
https://www.douyin.com/video/7663623587605089572
|
||||
https://www.douyin.com/video/7663761563040327665
|
||||
https://www.douyin.com/video/7663848518829739302
|
||||
https://www.douyin.com/video/7664041284713365242
|
||||
https://www.douyin.com/video/7664109347026291977
|
||||
https://www.douyin.com/video/7664208543336023674
|
||||
https://www.douyin.com/video/7664227832112876819
|
||||
https://www.douyin.com/video/7664440455664356623
|
||||
https://www.douyin.com/video/7664504179649432886
|
||||
https://www.douyin.com/video/7664867676778466579
|
||||
https://www.douyin.com/video/7664876816712125738
|
||||
https://www.douyin.com/video/7664896249895619894
|
||||
https://www.douyin.com/video/7664930885019503918
|
||||
https://www.douyin.com/video/7664961390760044537
|
||||
https://www.douyin.com/video/7665042774567760357
|
||||
https://www.douyin.com/video/7665081523623808271
|
||||
https://www.douyin.com/video/7665269536095145891
|
||||
https://www.douyin.com/video/7665333213509733695
|
||||
https://www.douyin.com/video/7665558591248244288
|
||||
https://www.douyin.com/video/7665691979073785130
|
||||
https://www.douyin.com/video/7665695224682157157
|
||||
https://www.douyin.com/video/7665728516148097896
|
||||
https://www.douyin.com/video/7665883741978035560
|
||||
https://www.douyin.com/video/7666016387768323368
|
||||
https://www.douyin.com/video/7666083728954854694
|
||||
https://www.douyin.com/video/7666252572591705384
|
||||
https://www.douyin.com/video/7666283773364958196
|
||||
https://www.douyin.com/video/7666338979893071154
|
||||
https://www.douyin.com/video/7666357810216586523
|
||||
https://www.douyin.com/video/7666359051386752291
|
||||
https://www.douyin.com/video/7666359674333121844
|
||||
https://www.douyin.com/video/7666435655073953253
|
||||
https://www.douyin.com/video/7666470100010077503
|
||||
https://www.douyin.com/video/7666631806632365967
|
||||
https://www.douyin.com/video/7666723775321314036
|
||||
https://www.douyin.com/video/7667014549647805731
|
||||
https://www.douyin.com/video/7667128120517199267
|
||||
https://www.douyin.com/video/7667138218667691273
|
||||
https://www.douyin.com/video/7667163827305205034
|
||||
https://www.douyin.com/video/7667177971279233189
|
||||
https://www.douyin.com/video/7667208626179820810
|
||||
https://www.douyin.com/video/7667416018432030641
|
||||
https://www.douyin.com/video/7667471259399212303
|
||||
https://www.douyin.com/video/7667510510514580687
|
||||
https://www.douyin.com/video/7667584140971085071
|
||||
https://www.douyin.com/video/7667837546495875810
|
||||
https://www.douyin.com/video/7667867655306530094
|
||||
https://www.douyin.com/video/7667880938403040527
|
||||
https://www.douyin.com/video/7667883685324148008
|
||||
https://www.douyin.com/video/7667921594988201637
|
||||
https://www.douyin.com/video/7667954880291654962
|
||||
https://www.douyin.com/video/7668142595268201734
|
||||
https://www.douyin.com/video/7668185235808341282
|
||||
https://www.douyin.com/video/7668232734145252671
|
||||
https://www.douyin.com/video/7668249316428194728
|
||||
https://www.douyin.com/video/7668430591771151666
|
||||
https://www.douyin.com/video/7668693467802963227
|
||||
https://www.douyin.com/video/7668701871880452264
|
||||
https://www.douyin.com/video/7668847824344629114
|
||||
https://www.douyin.com/video/7668983792408775942
|
||||
https://www.douyin.com/video/7669031795585197233
|
||||
https://www.douyin.com/video/7669335701900528942
|
||||
https://www.douyin.com/video/7669369157502242057
|
||||
https://www.douyin.com/video/7669642580174940325
|
||||
https://www.douyin.com/video/7669727308365810953
|
||||
https://www.douyin.com/video/7669775057494129509
|
||||
https://www.douyin.com/video/7669820347528531243
|
||||
https://www.douyin.com/video/7670106766973095209
|
||||
https://www.douyin.com/video/7670113416547241256
|
||||
https://www.douyin.com/video/7670139266109836594
|
||||
https://www.douyin.com/video/7670163128612844431
|
||||
https://www.douyin.com/video/7670461512384875810
|
||||
https://www.douyin.com/video/7670521663624076389
|
||||
https://www.douyin.com/video/7670528676970253618
|
||||
https://www.douyin.com/video/7670756266968989435
|
||||
https://www.douyin.com/video/7670813228960276047
|
||||
https://www.douyin.com/video/7670917275737705743
|
||||
https://www.douyin.com/video/7670936864647998897
|
||||
https://www.douyin.com/video/7671118098316954914
|
||||
https://www.douyin.com/video/7671168257755188521
|
||||
https://www.douyin.com/video/7671261026582766309
|
||||
https://www.douyin.com/video/7671558411390439586
|
||||
https://www.douyin.com/video/7671587635063754010
|
||||
https://www.douyin.com/video/7671624629550714131
|
||||
https://www.douyin.com/video/7671625098054389034
|
||||
https://www.douyin.com/video/7671640325530052346
|
||||
https://www.douyin.com/video/7671960967781043507
|
||||
https://www.douyin.com/video/7671992164376644915
|
||||
https://www.douyin.com/video/7672000964644424049
|
||||
https://www.douyin.com/video/7672249622015026341
|
||||
https://www.douyin.com/video/7672256384322309439
|
||||
https://www.douyin.com/video/7672308964353568616
|
||||
https://www.douyin.com/video/7672576055014106447
|
||||
https://www.douyin.com/video/7672668778479602987
|
||||
https://www.douyin.com/video/7672715650552401178
|
||||
https://www.douyin.com/video/7672720290434280569
|
||||
https://www.douyin.com/video/7672742807430267505
|
||||
https://www.douyin.com/video/7672755197218753835
|
||||
https://www.douyin.com/video/7672841488367095080
|
||||
https://www.douyin.com/video/7673038887052581711
|
||||
https://www.douyin.com/video/7673041515844767030
|
||||
https://www.douyin.com/video/7673054215530417448
|
||||
https://www.douyin.com/video/7673076486882069814
|
||||
https://www.douyin.com/video/7673228409090931429
|
||||
https://www.douyin.com/video/7673261493835533606
|
||||
https://www.douyin.com/video/7673321465336352038
|
||||
https://www.douyin.com/video/7673321926504287488
|
||||
https://www.douyin.com/video/7673464113044917546
|
||||
https://www.douyin.com/video/7673484982408620645
|
||||
https://www.douyin.com/video/7673516260797189414
|
||||
https://www.douyin.com/video/7673694364136213823
|
||||
https://www.douyin.com/video/7673694818773601577
|
||||
https://www.douyin.com/video/7673729957286202979
|
||||
https://www.douyin.com/video/7673730543112908785
|
||||
https://www.douyin.com/video/7673797066992176447
|
||||
https://www.douyin.com/video/7673807082516172074
|
||||
https://www.douyin.com/video/7673905956783231156
|
||||
https://www.douyin.com/video/7674082733324389668
|
||||
https://www.douyin.com/video/7674083224645045544
|
||||
https://www.douyin.com/video/7674161654128463104
|
||||
https://www.douyin.com/video/7674170244713254190
|
||||
https://www.douyin.com/video/7674177167744478095
|
||||
https://www.douyin.com/video/7674211166566042923
|
||||
https://www.douyin.com/video/7674231512276184805
|
||||
https://www.douyin.com/video/7674253499941572776
|
||||
https://www.douyin.com/video/7674569226421622458
|
||||
https://www.douyin.com/video/7674579131442449716
|
||||
https://www.douyin.com/video/7674702704652972923
|
||||
https://www.douyin.com/video/7674914537828633908
|
||||
https://www.douyin.com/video/7674937074646142259
|
||||
https://www.douyin.com/video/7674937637546806569
|
||||
https://www.douyin.com/video/7674972508814558394
|
||||
https://www.douyin.com/video/7675124829770632458
|
||||
https://www.douyin.com/video/7675365522011836904
|
||||
https://www.douyin.com/video/7675483265004622266
|
||||
https://www.douyin.com/video/7675539470892010778
|
||||
https://www.douyin.com/video/7675609427092819234
|
||||
https://www.douyin.com/video/7675627341812452660
|
||||
https://www.douyin.com/video/7675999529963856994
|
||||
https://www.douyin.com/video/7676028608536346810
|
||||
https://www.douyin.com/video/7676045799751378545
|
||||
https://www.douyin.com/video/7676068186245257723
|
||||
https://www.douyin.com/video/7676071873519553828
|
||||
https://www.douyin.com/video/7676206699581115698
|
||||
https://www.douyin.com/video/7676252942920758642
|
||||
https://www.douyin.com/video/7676324659302075721
|
||||
https://www.douyin.com/video/7676402775265217838
|
||||
https://www.douyin.com/video/7676443672221278555
|
||||
https://www.douyin.com/video/7676507753142246842
|
||||
https://www.douyin.com/video/7676651392870167434
|
||||
https://www.douyin.com/video/7676706546445666150
|
||||
https://www.douyin.com/video/7676726807206415679
|
||||
https://www.douyin.com/video/7676777703432970478
|
||||
https://www.douyin.com/video/7676806593186417963
|
||||
https://www.douyin.com/video/7676810949554195762
|
||||
https://www.douyin.com/video/7676816025307974927
|
||||
https://www.douyin.com/video/7676973090311893618
|
||||
https://www.douyin.com/video/7677139895508421926
|
||||
https://www.douyin.com/video/7677168765959703860
|
||||
https://www.douyin.com/video/7677225783300160486
|
||||
https://www.douyin.com/video/7677382859988774026
|
||||
https://www.douyin.com/video/7677436645130737529
|
||||
https://www.douyin.com/video/7677529104766669294
|
||||
https://www.douyin.com/video/7677544232332086528
|
||||
https://www.douyin.com/video/7677544710864391460
|
||||
https://www.douyin.com/video/7677726494199115891
|
||||
https://www.douyin.com/video/7677741684755087013
|
||||
https://www.douyin.com/video/7677830429231739546
|
||||
https://www.douyin.com/video/7677908207888567606
|
||||
https://www.douyin.com/video/7677919976723893510
|
||||
https://www.douyin.com/video/7677935094850024393
|
||||
https://www.douyin.com/video/7677957315656094446
|
||||
https://www.douyin.com/video/7678120340850132827
|
||||
https://www.douyin.com/video/7678171186031688378
|
||||
https://www.douyin.com/video/7678188866529038003
|
||||
https://www.douyin.com/video/7678243344095639205
|
||||
https://www.douyin.com/video/7678292689787435610
|
||||
https://www.douyin.com/video/7678486832644680649
|
||||
https://www.douyin.com/video/7678544297163055270
|
||||
https://www.douyin.com/video/7678546421313703208
|
||||
https://www.douyin.com/video/7678575927890038068
|
||||
https://www.douyin.com/video/7678585542967069998
|
||||
https://www.douyin.com/video/7678590557425323314
|
||||
https://www.douyin.com/video/7678613511093751055
|
||||
https://www.douyin.com/video/7678622962815077682
|
||||
https://www.douyin.com/video/7678625502013975899
|
||||
https://www.douyin.com/video/7678633564577025330
|
||||
https://www.douyin.com/video/7678636864873978106
|
||||
https://www.douyin.com/video/7678652628621164261
|
||||
https://www.douyin.com/video/7678654371135265467
|
||||
https://www.douyin.com/video/7678658151821937972
|
||||
https://www.douyin.com/video/7678661269704451323
|
||||
https://www.douyin.com/video/7678665392645999908
|
||||
https://www.douyin.com/video/7678670285122237797
|
||||
https://www.douyin.com/video/7678675913445264667
|
||||
https://www.douyin.com/video/7678682245799873826
|
||||
https://www.douyin.com/video/7678685367255551266
|
||||
https://www.douyin.com/video/7678687122127113507
|
||||
https://www.douyin.com/video/7678687138652732379
|
||||
https://www.douyin.com/video/7678749541306289446
|
||||
https://www.douyin.com/video/7678755783550455090
|
||||
https://www.douyin.com/video/7678763820864654619
|
||||
https://www.douyin.com/video/7678825975556034698
|
||||
https://www.douyin.com/video/7678886371372617014
|
||||
https://www.douyin.com/video/7678898058890329408
|
||||
https://www.douyin.com/video/7678915532866768138
|
||||
https://www.douyin.com/video/7678929655399691098
|
||||
https://www.douyin.com/video/7678937719686450474
|
||||
https://www.douyin.com/video/7678947950425496768
|
||||
https://www.douyin.com/video/7678968987773865231
|
||||
https://www.douyin.com/video/7678985176809998409
|
||||
https://www.douyin.com/video/7678986769739664640
|
||||
https://www.douyin.com/video/7678993425631377999
|
||||
https://www.douyin.com/video/7679008022307409193
|
||||
https://www.douyin.com/video/7679022154000408762
|
||||
https://www.douyin.com/video/7679027352965352730
|
||||
https://www.douyin.com/video/7679029813202758514
|
||||
https://www.douyin.com/video/7679034815433231631
|
||||
https://www.douyin.com/video/7679041467328137875
|
||||
https://www.douyin.com/video/7679046934312635667
|
||||
https://www.douyin.com/video/7679051682629602614
|
||||
https://www.douyin.com/video/7679068471501622579
|
||||
https://www.douyin.com/video/7679082616453360569
|
||||
https://www.douyin.com/video/7679085904597650740
|
||||
https://www.douyin.com/video/7679088520916389170
|
||||
https://www.douyin.com/video/7679101989992533284
|
||||
https://www.douyin.com/video/7679120738544285541
|
||||
https://www.douyin.com/video/7679221372983299379
|
||||
https://www.douyin.com/video/7679262208672455077
|
||||
https://www.douyin.com/video/7679264479682546984
|
||||
https://www.douyin.com/video/7679297146138715337
|
||||
https://www.douyin.com/video/7679299506656549627
|
||||
https://www.douyin.com/video/7679308946070408511
|
||||
https://www.douyin.com/video/7679325505833375936
|
||||
https://www.douyin.com/video/7679334312591904034
|
||||
https://www.douyin.com/video/7679355638165572890
|
||||
https://www.douyin.com/video/7679363844532489332
|
||||
https://www.douyin.com/video/7679381683296144686
|
||||
https://www.douyin.com/video/7679386621280112250
|
||||
https://www.douyin.com/video/7679389104014865242
|
||||
https://www.douyin.com/video/7679408597402537225
|
||||
https://www.douyin.com/video/7679414313097907494
|
||||
https://www.douyin.com/video/7679414794009954478
|
||||
https://www.douyin.com/video/7679418736025144603
|
||||
https://www.douyin.com/video/7679419565223588465
|
||||
https://www.douyin.com/video/7679424155271695525
|
||||
https://www.douyin.com/video/7679426058361343278
|
||||
https://www.douyin.com/video/7679427553739787554
|
||||
https://www.douyin.com/video/7679427741199944987
|
||||
https://www.douyin.com/video/7679440707580398858
|
||||
https://www.douyin.com/video/7679442397798347771
|
||||
https://www.douyin.com/video/7679445657974754598
|
||||
https://www.douyin.com/video/7679451408902425865
|
||||
https://www.douyin.com/video/7679553393639083299
|
||||
https://www.douyin.com/video/7679586455651487011
|
||||
https://www.douyin.com/video/7679594027264773606
|
||||
https://www.douyin.com/video/7679619228953754981
|
||||
https://www.douyin.com/video/7679669997992151278
|
||||
https://www.douyin.com/video/7679685319389678899
|
||||
https://www.douyin.com/video/7679719810942438675
|
||||
https://www.douyin.com/video/7679723554786643234
|
||||
https://www.douyin.com/video/7679736105419181331
|
||||
https://www.douyin.com/video/7679742979069709603
|
||||
https://www.douyin.com/video/7679744717130698018
|
||||
https://www.douyin.com/video/7679749023187782388
|
||||
https://www.douyin.com/video/7679762551609068841
|
||||
https://www.douyin.com/video/7679773030955977990
|
||||
https://www.douyin.com/video/7679777205719747955
|
||||
https://www.douyin.com/video/7679786760994610043
|
||||
https://www.douyin.com/video/7679788645968022838
|
||||
https://www.douyin.com/video/7679796367509769518
|
||||
https://www.douyin.com/video/7679797414697471238
|
||||
https://www.douyin.com/video/7679797886426664201
|
||||
https://www.douyin.com/video/7679800245206764843
|
||||
https://www.douyin.com/video/7679806984497432998
|
||||
https://www.douyin.com/video/7679812964446349203
|
||||
https://www.douyin.com/video/7679814965406092595
|
||||
https://www.douyin.com/video/7679816146242263680
|
||||
https://www.douyin.com/video/7679847926545337636
|
||||
https://www.douyin.com/video/7679859240772628901
|
||||
https://www.douyin.com/video/7679960541577678115
|
||||
https://www.douyin.com/video/7679973657594105103
|
||||
https://www.douyin.com/video/7679978692710321134
|
||||
https://www.douyin.com/video/7679984198715296098
|
||||
https://www.douyin.com/video/7680009929671926757
|
||||
https://www.douyin.com/video/7680032605705735460
|
||||
https://www.douyin.com/video/7680052463843560750
|
||||
https://www.douyin.com/video/7680056996313607781
|
||||
https://www.douyin.com/video/7680059155767652538
|
||||
https://www.douyin.com/video/7680081065259699490
|
||||
https://www.douyin.com/video/7680081287918456110
|
||||
https://www.douyin.com/video/7680095194250834926
|
||||
https://www.douyin.com/video/7680105715771583611
|
||||
https://www.douyin.com/video/7680106103991979315
|
||||
https://www.douyin.com/video/7680117761921666340
|
||||
https://www.douyin.com/video/7680117927642697000
|
||||
https://www.douyin.com/video/7680119645780659491
|
||||
https://www.douyin.com/video/7680133706740206875
|
||||
https://www.douyin.com/video/7680133802567486761
|
||||
https://www.douyin.com/video/7680140246826143026
|
||||
https://www.douyin.com/video/7680142839187000639
|
||||
https://www.douyin.com/video/7680144311337356590
|
||||
https://www.douyin.com/video/7680152788999928035
|
||||
https://www.douyin.com/video/7680171091331009838
|
||||
https://www.douyin.com/video/7680174467008003034
|
||||
https://www.douyin.com/video/7680181136361917748
|
||||
https://www.douyin.com/video/7680188118762548985
|
||||
https://www.douyin.com/video/7680189239446752552
|
||||
https://www.douyin.com/video/7680208337530637568
|
||||
https://www.douyin.com/video/7680263311061028115
|
||||
https://www.douyin.com/video/7680311538980293934
|
||||
https://www.douyin.com/video/7680340917585764570
|
||||
https://www.douyin.com/video/7680363025051798794
|
||||
https://www.douyin.com/video/7680422505299453539
|
||||
52360
lists/ai_62171.txt
Normal file
52360
lists/ai_62171.txt
Normal file
File diff suppressed because it is too large
Load Diff
1366
lists/ai_87110_严格.txt
Normal file
1366
lists/ai_87110_严格.txt
Normal file
File diff suppressed because it is too large
Load Diff
15218
lists/ai_videos.txt
Normal file
15218
lists/ai_videos.txt
Normal file
File diff suppressed because it is too large
Load Diff
66411
lists/ai_合并待下载.txt
Normal file
66411
lists/ai_合并待下载.txt
Normal file
File diff suppressed because it is too large
Load Diff
2611
lists/ai_待下载.txt
Normal file
2611
lists/ai_待下载.txt
Normal file
File diff suppressed because it is too large
Load Diff
3557
lists/all_video_links(3).txt
Normal file
3557
lists/all_video_links(3).txt
Normal file
File diff suppressed because it is too large
Load Diff
4839
lists/csv_links.txt
Normal file
4839
lists/csv_links.txt
Normal file
File diff suppressed because it is too large
Load Diff
1331
lists/platform无佐证_带货.txt
Normal file
1331
lists/platform无佐证_带货.txt
Normal file
File diff suppressed because it is too large
Load Diff
776
lists/platform无佐证_非带货.txt
Normal file
776
lists/platform无佐证_非带货.txt
Normal file
@@ -0,0 +1,776 @@
|
||||
https://www.douyin.com/video/7380366513469000987
|
||||
https://www.douyin.com/video/7514599593988869403
|
||||
https://www.douyin.com/video/7516418196564348211
|
||||
https://www.douyin.com/video/7520500385008012580
|
||||
https://www.douyin.com/video/7521986436939287862
|
||||
https://www.douyin.com/video/7523221568294751529
|
||||
https://www.douyin.com/video/7525070625204915519
|
||||
https://www.douyin.com/video/7530830172222115135
|
||||
https://www.douyin.com/video/7538042966386134291
|
||||
https://www.douyin.com/video/7539753234932698411
|
||||
https://www.douyin.com/video/7541340766095052068
|
||||
https://www.douyin.com/video/7546108774513691943
|
||||
https://www.douyin.com/video/7548819336112033063
|
||||
https://www.douyin.com/video/7549102755731426623
|
||||
https://www.douyin.com/video/7549370086735777087
|
||||
https://www.douyin.com/video/7549883472979512618
|
||||
https://www.douyin.com/video/7550105813105462571
|
||||
https://www.douyin.com/video/7550227999062248745
|
||||
https://www.douyin.com/video/7550516151760358675
|
||||
https://www.douyin.com/video/7550516954933005609
|
||||
https://www.douyin.com/video/7550524160104664363
|
||||
https://www.douyin.com/video/7550636242367614247
|
||||
https://www.douyin.com/video/7550869239898426660
|
||||
https://www.douyin.com/video/7551270473684045092
|
||||
https://www.douyin.com/video/7551772650604547364
|
||||
https://www.douyin.com/video/7552182479986740521
|
||||
https://www.douyin.com/video/7552736371346935103
|
||||
https://www.douyin.com/video/7552864085667679542
|
||||
https://www.douyin.com/video/7552872941857328427
|
||||
https://www.douyin.com/video/7553294319786020137
|
||||
https://www.douyin.com/video/7553529503444684068
|
||||
https://www.douyin.com/video/7553636686220152103
|
||||
https://www.douyin.com/video/7553927366579768619
|
||||
https://www.douyin.com/video/7554317903798914367
|
||||
https://www.douyin.com/video/7554655790427147539
|
||||
https://www.douyin.com/video/7555092541298806070
|
||||
https://www.douyin.com/video/7555735902288891172
|
||||
https://www.douyin.com/video/7555814062748962102
|
||||
https://www.douyin.com/video/7557200184175381799
|
||||
https://www.douyin.com/video/7557989911165553962
|
||||
https://www.douyin.com/video/7558025332268977471
|
||||
https://www.douyin.com/video/7558071415342681386
|
||||
https://www.douyin.com/video/7559126788111535403
|
||||
https://www.douyin.com/video/7559504316051148073
|
||||
https://www.douyin.com/video/7559803627863887146
|
||||
https://www.douyin.com/video/7560226318119865619
|
||||
https://www.douyin.com/video/7560611991967108415
|
||||
https://www.douyin.com/video/7560973202706812199
|
||||
https://www.douyin.com/video/7561330641275948324
|
||||
https://www.douyin.com/video/7561614819138260266
|
||||
https://www.douyin.com/video/7561666192981658934
|
||||
https://www.douyin.com/video/7562419181599640873
|
||||
https://www.douyin.com/video/7562420662062533907
|
||||
https://www.douyin.com/video/7562551310820035883
|
||||
https://www.douyin.com/video/7562720574498917674
|
||||
https://www.douyin.com/video/7563608198364532004
|
||||
https://www.douyin.com/video/7564277422092111158
|
||||
https://www.douyin.com/video/7564637942020066623
|
||||
https://www.douyin.com/video/7566118848739839295
|
||||
https://www.douyin.com/video/7566520703677140243
|
||||
https://www.douyin.com/video/7566875725262769462
|
||||
https://www.douyin.com/video/7567717713071820075
|
||||
https://www.douyin.com/video/7569439640647670762
|
||||
https://www.douyin.com/video/7569441572598104484
|
||||
https://www.douyin.com/video/7570276930193226118
|
||||
https://www.douyin.com/video/7570732614638705363
|
||||
https://www.douyin.com/video/7570919864518160326
|
||||
https://www.douyin.com/video/7570984598831091071
|
||||
https://www.douyin.com/video/7571031688211068772
|
||||
https://www.douyin.com/video/7571437498276289252
|
||||
https://www.douyin.com/video/7571674082590614774
|
||||
https://www.douyin.com/video/7571764351976277739
|
||||
https://www.douyin.com/video/7572418606560460115
|
||||
https://www.douyin.com/video/7572789807940673188
|
||||
https://www.douyin.com/video/7573311861044362730
|
||||
https://www.douyin.com/video/7574679807277046646
|
||||
https://www.douyin.com/video/7576146247150934633
|
||||
https://www.douyin.com/video/7576523801786280006
|
||||
https://www.douyin.com/video/7577010388609394793
|
||||
https://www.douyin.com/video/7577299367322733481
|
||||
https://www.douyin.com/video/7577765758047708516
|
||||
https://www.douyin.com/video/7578121891861939177
|
||||
https://www.douyin.com/video/7578861158439109814
|
||||
https://www.douyin.com/video/7579573049822073811
|
||||
https://www.douyin.com/video/7580308431392423906
|
||||
https://www.douyin.com/video/7580575269242495851
|
||||
https://www.douyin.com/video/7580761527533582251
|
||||
https://www.douyin.com/video/7581025934838847668
|
||||
https://www.douyin.com/video/7581302935750006250
|
||||
https://www.douyin.com/video/7581479376954150185
|
||||
https://www.douyin.com/video/7582524757486652713
|
||||
https://www.douyin.com/video/7582852963585871145
|
||||
https://www.douyin.com/video/7582952109878103359
|
||||
https://www.douyin.com/video/7583603476988890409
|
||||
https://www.douyin.com/video/7584267094024375593
|
||||
https://www.douyin.com/video/7585008821253442822
|
||||
https://www.douyin.com/video/7585796917876935990
|
||||
https://www.douyin.com/video/7586551044105112851
|
||||
https://www.douyin.com/video/7588368448867945670
|
||||
https://www.douyin.com/video/7588434109912239999
|
||||
https://www.douyin.com/video/7590233221331340479
|
||||
https://www.douyin.com/video/7591674198630089542
|
||||
https://www.douyin.com/video/7591718737636767977
|
||||
https://www.douyin.com/video/7591854025508475322
|
||||
https://www.douyin.com/video/7594102940194889188
|
||||
https://www.douyin.com/video/7595456098003588843
|
||||
https://www.douyin.com/video/7596185690266995539
|
||||
https://www.douyin.com/video/7598492128692778230
|
||||
https://www.douyin.com/video/7599260304396549203
|
||||
https://www.douyin.com/video/7602909894493396422
|
||||
https://www.douyin.com/video/7605070496028291027
|
||||
https://www.douyin.com/video/7606565420401609451
|
||||
https://www.douyin.com/video/7607318439020041654
|
||||
https://www.douyin.com/video/7609197330093304164
|
||||
https://www.douyin.com/video/7610048707459091839
|
||||
https://www.douyin.com/video/7613344864323755243
|
||||
https://www.douyin.com/video/7613638328865037673
|
||||
https://www.douyin.com/video/7615253818271607878
|
||||
https://www.douyin.com/video/7615255911167099301
|
||||
https://www.douyin.com/video/7617140307985149430
|
||||
https://www.douyin.com/video/7619955405104820139
|
||||
https://www.douyin.com/video/7620301079121492934
|
||||
https://www.douyin.com/video/7621162869261946138
|
||||
https://www.douyin.com/video/7621867088906390819
|
||||
https://www.douyin.com/video/7621899968571844742
|
||||
https://www.douyin.com/video/7624397028261472959
|
||||
https://www.douyin.com/video/7626252792714142570
|
||||
https://www.douyin.com/video/7627694322393091362
|
||||
https://www.douyin.com/video/7627695778076560676
|
||||
https://www.douyin.com/video/7628580359672965546
|
||||
https://www.douyin.com/video/7628985709738609960
|
||||
https://www.douyin.com/video/7629313385972513279
|
||||
https://www.douyin.com/video/7631055424447436651
|
||||
https://www.douyin.com/video/7631593191338183982
|
||||
https://www.douyin.com/video/7631593218768981298
|
||||
https://www.douyin.com/video/7631593230487915827
|
||||
https://www.douyin.com/video/7631593260376395017
|
||||
https://www.douyin.com/video/7631593296074116379
|
||||
https://www.douyin.com/video/7631593320753548570
|
||||
https://www.douyin.com/video/7631593348217851163
|
||||
https://www.douyin.com/video/7631593380090219803
|
||||
https://www.douyin.com/video/7631593407110057267
|
||||
https://www.douyin.com/video/7631593428953959690
|
||||
https://www.douyin.com/video/7631593453855542578
|
||||
https://www.douyin.com/video/7631593482141830438
|
||||
https://www.douyin.com/video/7631593510113692974
|
||||
https://www.douyin.com/video/7631593532419099950
|
||||
https://www.douyin.com/video/7631593558671166746
|
||||
https://www.douyin.com/video/7631593583572815131
|
||||
https://www.douyin.com/video/7631593600350014758
|
||||
https://www.douyin.com/video/7631593624249158950
|
||||
https://www.douyin.com/video/7631593919100341556
|
||||
https://www.douyin.com/video/7631593927895829760
|
||||
https://www.douyin.com/video/7631593973429079330
|
||||
https://www.douyin.com/video/7631593995675749666
|
||||
https://www.douyin.com/video/7631594029892832512
|
||||
https://www.douyin.com/video/7631939367108746089
|
||||
https://www.douyin.com/video/7632679042445947007
|
||||
https://www.douyin.com/video/7633392733889023275
|
||||
https://www.douyin.com/video/7633407841701149823
|
||||
https://www.douyin.com/video/7634910834091915947
|
||||
https://www.douyin.com/video/7634917476601548032
|
||||
https://www.douyin.com/video/7635137905534749998
|
||||
https://www.douyin.com/video/7635141415127614419
|
||||
https://www.douyin.com/video/7635316122467814708
|
||||
https://www.douyin.com/video/7635316191770332450
|
||||
https://www.douyin.com/video/7635316204126686479
|
||||
https://www.douyin.com/video/7635316223533649204
|
||||
https://www.douyin.com/video/7635316263794838799
|
||||
https://www.douyin.com/video/7635316290885815604
|
||||
https://www.douyin.com/video/7635316298918006016
|
||||
https://www.douyin.com/video/7635316356996467983
|
||||
https://www.douyin.com/video/7635316366446316800
|
||||
https://www.douyin.com/video/7635316380048346420
|
||||
https://www.douyin.com/video/7635316424906378548
|
||||
https://www.douyin.com/video/7635316445332770100
|
||||
https://www.douyin.com/video/7635316494372490548
|
||||
https://www.douyin.com/video/7635316504510221603
|
||||
https://www.douyin.com/video/7635316517235723555
|
||||
https://www.douyin.com/video/7635316535434759439
|
||||
https://www.douyin.com/video/7635316579839855912
|
||||
https://www.douyin.com/video/7635316599574105379
|
||||
https://www.douyin.com/video/7635316640426626339
|
||||
https://www.douyin.com/video/7635316664560618786
|
||||
https://www.douyin.com/video/7635316678255004943
|
||||
https://www.douyin.com/video/7635316695493594408
|
||||
https://www.douyin.com/video/7635316741647699240
|
||||
https://www.douyin.com/video/7635316765005794560
|
||||
https://www.douyin.com/video/7635316798203759887
|
||||
https://www.douyin.com/video/7636339035187956607
|
||||
https://www.douyin.com/video/7636679442659691828
|
||||
https://www.douyin.com/video/7637061523759795498
|
||||
https://www.douyin.com/video/7637133310971497780
|
||||
https://www.douyin.com/video/7637842351964048666
|
||||
https://www.douyin.com/video/7638181849688182016
|
||||
https://www.douyin.com/video/7638869949355817833
|
||||
https://www.douyin.com/video/7639958454355292644
|
||||
https://www.douyin.com/video/7639984787955977512
|
||||
https://www.douyin.com/video/7640498341511482633
|
||||
https://www.douyin.com/video/7641193525614349577
|
||||
https://www.douyin.com/video/7641933242391711017
|
||||
https://www.douyin.com/video/7642704719329425829
|
||||
https://www.douyin.com/video/7642718672406023475
|
||||
https://www.douyin.com/video/7642942635859514283
|
||||
https://www.douyin.com/video/7643399569413311794
|
||||
https://www.douyin.com/video/7643399583900519707
|
||||
https://www.douyin.com/video/7643399621305273646
|
||||
https://www.douyin.com/video/7643399652410232114
|
||||
https://www.douyin.com/video/7643399679262149915
|
||||
https://www.douyin.com/video/7643399706835520818
|
||||
https://www.douyin.com/video/7643399731854511370
|
||||
https://www.douyin.com/video/7643399762783292699
|
||||
https://www.douyin.com/video/7643836448696749321
|
||||
https://www.douyin.com/video/7643836482079165742
|
||||
https://www.douyin.com/video/7643836507152731430
|
||||
https://www.douyin.com/video/7643836537339202867
|
||||
https://www.douyin.com/video/7643836556603690267
|
||||
https://www.douyin.com/video/7643836587238853926
|
||||
https://www.douyin.com/video/7643836628036783411
|
||||
https://www.douyin.com/video/7643836645401300250
|
||||
https://www.douyin.com/video/7644524372230142578
|
||||
https://www.douyin.com/video/7644942938593627455
|
||||
https://www.douyin.com/video/7644964061846850868
|
||||
https://www.douyin.com/video/7645680643090746678
|
||||
https://www.douyin.com/video/7645885150408294633
|
||||
https://www.douyin.com/video/7646438552036805951
|
||||
https://www.douyin.com/video/7646438584496442660
|
||||
https://www.douyin.com/video/7646438637361564947
|
||||
https://www.douyin.com/video/7646438674032282922
|
||||
https://www.douyin.com/video/7646438713542659382
|
||||
https://www.douyin.com/video/7646438745310268735
|
||||
https://www.douyin.com/video/7646438761672363318
|
||||
https://www.douyin.com/video/7646438789157588287
|
||||
https://www.douyin.com/video/7646438828449795371
|
||||
https://www.douyin.com/video/7646438847320050950
|
||||
https://www.douyin.com/video/7646438887044189459
|
||||
https://www.douyin.com/video/7646438946464992553
|
||||
https://www.douyin.com/video/7648511196286795689
|
||||
https://www.douyin.com/video/7648947051494154687
|
||||
https://www.douyin.com/video/7649956286361405062
|
||||
https://www.douyin.com/video/7651929550519012642
|
||||
https://www.douyin.com/video/7652243566080658723
|
||||
https://www.douyin.com/video/7652347898604672932
|
||||
https://www.douyin.com/video/7653300362463546624
|
||||
https://www.douyin.com/video/7653356746279597750
|
||||
https://www.douyin.com/video/7654943277090721059
|
||||
https://www.douyin.com/video/7655139806577511720
|
||||
https://www.douyin.com/video/7655552438715439103
|
||||
https://www.douyin.com/video/7655888850384297471
|
||||
https://www.douyin.com/video/7656070078584065318
|
||||
https://www.douyin.com/video/7656070122267839753
|
||||
https://www.douyin.com/video/7657735435922148585
|
||||
https://www.douyin.com/video/7658255875513077169
|
||||
https://www.douyin.com/video/7658256251562776933
|
||||
https://www.douyin.com/video/7658320102064339633
|
||||
https://www.douyin.com/video/7658516450584239798
|
||||
https://www.douyin.com/video/7660041691113458987
|
||||
https://www.douyin.com/video/7661469312670209515
|
||||
https://www.douyin.com/video/7661931085793676544
|
||||
https://www.douyin.com/video/7661931107813838114
|
||||
https://www.douyin.com/video/7661931138289503522
|
||||
https://www.douyin.com/video/7661931167666507042
|
||||
https://www.douyin.com/video/7661931190332542242
|
||||
https://www.douyin.com/video/7661931197441821952
|
||||
https://www.douyin.com/video/7661931283458575650
|
||||
https://www.douyin.com/video/7661931342665354496
|
||||
https://www.douyin.com/video/7661931357584624948
|
||||
https://www.douyin.com/video/7661931395362639104
|
||||
https://www.douyin.com/video/7661931415226928419
|
||||
https://www.douyin.com/video/7661931424836095267
|
||||
https://www.douyin.com/video/7661931451125992744
|
||||
https://www.douyin.com/video/7661931507992268072
|
||||
https://www.douyin.com/video/7661931551202102563
|
||||
https://www.douyin.com/video/7661931569287892264
|
||||
https://www.douyin.com/video/7661931630545636608
|
||||
https://www.douyin.com/video/7661931643493436706
|
||||
https://www.douyin.com/video/7661931691321068835
|
||||
https://www.douyin.com/video/7661931718554766626
|
||||
https://www.douyin.com/video/7661931741958900992
|
||||
https://www.douyin.com/video/7661931776390040867
|
||||
https://www.douyin.com/video/7661931836888681780
|
||||
https://www.douyin.com/video/7661931862633286946
|
||||
https://www.douyin.com/video/7662218035515641140
|
||||
https://www.douyin.com/video/7662218053077208360
|
||||
https://www.douyin.com/video/7662218088900840704
|
||||
https://www.douyin.com/video/7662218102171634979
|
||||
https://www.douyin.com/video/7662218132945259811
|
||||
https://www.douyin.com/video/7662218165438516515
|
||||
https://www.douyin.com/video/7662218196191137059
|
||||
https://www.douyin.com/video/7662218232337616180
|
||||
https://www.douyin.com/video/7662218266663718144
|
||||
https://www.douyin.com/video/7662218286230244642
|
||||
https://www.douyin.com/video/7662218312989855016
|
||||
https://www.douyin.com/video/7662218347819306280
|
||||
https://www.douyin.com/video/7662218380635540776
|
||||
https://www.douyin.com/video/7662218399677680911
|
||||
https://www.douyin.com/video/7662218434167540992
|
||||
https://www.douyin.com/video/7662218452119178511
|
||||
https://www.douyin.com/video/7662218489884675368
|
||||
https://www.douyin.com/video/7662218514773691663
|
||||
https://www.douyin.com/video/7662218545916284212
|
||||
https://www.douyin.com/video/7662218571778395392
|
||||
https://www.douyin.com/video/7662218600618429731
|
||||
https://www.douyin.com/video/7662218640300756259
|
||||
https://www.douyin.com/video/7662218664111820066
|
||||
https://www.douyin.com/video/7662218697624309007
|
||||
https://www.douyin.com/video/7662218710391770420
|
||||
https://www.douyin.com/video/7662218752864947508
|
||||
https://www.douyin.com/video/7662218774661106978
|
||||
https://www.douyin.com/video/7662218802544839936
|
||||
https://www.douyin.com/video/7662218830264962304
|
||||
https://www.douyin.com/video/7662218864784067874
|
||||
https://www.douyin.com/video/7662218886854561039
|
||||
https://www.douyin.com/video/7662218915753381172
|
||||
https://www.douyin.com/video/7662218948678651151
|
||||
https://www.douyin.com/video/7662218971994803491
|
||||
https://www.douyin.com/video/7662219008879463714
|
||||
https://www.douyin.com/video/7662219029083360546
|
||||
https://www.douyin.com/video/7662219060616170792
|
||||
https://www.douyin.com/video/7662219091423366415
|
||||
https://www.douyin.com/video/7662219114114551055
|
||||
https://www.douyin.com/video/7662219151301152000
|
||||
https://www.douyin.com/video/7662219174516854016
|
||||
https://www.douyin.com/video/7662219199598710051
|
||||
https://www.douyin.com/video/7662219244079287604
|
||||
https://www.douyin.com/video/7662697422921646565
|
||||
https://www.douyin.com/video/7663057070836501796
|
||||
https://www.douyin.com/video/7663381590835285275
|
||||
https://www.douyin.com/video/7663415295486250275
|
||||
https://www.douyin.com/video/7664341370543694565
|
||||
https://www.douyin.com/video/7665209859366236342
|
||||
https://www.douyin.com/video/7665546702552794212
|
||||
https://www.douyin.com/video/7666446552147315983
|
||||
https://www.douyin.com/video/7666446614600404224
|
||||
https://www.douyin.com/video/7667226159591970277
|
||||
https://www.douyin.com/video/7667888097522158884
|
||||
https://www.douyin.com/video/7668949964085438890
|
||||
https://www.douyin.com/video/7668970566061165864
|
||||
https://www.douyin.com/video/7668970593844235535
|
||||
https://www.douyin.com/video/7668970615088467240
|
||||
https://www.douyin.com/video/7668970621757426995
|
||||
https://www.douyin.com/video/7668970652136721715
|
||||
https://www.douyin.com/video/7668970688992087342
|
||||
https://www.douyin.com/video/7668970709305101577
|
||||
https://www.douyin.com/video/7668970732763876634
|
||||
https://www.douyin.com/video/7668970755870248202
|
||||
https://www.douyin.com/video/7668970793174355241
|
||||
https://www.douyin.com/video/7668970808127016235
|
||||
https://www.douyin.com/video/7668970834920213779
|
||||
https://www.douyin.com/video/7668970849373900073
|
||||
https://www.douyin.com/video/7668970879883201811
|
||||
https://www.douyin.com/video/7668970897864265002
|
||||
https://www.douyin.com/video/7668970920089898283
|
||||
https://www.douyin.com/video/7668970947268955432
|
||||
https://www.douyin.com/video/7668970963693882639
|
||||
https://www.douyin.com/video/7668970995868290338
|
||||
https://www.douyin.com/video/7668971017531870498
|
||||
https://www.douyin.com/video/7668971039824612660
|
||||
https://www.douyin.com/video/7668971073014189321
|
||||
https://www.douyin.com/video/7668971097697570100
|
||||
https://www.douyin.com/video/7668971113472347443
|
||||
https://www.douyin.com/video/7668971142786338074
|
||||
https://www.douyin.com/video/7668971155079941403
|
||||
https://www.douyin.com/video/7668971180715560202
|
||||
https://www.douyin.com/video/7668971212848123177
|
||||
https://www.douyin.com/video/7668971226374720787
|
||||
https://www.douyin.com/video/7668971259891354923
|
||||
https://www.douyin.com/video/7669621272674099627
|
||||
https://www.douyin.com/video/7670203373631586993
|
||||
https://www.douyin.com/video/7670746061603404266
|
||||
https://www.douyin.com/video/7671489185173820406
|
||||
https://www.douyin.com/video/7671561244180057387
|
||||
https://www.douyin.com/video/7671561244997979398
|
||||
https://www.douyin.com/video/7671875606372099769
|
||||
https://www.douyin.com/video/7672299769552702756
|
||||
https://www.douyin.com/video/7672449961476295945
|
||||
https://www.douyin.com/video/7673097678699400458
|
||||
https://www.douyin.com/video/7673855632486840959
|
||||
https://www.douyin.com/video/7673962055442976019
|
||||
https://www.douyin.com/video/7674961051557993983
|
||||
https://www.douyin.com/video/7675723718471269247
|
||||
https://www.douyin.com/video/7676226287258187941
|
||||
https://www.douyin.com/video/7676750698175860008
|
||||
https://www.douyin.com/video/7677128559848299434
|
||||
https://www.douyin.com/video/7677634241332530478
|
||||
https://www.douyin.com/video/7678205849635159695
|
||||
https://www.douyin.com/video/7678623873885130019
|
||||
https://www.douyin.com/video/7678638721272433929
|
||||
https://www.douyin.com/video/7678703788907171072
|
||||
https://www.douyin.com/video/7678758443209035058
|
||||
https://www.douyin.com/video/7678940960503434542
|
||||
https://www.douyin.com/video/7678984837763598582
|
||||
https://www.douyin.com/video/7678993228523228416
|
||||
https://www.douyin.com/video/7679067350875901230
|
||||
https://www.douyin.com/video/7679105824676990235
|
||||
https://www.douyin.com/video/7679335586229996800
|
||||
https://www.douyin.com/video/7679492601623416127
|
||||
https://www.douyin.com/video/7679619730159766822
|
||||
https://www.douyin.com/video/7679741025519654803
|
||||
https://www.douyin.com/video/7680045498736201003
|
||||
https://www.douyin.com/video/7680046931812505770
|
||||
https://www.douyin.com/video/7680189142793227572
|
||||
https://www.douyin.com/video/7680392554445032744
|
||||
https://www.douyin.com/video/7433698967734734120
|
||||
https://www.douyin.com/video/7464796192308858163
|
||||
https://www.douyin.com/video/7495303857429204283
|
||||
https://www.douyin.com/video/7506844801183091980
|
||||
https://www.douyin.com/video/7507459959304637750
|
||||
https://www.douyin.com/video/7512465191726468411
|
||||
https://www.douyin.com/video/7538701345206832438
|
||||
https://www.douyin.com/video/7540828775166545212
|
||||
https://www.douyin.com/video/7541675694007438650
|
||||
https://www.douyin.com/video/7542040788307283257
|
||||
https://www.douyin.com/video/7542509036920311097
|
||||
https://www.douyin.com/video/7542561195904224572
|
||||
https://www.douyin.com/video/7542768053986135356
|
||||
https://www.douyin.com/video/7543312766431481148
|
||||
https://www.douyin.com/video/7543656576851561787
|
||||
https://www.douyin.com/video/7543881121785105724
|
||||
https://www.douyin.com/video/7544777829592010027
|
||||
https://www.douyin.com/video/7550593764830579995
|
||||
https://www.douyin.com/video/7551253123344846121
|
||||
https://www.douyin.com/video/7552099919394721074
|
||||
https://www.douyin.com/video/7553721758868819241
|
||||
https://www.douyin.com/video/7553725286257872170
|
||||
https://www.douyin.com/video/7553726079598218515
|
||||
https://www.douyin.com/video/7553949930626125107
|
||||
https://www.douyin.com/video/7556127462901583143
|
||||
https://www.douyin.com/video/7557016051545427260
|
||||
https://www.douyin.com/video/7557304971768991003
|
||||
https://www.douyin.com/video/7557309880038640954
|
||||
https://www.douyin.com/video/7557342898887937337
|
||||
https://www.douyin.com/video/7560698483473911078
|
||||
https://www.douyin.com/video/7564713147735526662
|
||||
https://www.douyin.com/video/7565140842861284667
|
||||
https://www.douyin.com/video/7565678143320689977
|
||||
https://www.douyin.com/video/7568105965745702372
|
||||
https://www.douyin.com/video/7568853059900654195
|
||||
https://www.douyin.com/video/7570274551029674889
|
||||
https://www.douyin.com/video/7570321736815552137
|
||||
https://www.douyin.com/video/7573132562995864486
|
||||
https://www.douyin.com/video/7574377445916175019
|
||||
https://www.douyin.com/video/7574382031670187219
|
||||
https://www.douyin.com/video/7575960113161613402
|
||||
https://www.douyin.com/video/7577037635287703411
|
||||
https://www.douyin.com/video/7577550226790787570
|
||||
https://www.douyin.com/video/7577628611085599859
|
||||
https://www.douyin.com/video/7577635516536109750
|
||||
https://www.douyin.com/video/7577641560546191979
|
||||
https://www.douyin.com/video/7577963828598006498
|
||||
https://www.douyin.com/video/7579019462911000202
|
||||
https://www.douyin.com/video/7579859975944071026
|
||||
https://www.douyin.com/video/7579924192284720870
|
||||
https://www.douyin.com/video/7580151398587922344
|
||||
https://www.douyin.com/video/7580157386016814671
|
||||
https://www.douyin.com/video/7580159693399831412
|
||||
https://www.douyin.com/video/7580377323942954085
|
||||
https://www.douyin.com/video/7580388092725515368
|
||||
https://www.douyin.com/video/7580394956884948864
|
||||
https://www.douyin.com/video/7580620107187210875
|
||||
https://www.douyin.com/video/7580626018387171176
|
||||
https://www.douyin.com/video/7580662055020317300
|
||||
https://www.douyin.com/video/7581070847160370533
|
||||
https://www.douyin.com/video/7581077985031479675
|
||||
https://www.douyin.com/video/7581087921794090634
|
||||
https://www.douyin.com/video/7581489020892286258
|
||||
https://www.douyin.com/video/7581489540586818825
|
||||
https://www.douyin.com/video/7582035082827926835
|
||||
https://www.douyin.com/video/7584341688141090088
|
||||
https://www.douyin.com/video/7585496206969818427
|
||||
https://www.douyin.com/video/7586118751117921595
|
||||
https://www.douyin.com/video/7586265564667071781
|
||||
https://www.douyin.com/video/7586454001065200953
|
||||
https://www.douyin.com/video/7587052387870444827
|
||||
https://www.douyin.com/video/7587196537789304091
|
||||
https://www.douyin.com/video/7587446797670452532
|
||||
https://www.douyin.com/video/7587448044849728783
|
||||
https://www.douyin.com/video/7588194471104632762
|
||||
https://www.douyin.com/video/7588487482376931493
|
||||
https://www.douyin.com/video/7588791631819731658
|
||||
https://www.douyin.com/video/7588908317163379571
|
||||
https://www.douyin.com/video/7588919048840399033
|
||||
https://www.douyin.com/video/7589127362404041627
|
||||
https://www.douyin.com/video/7590096280598188523
|
||||
https://www.douyin.com/video/7590207026211343626
|
||||
https://www.douyin.com/video/7590449998632179112
|
||||
https://www.douyin.com/video/7590453788919998312
|
||||
https://www.douyin.com/video/7590806848073780968
|
||||
https://www.douyin.com/video/7591110669455099546
|
||||
https://www.douyin.com/video/7591292418127230619
|
||||
https://www.douyin.com/video/7591883506797366646
|
||||
https://www.douyin.com/video/7592023205097192794
|
||||
https://www.douyin.com/video/7592099294716042418
|
||||
https://www.douyin.com/video/7592390534959593454
|
||||
https://www.douyin.com/video/7592809534818733806
|
||||
https://www.douyin.com/video/7593354379336934577
|
||||
https://www.douyin.com/video/7593356172943964773
|
||||
https://www.douyin.com/video/7593369478470715535
|
||||
https://www.douyin.com/video/7593390272953164392
|
||||
https://www.douyin.com/video/7593564907442805102
|
||||
https://www.douyin.com/video/7593638829124440307
|
||||
https://www.douyin.com/video/7593653145739701998
|
||||
https://www.douyin.com/video/7593656029210500467
|
||||
https://www.douyin.com/video/7593663933921047690
|
||||
https://www.douyin.com/video/7593967261745701987
|
||||
https://www.douyin.com/video/7593968011066922164
|
||||
https://www.douyin.com/video/7594314269124242874
|
||||
https://www.douyin.com/video/7594481307385064010
|
||||
https://www.douyin.com/video/7594617234997279386
|
||||
https://www.douyin.com/video/7594619977484269258
|
||||
https://www.douyin.com/video/7594668942623730035
|
||||
https://www.douyin.com/video/7595063318445789670
|
||||
https://www.douyin.com/video/7595212613442312655
|
||||
https://www.douyin.com/video/7595924519417910099
|
||||
https://www.douyin.com/video/7595926644872445859
|
||||
https://www.douyin.com/video/7596185230475142754
|
||||
https://www.douyin.com/video/7596290025680229028
|
||||
https://www.douyin.com/video/7596323573515555963
|
||||
https://www.douyin.com/video/7596649064205540721
|
||||
https://www.douyin.com/video/7596679623200868709
|
||||
https://www.douyin.com/video/7596706946221486243
|
||||
https://www.douyin.com/video/7596707276998066895
|
||||
https://www.douyin.com/video/7597075505905363941
|
||||
https://www.douyin.com/video/7597206852879821811
|
||||
https://www.douyin.com/video/7597293058317401707
|
||||
https://www.douyin.com/video/7597668967721416768
|
||||
https://www.douyin.com/video/7597786037558180586
|
||||
https://www.douyin.com/video/7597801533107941609
|
||||
https://www.douyin.com/video/7597805982861915813
|
||||
https://www.douyin.com/video/7597812557995213434
|
||||
https://www.douyin.com/video/7598004652042482600
|
||||
https://www.douyin.com/video/7598037325616334299
|
||||
https://www.douyin.com/video/7598326102309996526
|
||||
https://www.douyin.com/video/7599084045054519369
|
||||
https://www.douyin.com/video/7599815578216540134
|
||||
https://www.douyin.com/video/7599973946096934393
|
||||
https://www.douyin.com/video/7600566663646500774
|
||||
https://www.douyin.com/video/7600569180350231667
|
||||
https://www.douyin.com/video/7600927418229450842
|
||||
https://www.douyin.com/video/7601308862616807654
|
||||
https://www.douyin.com/video/7601352429963895473
|
||||
https://www.douyin.com/video/7601696173858583662
|
||||
https://www.douyin.com/video/7602663496509695461
|
||||
https://www.douyin.com/video/7603265820214512946
|
||||
https://www.douyin.com/video/7603705839341461594
|
||||
https://www.douyin.com/video/7604096102266559334
|
||||
https://www.douyin.com/video/7604121099990831305
|
||||
https://www.douyin.com/video/7604260701607805322
|
||||
https://www.douyin.com/video/7604266795591364682
|
||||
https://www.douyin.com/video/7604390284369827249
|
||||
https://www.douyin.com/video/7604501352417079291
|
||||
https://www.douyin.com/video/7604637395715431817
|
||||
https://www.douyin.com/video/7605005197412160754
|
||||
https://www.douyin.com/video/7605012997388896626
|
||||
https://www.douyin.com/video/7605379095523094258
|
||||
https://www.douyin.com/video/7605771935038288101
|
||||
https://www.douyin.com/video/7605966531276770534
|
||||
https://www.douyin.com/video/7606778214602262080
|
||||
https://www.douyin.com/video/7606862005216141705
|
||||
https://www.douyin.com/video/7607346629067314170
|
||||
https://www.douyin.com/video/7607422200546731379
|
||||
https://www.douyin.com/video/7607430271919286437
|
||||
https://www.douyin.com/video/7607499082919925755
|
||||
https://www.douyin.com/video/7608129969706005938
|
||||
https://www.douyin.com/video/7608133336553143731
|
||||
https://www.douyin.com/video/7608823449670244325
|
||||
https://www.douyin.com/video/7609165080932928627
|
||||
https://www.douyin.com/video/7609178002768865393
|
||||
https://www.douyin.com/video/7609243114637537774
|
||||
https://www.douyin.com/video/7610216873716747502
|
||||
https://www.douyin.com/video/7610375575123831525
|
||||
https://www.douyin.com/video/7610379408163698043
|
||||
https://www.douyin.com/video/7610420931296014218
|
||||
https://www.douyin.com/video/7610788481493047497
|
||||
https://www.douyin.com/video/7610790050036896201
|
||||
https://www.douyin.com/video/7610799622252426149
|
||||
https://www.douyin.com/video/7611054411524655609
|
||||
https://www.douyin.com/video/7611143485625996773
|
||||
https://www.douyin.com/video/7611912599945016243
|
||||
https://www.douyin.com/video/7612272421220315739
|
||||
https://www.douyin.com/video/7612283093549871689
|
||||
https://www.douyin.com/video/7613443721098954122
|
||||
https://www.douyin.com/video/7613788616028458163
|
||||
https://www.douyin.com/video/7614154312163584714
|
||||
https://www.douyin.com/video/7614769013458184858
|
||||
https://www.douyin.com/video/7618369488867068594
|
||||
https://www.douyin.com/video/7618621798645655194
|
||||
https://www.douyin.com/video/7619763243272191409
|
||||
https://www.douyin.com/video/7619880183122074862
|
||||
https://www.douyin.com/video/7620968566594347110
|
||||
https://www.douyin.com/video/7621331487908878194
|
||||
https://www.douyin.com/video/7621974163138711214
|
||||
https://www.douyin.com/video/7622574727391029178
|
||||
https://www.douyin.com/video/7622575767101211569
|
||||
https://www.douyin.com/video/7622576693249580219
|
||||
https://www.douyin.com/video/7622577382738844603
|
||||
https://www.douyin.com/video/7623193019621576347
|
||||
https://www.douyin.com/video/7623391518167367526
|
||||
https://www.douyin.com/video/7623549232067271654
|
||||
https://www.douyin.com/video/7624509633576101499
|
||||
https://www.douyin.com/video/7624814177739787889
|
||||
https://www.douyin.com/video/7624841196913755121
|
||||
https://www.douyin.com/video/7625300200428922266
|
||||
https://www.douyin.com/video/7625668208544558523
|
||||
https://www.douyin.com/video/7625787550838330250
|
||||
https://www.douyin.com/video/7625996989730245283
|
||||
https://www.douyin.com/video/7626179454759549882
|
||||
https://www.douyin.com/video/7626306252448540131
|
||||
https://www.douyin.com/video/7626317662599703110
|
||||
https://www.douyin.com/video/7626351694943828883
|
||||
https://www.douyin.com/video/7626373431710268322
|
||||
https://www.douyin.com/video/7626514382880668262
|
||||
https://www.douyin.com/video/7626525300621251392
|
||||
https://www.douyin.com/video/7626898975647879014
|
||||
https://www.douyin.com/video/7626904838442180431
|
||||
https://www.douyin.com/video/7627269341940254435
|
||||
https://www.douyin.com/video/7627273762317137728
|
||||
https://www.douyin.com/video/7627452491014400434
|
||||
https://www.douyin.com/video/7627640946021222258
|
||||
https://www.douyin.com/video/7627791599888606452
|
||||
https://www.douyin.com/video/7628382270520418930
|
||||
https://www.douyin.com/video/7628499181401977253
|
||||
https://www.douyin.com/video/7628998055552151232
|
||||
https://www.douyin.com/video/7629134053551417512
|
||||
https://www.douyin.com/video/7629337784075471203
|
||||
https://www.douyin.com/video/7629377455199190671
|
||||
https://www.douyin.com/video/7629667721244818895
|
||||
https://www.douyin.com/video/7630238186673142504
|
||||
https://www.douyin.com/video/7630481013077415272
|
||||
https://www.douyin.com/video/7630482000382375104
|
||||
https://www.douyin.com/video/7630610031935653007
|
||||
https://www.douyin.com/video/7630760366989272936
|
||||
https://www.douyin.com/video/7630983513302965851
|
||||
https://www.douyin.com/video/7631145987604583921
|
||||
https://www.douyin.com/video/7631365186067522383
|
||||
https://www.douyin.com/video/7631411443455487099
|
||||
https://www.douyin.com/video/7631870088674057320
|
||||
https://www.douyin.com/video/7631952471640582321
|
||||
https://www.douyin.com/video/7632091005480074432
|
||||
https://www.douyin.com/video/7632100847442779535
|
||||
https://www.douyin.com/video/7632176798050410850
|
||||
https://www.douyin.com/video/7632619088523977690
|
||||
https://www.douyin.com/video/7632838030857000026
|
||||
https://www.douyin.com/video/7632839301714046115
|
||||
https://www.douyin.com/video/7633197171035519075
|
||||
https://www.douyin.com/video/7633468311833374950
|
||||
https://www.douyin.com/video/7633499322625701358
|
||||
https://www.douyin.com/video/7633580630672337097
|
||||
https://www.douyin.com/video/7633679557077273910
|
||||
https://www.douyin.com/video/7633949502793552820
|
||||
https://www.douyin.com/video/7634174075011095770
|
||||
https://www.douyin.com/video/7634192569665203363
|
||||
https://www.douyin.com/video/7634326396873748559
|
||||
https://www.douyin.com/video/7634490747660657059
|
||||
https://www.douyin.com/video/7634801908238843791
|
||||
https://www.douyin.com/video/7635301535589435107
|
||||
https://www.douyin.com/video/7635518586224318195
|
||||
https://www.douyin.com/video/7635819894857410688
|
||||
https://www.douyin.com/video/7635892928008165050
|
||||
https://www.douyin.com/video/7635895818876490664
|
||||
https://www.douyin.com/video/7636542453062950336
|
||||
https://www.douyin.com/video/7636546948492899712
|
||||
https://www.douyin.com/video/7636660856373876901
|
||||
https://www.douyin.com/video/7636919603901669108
|
||||
https://www.douyin.com/video/7637109323545959284
|
||||
https://www.douyin.com/video/7637288453793885858
|
||||
https://www.douyin.com/video/7637475460754188648
|
||||
https://www.douyin.com/video/7637515496460672627
|
||||
https://www.douyin.com/video/7637665267543820968
|
||||
https://www.douyin.com/video/7638032134292445800
|
||||
https://www.douyin.com/video/7638200653047250815
|
||||
https://www.douyin.com/video/7638388752579505512
|
||||
https://www.douyin.com/video/7639369373060554356
|
||||
https://www.douyin.com/video/7639369904672932579
|
||||
https://www.douyin.com/video/7639989670817942820
|
||||
https://www.douyin.com/video/7640850746828780724
|
||||
https://www.douyin.com/video/7641440697664556351
|
||||
https://www.douyin.com/video/7641566174120398504
|
||||
https://www.douyin.com/video/7641638687881716082
|
||||
https://www.douyin.com/video/7641639195727950811
|
||||
https://www.douyin.com/video/7642113955414440692
|
||||
https://www.douyin.com/video/7642498563649423796
|
||||
https://www.douyin.com/video/7642534545985170651
|
||||
https://www.douyin.com/video/7642562788644212011
|
||||
https://www.douyin.com/video/7642864713538415055
|
||||
https://www.douyin.com/video/7642866734747867727
|
||||
https://www.douyin.com/video/7642934586229476649
|
||||
https://www.douyin.com/video/7643233091314462927
|
||||
https://www.douyin.com/video/7643800575389737359
|
||||
https://www.douyin.com/video/7644085432163211235
|
||||
https://www.douyin.com/video/7644181694957866280
|
||||
https://www.douyin.com/video/7644192133059029193
|
||||
https://www.douyin.com/video/7644339751276866255
|
||||
https://www.douyin.com/video/7644520394418114767
|
||||
https://www.douyin.com/video/7644825112927552168
|
||||
https://www.douyin.com/video/7645622929387965737
|
||||
https://www.douyin.com/video/7647391912759987492
|
||||
https://www.douyin.com/video/7648296162151783119
|
||||
https://www.douyin.com/video/7648427494836522722
|
||||
https://www.douyin.com/video/7650876507203865582
|
||||
https://www.douyin.com/video/7651250186241233059
|
||||
https://www.douyin.com/video/7651625585542638991
|
||||
https://www.douyin.com/video/7652645553058088787
|
||||
https://www.douyin.com/video/7653434228675987065
|
||||
https://www.douyin.com/video/7653995176151290217
|
||||
https://www.douyin.com/video/7654557816787303827
|
||||
https://www.douyin.com/video/7654939453601182443
|
||||
https://www.douyin.com/video/7655183708684787691
|
||||
https://www.douyin.com/video/7656221180017303039
|
||||
https://www.douyin.com/video/7656310984231825050
|
||||
https://www.douyin.com/video/7657729145178568851
|
||||
https://www.douyin.com/video/7658597149303152997
|
||||
https://www.douyin.com/video/7658888903580400869
|
||||
https://www.douyin.com/video/7659048357873365321
|
||||
https://www.douyin.com/video/7659051992136263653
|
||||
https://www.douyin.com/video/7659928756119325155
|
||||
https://www.douyin.com/video/7660018534835047726
|
||||
https://www.douyin.com/video/7660368742513639162
|
||||
https://www.douyin.com/video/7660442087179422705
|
||||
https://www.douyin.com/video/7660753830233050368
|
||||
https://www.douyin.com/video/7660886310495916198
|
||||
https://www.douyin.com/video/7661054141707764465
|
||||
https://www.douyin.com/video/7661304061936922993
|
||||
https://www.douyin.com/video/7661556062293790073
|
||||
https://www.douyin.com/video/7661903439354258043
|
||||
https://www.douyin.com/video/7661921217062696441
|
||||
https://www.douyin.com/video/7661927054447957169
|
||||
https://www.douyin.com/video/7661927362422435321
|
||||
https://www.douyin.com/video/7661938193302450929
|
||||
https://www.douyin.com/video/7661943639765449137
|
||||
https://www.douyin.com/video/7662650281771930865
|
||||
https://www.douyin.com/video/7663001683631615409
|
||||
https://www.douyin.com/video/7663003105881069169
|
||||
https://www.douyin.com/video/7663068486411357873
|
||||
https://www.douyin.com/video/7663424935708572785
|
||||
https://www.douyin.com/video/7663805570860004810
|
||||
https://www.douyin.com/video/7664442574513639269
|
||||
https://www.douyin.com/video/7664482373987529137
|
||||
https://www.douyin.com/video/7664505249533845617
|
||||
https://www.douyin.com/video/7664582094164712433
|
||||
https://www.douyin.com/video/7664582866961897210
|
||||
https://www.douyin.com/video/7664622902432573809
|
||||
https://www.douyin.com/video/7664655382844902682
|
||||
https://www.douyin.com/video/7664783867907499240
|
||||
https://www.douyin.com/video/7664794487819001841
|
||||
https://www.douyin.com/video/7664795428417196529
|
||||
https://www.douyin.com/video/7664886189099108473
|
||||
https://www.douyin.com/video/7664893250041720058
|
||||
https://www.douyin.com/video/7665331346969312881
|
||||
https://www.douyin.com/video/7665353074157146545
|
||||
https://www.douyin.com/video/7665544450933043377
|
||||
https://www.douyin.com/video/7665551203161884529
|
||||
https://www.douyin.com/video/7665985902460201329
|
||||
https://www.douyin.com/video/7666148481166800049
|
||||
https://www.douyin.com/video/7666471527055626932
|
||||
https://www.douyin.com/video/7667231182241755825
|
||||
https://www.douyin.com/video/7667257077496603109
|
||||
https://www.douyin.com/video/7667425025976313445
|
||||
https://www.douyin.com/video/7670588545283971441
|
||||
https://www.douyin.com/video/7671910059202350370
|
||||
https://www.douyin.com/video/7672039534930308785
|
||||
https://www.douyin.com/video/7672307700373921956
|
||||
https://www.douyin.com/video/7672323816546818481
|
||||
https://www.douyin.com/video/7672556460832535034
|
||||
https://www.douyin.com/video/7673479545232496219
|
||||
https://www.douyin.com/video/7674268500702031990
|
||||
https://www.douyin.com/video/7674550014517696586
|
||||
https://www.douyin.com/video/7674784828718340863
|
||||
https://www.douyin.com/video/7675052407059298408
|
||||
https://www.douyin.com/video/7675174050017436136
|
||||
https://www.douyin.com/video/7675196882150542001
|
||||
https://www.douyin.com/video/7675275660490924724
|
||||
https://www.douyin.com/video/7675615053123912438
|
||||
https://www.douyin.com/video/7676685798481319016
|
||||
https://www.douyin.com/video/7678101190555682530
|
||||
https://www.douyin.com/video/7678639376749940014
|
||||
533
pipeline.py
Normal file
533
pipeline.py
Normal file
@@ -0,0 +1,533 @@
|
||||
#!/usr/bin/env python3
|
||||
"""抖音采集一体化流水线：下载 -> 抽 mp3 -> 传 COS -> 记元信息 -> 删本地。
|
||||
|
||||
跟原来分阶段跑（全下完再全抽完再全传完）最大的区别是磁盘占用：
|
||||
分阶段必须把所有 mp4 摊在盘上（上一批是 53 GB），这里每条走完就删，
|
||||
峰值只有 workers x (mp4+mp3)，约 100 MB，且与总量无关。
|
||||
|
||||
每条视频端到端走 9 步：
|
||||
|
||||
1. fetch_detail 拿元信息 + raw
|
||||
2. 下载 mp4
|
||||
3. ffmpeg 抽 mp3 [信号量限流，见下]
|
||||
4. ffprobe 探两个的时长
|
||||
5. 校验 mp3 时长 ~= mp4 时长 <- 抽取闸门
|
||||
6. 上传 mp4（覆盖同名）
|
||||
7. 上传 mp3（覆盖同名）
|
||||
8. head_object 核对大小 <- 删除闸门（在 CosUploader.put 里）
|
||||
9. 写 record，删本地两个文件
|
||||
|
||||
任何一步失败：保留本地文件不删，写进 failed 文件，继续下一条。
|
||||
|
||||
并发：下载和上传是网络密集，按 --workers 并发；ffmpeg 是 CPU 密集，
|
||||
单独用信号量限到 CPU 核数，避免 8 个 worker 同时撞进第 3 步把 CPU 打满。
|
||||
|
||||
用法:
|
||||
$env:COS_SECRET_ID="xxx"; $env:COS_SECRET_KEY="yyy"
|
||||
python pipeline.py --from-file links.txt --region ap-chongqing --limit 20 --keep-local
|
||||
python pipeline.py --from-file links.txt --region ap-chongqing --workers 8
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
import requests
|
||||
|
||||
from douyin import (
|
||||
AppClient,
|
||||
DeviceRegisterError,
|
||||
DouyinError,
|
||||
NotAVideo,
|
||||
ParseError,
|
||||
download,
|
||||
extract_aweme_id,
|
||||
)
|
||||
from douyin.audio import build_cmd, ensure_ffmpeg, probe
|
||||
from douyin.batch import DevicePool
|
||||
from douyin.cos import CosError, CosUploader, DEFAULT_BUCKET, DEFAULT_PREFIX
|
||||
from douyin.meta import RecordLog, build_record
|
||||
|
||||
|
||||
class StageError(RuntimeError):
|
||||
"""带阶段名的失败，便于 failed 文件里归因。"""
|
||||
|
||||
def __init__(self, stage: str, msg: str) -> None:
|
||||
super().__init__(msg)
|
||||
self.stage = stage
|
||||
|
||||
|
||||
@dataclass
|
||||
class Stats:
|
||||
total: int = 0
|
||||
done: int = 0
|
||||
skipped: int = 0
|
||||
failed: int = 0
|
||||
bytes_up: int = 0
|
||||
lock: threading.Lock = field(default_factory=threading.Lock)
|
||||
|
||||
|
||||
_tls = threading.local()
|
||||
|
||||
|
||||
def session() -> requests.Session:
|
||||
"""每个线程一个下载用 session。CDN 直链不需要设备头，裸 session 即可。"""
|
||||
s = getattr(_tls, "dl", None)
|
||||
if s is None:
|
||||
s = requests.Session()
|
||||
s.headers.update({"User-Agent": "okhttp/3.12.13"})
|
||||
_tls.dl = s
|
||||
return s
|
||||
|
||||
|
||||
def read_targets(args) -> List[str]:
|
||||
raw: List[str] = list(args.target or [])
|
||||
if args.from_file:
|
||||
raw += [ln.strip() for ln in
|
||||
Path(args.from_file).read_text(encoding="utf-8").splitlines()
|
||||
if ln.strip() and not ln.strip().startswith("#")]
|
||||
return raw
|
||||
|
||||
|
||||
def next_abort_wait(streak: int, base: int, cap: int) -> int:
|
||||
"""撞 IP 级注册限流后等多久再续跑。streak = 连着撞了几次（不含这次）。
|
||||
|
||||
抖音封的是 IP，不是设备指纹——DevicePool 换 UA 没用，只能等。实测一轮
|
||||
能跑 5~6 小时、44 次设备轮换才触顶，冷却大概几小时。连撞就翻倍，说明
|
||||
还没解封；有一段跑成了就把 streak 清零、等待打回底价。
|
||||
"""
|
||||
if base <= 0:
|
||||
return 0
|
||||
return min(base * (2 ** streak), cap)
|
||||
|
||||
|
||||
def plan_batch(pending, max_items: int):
|
||||
"""把待处理队列切一段出来跑，返回 (这一段, 还有没有剩下的)。
|
||||
|
||||
为什么要切段：单个 python 进程跑到 1.5 万条上下，Windows 会话的桌面堆
|
||||
就被几万个 ffmpeg/ffprobe 子进程耗光，之后 CreateProcess 一律失败
|
||||
（rc=3221225794 / STATUS_DLL_INIT_FAILED）。表现很好认：完成数冻结、
|
||||
失败数猛涨、上传量不动、work/ 暴涨（下载成功但抽音失败的 mp4 按设计
|
||||
留着）。实测撞过两次 —— 三进程并跑时一次，单进程跑到 16399 条又一次，
|
||||
所以是**累积量**的问题，不是并发数。
|
||||
|
||||
段跑完 os.execv 重启自己，桌面堆跟着重置。断点续跑本来就靠
|
||||
records.jsonl，重启不会重下。max_items=0 表示不切。
|
||||
"""
|
||||
if max_items and len(pending) > max_items:
|
||||
return pending[:max_items], True
|
||||
return pending, False
|
||||
|
||||
|
||||
def sweep_work_dir(work: Path, records: Path) -> int:
|
||||
"""清掉 work/ 里已经没用的残留，返回删了几个文件。
|
||||
|
||||
删两类：
|
||||
* aweme_id 已经在 records.jsonl 里的 —— 早传上 COS 了，留着纯属占地方；
|
||||
* `.part` 半截下载 —— 断点续传的 Range 起点，进程被杀后就是垃圾。
|
||||
|
||||
**不碰失败项的文件**：上传失败、抽音截断这些是故意留在本地的，重跑时
|
||||
省一次下载，见 test_upload_failure_keeps_local_and_writes_no_record。
|
||||
|
||||
为什么要有这个函数：正常路径删文件（见下面 unlink 那两行），失败路径
|
||||
故意不删，可失败项的文件从此再没人回收 —— 66411 条那批跑到 2.7% 就在
|
||||
work/ 里攒了 11 GB，其中 414 个的 aweme_id 已经在 records.jsonl 里。
|
||||
（那 414 个的 mp4 修改时间比记录时间还晚 17~166 秒，说明是记录写完之后
|
||||
才落地的，具体怎么产生的没查清；这个清扫无论如何都能兜住。）
|
||||
"""
|
||||
if not work.is_dir():
|
||||
return 0
|
||||
done = set()
|
||||
if records.exists():
|
||||
# aweme_id 是 RecordLog 写出的第一个字段，按行前缀扫，不解析整行 JSON
|
||||
# （records.jsonl 带 raw，1.8 GB，逐行 json.loads 要几分钟）。
|
||||
# 容忍冒号后的空格：紧凑写法没有，但别让格式差异把清扫悄悄变成空操作。
|
||||
pat = re.compile(rb'^\{\s*"aweme_id"\s*:\s*"(\d+)"')
|
||||
with open(records, "rb") as f:
|
||||
for line in f:
|
||||
m = pat.match(line)
|
||||
if m:
|
||||
done.add(m.group(1).decode())
|
||||
freed = 0
|
||||
for p in work.iterdir():
|
||||
if not p.is_file():
|
||||
continue
|
||||
if p.name.endswith(".part") or p.stem in done:
|
||||
try:
|
||||
p.unlink()
|
||||
freed += 1
|
||||
except OSError:
|
||||
pass # 还被占着，下一轮再说
|
||||
return freed
|
||||
|
||||
|
||||
def extract_mp3(ffmpeg: str, src: Path, dst: Path, bitrate: str,
|
||||
timeout: float = 600.0) -> None:
|
||||
cmd = build_cmd(ffmpeg, src, dst, "mp3", bitrate)
|
||||
try:
|
||||
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise StageError("audio", f"ffmpeg 超时 >{timeout}s")
|
||||
if p.returncode != 0 or not dst.exists() or dst.stat().st_size == 0:
|
||||
err = (p.stderr or "").strip().splitlines()
|
||||
raise StageError("audio", err[-1][:150] if err else f"rc={p.returncode}")
|
||||
|
||||
|
||||
def decoded_seconds(path: Path, ffprobe: str, timeout: float = 300.0) -> Optional[float]:
|
||||
"""实际解码算出的音频时长。
|
||||
|
||||
不能用 `format=duration`：LAME 写的 Xing 头声明了原始总帧数，ffprobe
|
||||
直接信它而不解码。实测把 mp3 截断到 20%，format=duration 报的还是
|
||||
10.0078s，和完整文件一模一样——拿它做截断校验等于没做。
|
||||
|
||||
`-count_frames` 会真解码。代价很小：120s 的 mp3 是 0.109s 对 0.053s。
|
||||
"""
|
||||
cmd = [ffprobe, "-v", "error", "-select_streams", "a:0", "-count_frames",
|
||||
"-show_entries", "stream=nb_read_frames,sample_rate",
|
||||
"-of", "default=noprint_wrappers=1"] # 保留 key= 前缀
|
||||
try:
|
||||
p = subprocess.run(cmd + [str(path)], capture_output=True, text=True,
|
||||
timeout=timeout)
|
||||
except subprocess.TimeoutExpired:
|
||||
return None
|
||||
|
||||
# 按 key 解析，不靠位置：ffprobe 是按流内字段顺序输出的，不是按请求顺序
|
||||
kv = {}
|
||||
for line in (p.stdout or "").splitlines():
|
||||
if "=" in line:
|
||||
k, _, v = line.partition("=")
|
||||
kv[k.strip()] = v.strip()
|
||||
try:
|
||||
frames = int(kv["nb_read_frames"])
|
||||
rate = int(kv["sample_rate"])
|
||||
except (KeyError, ValueError):
|
||||
return None
|
||||
if frames <= 0 or rate <= 0:
|
||||
return None
|
||||
# MPEG1 Layer III 每帧 1152 采样；MPEG2/2.5（<32kHz）是 576
|
||||
per_frame = 1152 if rate >= 32000 else 576
|
||||
return frames * per_frame / rate
|
||||
|
||||
|
||||
def check_duration(mp4: Path, mp3: Path, ffprobe: str) -> float:
|
||||
"""返回 mp4 的精确时长；顺便校验 mp3 没被截断。
|
||||
|
||||
ffmpeg 返回 0 但产出截断文件是见过的，光看 returncode 不够。mp4 传上去
|
||||
之后本地就删了，所以这道闸门必须在删除之前把话说死。
|
||||
"""
|
||||
d4 = probe(mp4, ffprobe)
|
||||
if not d4:
|
||||
raise StageError("probe", "ffprobe 读不出 mp4 时长")
|
||||
d3 = decoded_seconds(mp3, ffprobe)
|
||||
if not d3:
|
||||
raise StageError("probe", "mp3 解不出有效音频帧")
|
||||
|
||||
tol = max(1.0, d4 * 0.02)
|
||||
if abs(d3 - d4) > tol:
|
||||
raise StageError("probe",
|
||||
f"mp3 实解 {d3:.3f}s 与 mp4 {d4:.3f}s 相差超过 {tol:.3f}s，"
|
||||
"疑似抽取被截断")
|
||||
return d4
|
||||
|
||||
|
||||
def run_one(aweme_id: str, source_url: str, *, pool: DevicePool, up: CosUploader,
|
||||
log: RecordLog, args, ffmpeg: str, ff_sem: threading.Semaphore,
|
||||
stats: Stats, fail, abort: threading.Event) -> None:
|
||||
# 注册配额是 IP 级的，耗尽后换设备也没用。这时候必须整批停下来等，
|
||||
# 而不是让剩下几千条一条条去撞同一堵墙、全标成失败——上一批就是这么
|
||||
# 烧掉 678 条的。abort 一旦拉起，后续任务直接返回，不计失败。
|
||||
if abort.is_set():
|
||||
with stats.lock:
|
||||
stats.skipped += 1
|
||||
return
|
||||
|
||||
work = Path(args.work_dir)
|
||||
mp4 = work / f"{aweme_id}.mp4"
|
||||
mp3 = work / f"{aweme_id}.mp3"
|
||||
|
||||
try:
|
||||
# 1. 元信息
|
||||
try:
|
||||
device, gen = pool.current()
|
||||
except DeviceRegisterError as exc:
|
||||
abort.set()
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
fail(aweme_id, "abort", f"IP 级注册配额耗尽，已中止整批：{exc}")
|
||||
return
|
||||
try:
|
||||
aweme = AppClient(device).fetch_detail(aweme_id)
|
||||
except NotAVideo as exc:
|
||||
raise StageError("detail", str(exc))
|
||||
except DouyinError as exc:
|
||||
# 空 body 多半是设备配额到了，换设备重来一次
|
||||
if "空 body" in str(exc):
|
||||
pool.report_dead(gen)
|
||||
device, gen = pool.current()
|
||||
aweme = AppClient(device).fetch_detail(aweme_id)
|
||||
else:
|
||||
raise StageError("detail", str(exc))
|
||||
|
||||
if not aweme.video_urls and not aweme.uri:
|
||||
raise StageError("detail", "没有可用的视频直链")
|
||||
|
||||
# 2. 下载
|
||||
try:
|
||||
res = download(session(), aweme.video_urls or [aweme.play_url()],
|
||||
mp4, progress=False)
|
||||
except Exception as exc:
|
||||
raise StageError("download", f"{type(exc).__name__}: {exc}")
|
||||
if res.size <= 0:
|
||||
raise StageError("download", "下载产物是空文件")
|
||||
|
||||
# 3. 抽 mp3（CPU 密集，限流）
|
||||
with ff_sem:
|
||||
extract_mp3(ffmpeg, mp4, mp3, args.bitrate)
|
||||
|
||||
# 4~5. 探时长 + 校验没被截断
|
||||
duration = check_duration(mp4, mp3, args.ffprobe)
|
||||
|
||||
# 6~8. 上传，put() 内部做 head_object 大小校验
|
||||
try:
|
||||
url4, size4 = up.put(mp4, up.key_for(mp4.name))
|
||||
url3, size3 = up.put(mp3, up.key_for(mp3.name))
|
||||
except CosError as exc:
|
||||
raise StageError("upload", str(exc))
|
||||
|
||||
# 9. 记录，然后才敢删
|
||||
log.write(build_record(aweme, duration_sec=duration,
|
||||
cos_mp4=url4, cos_mp3=url3,
|
||||
size_mp4=size4, size_mp3=size3,
|
||||
source_url=source_url))
|
||||
|
||||
if not args.keep_local:
|
||||
mp4.unlink(missing_ok=True)
|
||||
mp3.unlink(missing_ok=True)
|
||||
|
||||
with stats.lock:
|
||||
stats.done += 1
|
||||
stats.bytes_up += size4 + size3
|
||||
|
||||
except DeviceRegisterError as exc:
|
||||
abort.set()
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
fail(aweme_id, "abort", f"IP 级注册配额耗尽，已中止整批：{exc}")
|
||||
except StageError as exc:
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
fail(aweme_id, exc.stage, str(exc))
|
||||
except Exception as exc: # 兜底，别让一条炸掉整批
|
||||
with stats.lock:
|
||||
stats.failed += 1
|
||||
fail(aweme_id, "unknown", f"{type(exc).__name__}: {exc}")
|
||||
|
||||
|
||||
def main(argv: Optional[List[str]] = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="抖音采集一体化流水线")
|
||||
ap.add_argument("target", nargs="*", help="链接或 aweme_id")
|
||||
ap.add_argument("--from-file", default="", help="从文件读链接，一行一个")
|
||||
ap.add_argument("--records", default="records/records.jsonl", help="元信息落点")
|
||||
ap.add_argument("--failed", default="records/pipeline_failed.txt")
|
||||
ap.add_argument("--work-dir", default="work", help="mp4/mp3 临时目录，传完即删")
|
||||
ap.add_argument("--region", required=True)
|
||||
ap.add_argument("--bucket", default=DEFAULT_BUCKET)
|
||||
ap.add_argument("--prefix", default=DEFAULT_PREFIX)
|
||||
ap.add_argument("--url-style", choices=["cos", "https", "path"], default="cos")
|
||||
ap.add_argument("--workers", type=int, default=8)
|
||||
ap.add_argument("--bitrate", default="192k")
|
||||
ap.add_argument("--ffmpeg", default="ffmpeg")
|
||||
ap.add_argument("--ffprobe", default="ffprobe")
|
||||
ap.add_argument("--ffmpeg-workers", type=int, default=0,
|
||||
help="并发 ffmpeg 数，0=CPU 核数")
|
||||
ap.add_argument("--device-quota", type=int, default=550,
|
||||
help="单设备 detail 调用上限。实测约 575 次后被限流，"
|
||||
"留一点余量；调高可减少注册次数")
|
||||
ap.add_argument("--limit", type=int, default=0, help="只跑前 N 条，用于试跑")
|
||||
ap.add_argument("--max-items", type=int, default=12000,
|
||||
help="单个进程最多跑多少条，跑满就重启自己接着跑。"
|
||||
"防 Windows 桌面堆耗尽，见 plan_batch。0=不切段")
|
||||
ap.add_argument("--abort-wait", type=int, default=1800,
|
||||
help="撞上 IP 级注册限流后等多少秒再自动续跑。"
|
||||
"连撞会翻倍，见 next_abort_wait。0=不等，直接退出")
|
||||
ap.add_argument("--abort-wait-cap", type=int, default=14400,
|
||||
help="上面那个等待的上限秒数，默认 4 小时")
|
||||
ap.add_argument("--keep-local", action="store_true",
|
||||
help="上传后不删本地文件（试跑时用）")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
targets = read_targets(args)
|
||||
if not targets:
|
||||
ap.error("没有目标。给链接，或用 --from-file。")
|
||||
|
||||
try:
|
||||
ffmpeg = ensure_ffmpeg(args.ffmpeg)
|
||||
except Exception as exc:
|
||||
print(f"[错误] {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
try:
|
||||
up = CosUploader(region=args.region, bucket=args.bucket, prefix=args.prefix,
|
||||
url_style=args.url_style)
|
||||
except CosError as exc:
|
||||
print(f"[错误] {exc}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
log = RecordLog(Path(args.records))
|
||||
print(f"已完成 {len(log)} 条（从 {args.records} 读出），本次输入 {len(targets)} 条")
|
||||
|
||||
# 解析 id，顺便去重和跳过已完成的
|
||||
pending = []
|
||||
seen = set()
|
||||
bad = 0
|
||||
for raw in targets:
|
||||
try:
|
||||
aid = extract_aweme_id(raw, session())
|
||||
except ParseError:
|
||||
bad += 1
|
||||
continue
|
||||
if aid in seen:
|
||||
continue
|
||||
seen.add(aid)
|
||||
if aid in log:
|
||||
continue
|
||||
pending.append((aid, raw))
|
||||
if args.limit:
|
||||
pending = pending[: args.limit]
|
||||
|
||||
skipped = len(seen) - len(pending) if not args.limit else 0
|
||||
print(f"待处理 {len(pending)} 条"
|
||||
+ (f"，跳过已完成 {skipped} 条" if skipped else "")
|
||||
+ (f"，无法解析 {bad} 条" if bad else ""))
|
||||
if not pending:
|
||||
return 0
|
||||
|
||||
pending, more = plan_batch(pending, args.max_items)
|
||||
if more:
|
||||
print(f"本段只跑 {len(pending)} 条，跑完自动重启进程接着来"
|
||||
f"（见 plan_batch 的注释：桌面堆）")
|
||||
|
||||
Path(args.work_dir).mkdir(parents=True, exist_ok=True)
|
||||
if not args.keep_local:
|
||||
n = sweep_work_dir(Path(args.work_dir), Path(args.records))
|
||||
if n:
|
||||
print(f"清掉 work/ 里 {n} 个已入库或半截的残留文件")
|
||||
stats = Stats(total=len(pending))
|
||||
fail_lock = threading.Lock()
|
||||
fail_path = Path(args.failed)
|
||||
|
||||
def fail(aweme_id: str, stage: str, msg: str) -> None:
|
||||
with fail_lock:
|
||||
with open(fail_path, "a", encoding="utf-8") as f:
|
||||
f.write(f"{aweme_id}\t{stage}\t{msg}\n")
|
||||
|
||||
pool = DevicePool(max_calls=args.device_quota)
|
||||
# DevicePool 是惰性注册的。先主动要一次，注册不了就当场退出，
|
||||
# 而不是让每个 worker 各自撞一遍、把整批标成失败。
|
||||
try:
|
||||
pool.current()
|
||||
except DeviceRegisterError as exc:
|
||||
# 开跑就注册不上 = IP 还在冷却期。这条路以前直接退出，等于
|
||||
# "撞限流后自动续跑"在冷却没结束时会白白放弃，所以这里也要等。
|
||||
streak = int(os.environ.get("_DY_ABORT_STREAK", "0"))
|
||||
wait = next_abort_wait(streak, args.abort_wait, args.abort_wait_cap)
|
||||
if wait:
|
||||
os.environ["_DY_ABORT_STREAK"] = str(streak + 1)
|
||||
until = time.strftime("%H:%M:%S", time.localtime(time.time() + wait))
|
||||
print(f"[限流] 设备注册失败：{exc}", file=sys.stderr)
|
||||
print(f"IP 还在冷却，等 {wait // 60} 分钟到 {until} 再试…", flush=True)
|
||||
sys.stdout.flush()
|
||||
sys.stderr.flush()
|
||||
time.sleep(wait)
|
||||
os.execv(sys.executable, [sys.executable] + sys.argv)
|
||||
print(f"[错误] 设备注册失败：{exc}", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
n_ff = args.ffmpeg_workers or (os.cpu_count() or 4)
|
||||
ff_sem = threading.Semaphore(n_ff)
|
||||
print(f"并发 {args.workers}，其中 ffmpeg 限 {n_ff}"
|
||||
+ ("，本地文件保留不删" if args.keep_local else ""))
|
||||
|
||||
t0 = time.time()
|
||||
stop = threading.Event()
|
||||
abort = threading.Event()
|
||||
|
||||
def report() -> None:
|
||||
ticks = 0
|
||||
while not stop.wait(15):
|
||||
ticks += 1
|
||||
with stats.lock:
|
||||
d, f_, tot = stats.done, stats.failed, stats.total
|
||||
gb = stats.bytes_up / 1024 ** 3
|
||||
el = time.time() - t0
|
||||
rate = d / el * 60 if el else 0
|
||||
left = (tot - d - f_) / rate if rate else 0
|
||||
print(f" [{d + f_}/{tot}] 完成{d} 失败{f_} | {gb:.2f}GB 上传 | "
|
||||
f"{rate:.0f}条/分 | 剩约 {left:.0f}分", flush=True)
|
||||
# 每 5 分钟回收一次：光靠开跑前扫一次，长批跑到一半磁盘就满了
|
||||
if ticks % 20 == 0 and not args.keep_local:
|
||||
n = sweep_work_dir(Path(args.work_dir), Path(args.records))
|
||||
if n:
|
||||
print(f" [清扫] 回收 {n} 个残留文件", flush=True)
|
||||
|
||||
threading.Thread(target=report, daemon=True).start()
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=args.workers) as ex:
|
||||
for aid, raw in pending:
|
||||
ex.submit(run_one, aid, raw, pool=pool, up=up, log=log, args=args,
|
||||
ffmpeg=ffmpeg, ff_sem=ff_sem, stats=stats, fail=fail,
|
||||
abort=abort)
|
||||
finally:
|
||||
stop.set()
|
||||
|
||||
el = time.time() - t0
|
||||
print(f"\n完成 {stats.done}，失败 {stats.failed}，"
|
||||
f"上传 {stats.bytes_up / 1024 ** 3:.2f} GB，耗时 {el / 60:.1f} 分")
|
||||
print(f"元信息 -> {args.records}（累计 {len(log)} 条）")
|
||||
if stats.failed:
|
||||
print(f"失败明细 -> {fail_path}（本地文件已保留，可直接重跑本脚本续传）")
|
||||
|
||||
def _restart(why: str, wait: int = 0) -> None:
|
||||
"""重启自己接着跑。断点续跑靠 records.jsonl，重启不会重下。"""
|
||||
if wait:
|
||||
until = time.strftime("%H:%M:%S", time.localtime(time.time() + wait))
|
||||
print(f"{why}，等 {wait // 60} 分钟到 {until} 再续…", flush=True)
|
||||
else:
|
||||
print(f"{why}，重启进程接着跑…", flush=True)
|
||||
sys.stdout.flush()
|
||||
sys.stderr.flush()
|
||||
if wait:
|
||||
time.sleep(wait)
|
||||
os.execv(sys.executable, [sys.executable] + sys.argv)
|
||||
|
||||
# 连撞计数走环境变量，execv 换掉进程映像后靠它记住上一轮的状态
|
||||
streak = int(os.environ.get("_DY_ABORT_STREAK", "0"))
|
||||
if abort.is_set():
|
||||
# IP 级注册配额耗尽。抖音封的是 IP，换设备没用，只能等冷却。
|
||||
wait = next_abort_wait(streak, args.abort_wait, args.abort_wait_cap)
|
||||
if wait:
|
||||
# 这一轮但凡出了活就说明限流是跑着跑着才触发的，下次从底价重等
|
||||
os.environ["_DY_ABORT_STREAK"] = "0" if stats.done else str(streak + 1)
|
||||
_restart("撞上 IP 级注册限流", wait)
|
||||
print("撞上 IP 级注册限流，已停下。等几小时 IP 冷却后重跑本命令即可续传。")
|
||||
return 5
|
||||
|
||||
os.environ["_DY_ABORT_STREAK"] = "0"
|
||||
if more:
|
||||
_restart("本段跑完")
|
||||
|
||||
return 0 if stats.failed == 0 else 4
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
11
requirements.txt
Normal file
11
requirements.txt
Normal file
@@ -0,0 +1,11 @@
|
||||
# 下载 / 接口调用（douyin/app.py、download.py、urls.py、pipeline.py）
|
||||
requests>=2.32
|
||||
|
||||
# COS 上传（douyin/cos.py，延迟导入，不传 COS 就不需要）
|
||||
cos-python-sdk-v5>=1.9.41
|
||||
|
||||
# xlsx 导出（jsonl_to_xlsx.py，延迟导入，不出 xlsx 就不需要）
|
||||
openpyxl>=3.1
|
||||
|
||||
# 另需系统里有 ffmpeg / ffprobe（pipeline.py 抽音频、探时长用）
|
||||
# 实测环境：Python 3.11.0 + ffmpeg 2025-07-31 gyan.dev full build
|
||||
174
scripts/clean_ai.py
Normal file
174
scripts/clean_ai.py
Normal file
@@ -0,0 +1,174 @@
|
||||
"""从抖音链接 jsonl 里筛出 AI 相关的作品。
|
||||
|
||||
判定逻辑在 douyin/aiclass.py，分 platform / strong / topic / author 四档，
|
||||
这里只负责流式读写、统计和落盘。
|
||||
|
||||
输出：
|
||||
*.txt 纯链接一行一个，直接喂给 pipeline.py --from-file
|
||||
*.jsonl 原记录 + ai_tier + ai_signals，字段全留着，方便复核和再切。
|
||||
这份体积接近源文件（实测 13967 条就有 169MB），而且随时能重跑
|
||||
生成，所以默认**不写**，要复核再加 --with-jsonl。
|
||||
|
||||
用法:
|
||||
python clean_ai.py -i 抖音链接20260827_40087.jsonl --stats
|
||||
python clean_ai.py -i 抖音链接20260827_40087.jsonl --tier platform,strong -o ai
|
||||
python clean_ai.py -i 抖音链接20260827_40087.jsonl --tier platform,strong -o ai --with-jsonl
|
||||
python clean_ai.py -i xxx.jsonl --tier platform,strong --exclude-forbidden -o ai
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
from douyin.aiclass import TIERS, classify
|
||||
|
||||
|
||||
def iter_jsonl(path: Path):
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
||||
for n, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
yield n, json.loads(line)
|
||||
except ValueError:
|
||||
print(f"[跳过] 第 {n} 行不是合法 JSON", file=sys.stderr)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser(description="筛出 AI 相关作品")
|
||||
ap.add_argument("-i", "--input", required=True)
|
||||
ap.add_argument("-o", "--out", default="", help="输出前缀，生成 <前缀>.jsonl 和 <前缀>.txt")
|
||||
ap.add_argument("--tier", default="platform,strong",
|
||||
help=f"保留哪些档，逗号分隔。可选 {'/'.join(TIERS)}")
|
||||
ap.add_argument("--exclude-forbidden", action="store_true",
|
||||
help="剔掉 allow_download=false 的（抖音对 AIGC 常给 aigc_forbidden）")
|
||||
ap.add_argument("--with-jsonl", action="store_true",
|
||||
help="连带写出 <前缀>.jsonl（体积接近源文件，默认不写）")
|
||||
ap.add_argument("--platform-needs-text", action="store_true",
|
||||
help="platform 档要求文案/标签也有 AI 信号才算。抖音给 AI 配音、"
|
||||
"AI 封面的普通带货视频也打 AIGC 标，露营那批 87110 条里"
|
||||
"有 1288 条是这种，占 platform 档的一半")
|
||||
ap.add_argument("--stats", action="store_true", help="只统计不落盘")
|
||||
ap.add_argument("--samples", type=int, default=0, help="每档打印几条样例到 stderr")
|
||||
args = ap.parse_args()
|
||||
|
||||
keep = {t.strip() for t in args.tier.split(",") if t.strip()}
|
||||
bad = keep - set(TIERS)
|
||||
if bad:
|
||||
ap.error(f"未知档位 {sorted(bad)}，可选 {TIERS}")
|
||||
|
||||
src = Path(args.input)
|
||||
if not src.exists():
|
||||
ap.error(f"找不到 {src}")
|
||||
|
||||
tiers = Counter()
|
||||
reasons = Counter()
|
||||
sigs = Counter()
|
||||
dl = Counter()
|
||||
dur_ms = 0
|
||||
kept = 0
|
||||
seen = set()
|
||||
dup = 0
|
||||
samples = {t: [] for t in TIERS}
|
||||
|
||||
out_j = out_t = None
|
||||
if not args.stats:
|
||||
if not args.out:
|
||||
ap.error("要落盘就给 -o 前缀，或者加 --stats 只看统计")
|
||||
out_t = open(f"{args.out}.txt", "w", encoding="utf-8")
|
||||
out_j = open(f"{args.out}.jsonl", "w", encoding="utf-8") if args.with_jsonl else None
|
||||
|
||||
total = 0
|
||||
for _, r in iter_jsonl(src):
|
||||
total += 1
|
||||
aid = str(r.get("aweme_id") or "")
|
||||
if aid in seen:
|
||||
dup += 1
|
||||
continue
|
||||
seen.add(aid)
|
||||
|
||||
v = classify(r)
|
||||
t = v["tier"]
|
||||
if t == "platform" and args.platform_needs_text:
|
||||
# 平台标了 AIGC，但作品自己一个 AI 信号都没有 -> 多半是 AI 配音/封面。
|
||||
# 作者昵称带 AI（@可灵AI、@松饼大人Ai剧）也算佐证 —— 是本人在做 AI，
|
||||
# 不像 music 那条路会被 `@X创作的原声` 和 studio/Fluxxwave 污染。
|
||||
if not any(s.startswith(("desc:", "author:")) for s in v["signals"]):
|
||||
t = "none"
|
||||
reasons["platform 但无文案/作者佐证，被 --platform-needs-text 剔除"] += 1
|
||||
tiers[t] += 1
|
||||
if args.samples and len(samples[t]) < args.samples:
|
||||
samples[t].append((aid, (r.get("desc") or "").replace("\n", " ")[:90],
|
||||
"|".join(v["signals"][:5])))
|
||||
if t == "none":
|
||||
continue
|
||||
for s in v["signals"]:
|
||||
sigs[s.split(":", 1)[0] + ":" + s.split(":")[1] if ":" in s else s] += 1
|
||||
|
||||
if t not in keep:
|
||||
continue
|
||||
|
||||
ai = r.get("ai") or {}
|
||||
if not ai.get("allow_download", True):
|
||||
dl[ai.get("download_fail_reason") or "unknown"] += 1
|
||||
if args.exclude_forbidden:
|
||||
reasons["被 allow_download=false 剔除"] += 1
|
||||
continue
|
||||
|
||||
kept += 1
|
||||
dur_ms += int(r.get("duration_ms") or 0)
|
||||
if out_t:
|
||||
out_t.write((r.get("url") or f"https://www.douyin.com/video/{aid}") + "\n")
|
||||
if out_j:
|
||||
rec = {"aweme_id": aid, "ai_tier": t, "ai_signals": v["signals"]}
|
||||
rec.update({k: val for k, val in r.items() if k != "aweme_id"})
|
||||
out_j.write(json.dumps(rec, ensure_ascii=False, separators=(",", ":")) + "\n")
|
||||
|
||||
if out_t:
|
||||
out_t.close()
|
||||
if out_j:
|
||||
out_j.close()
|
||||
|
||||
w = sys.stderr
|
||||
print(f"读入 {total} 条，去重后 {len(seen)} 条（重复 {dup}）", file=w)
|
||||
print("\n档位分布:", file=w)
|
||||
for t in TIERS:
|
||||
mark = " <- 保留" if t in keep else ""
|
||||
print(f" {t:9s} {tiers[t]:7d}{mark}", file=w)
|
||||
ai_any = sum(tiers[t] for t in TIERS if t != "none")
|
||||
print(f" {'AI 相关合计':9s} {ai_any:7d} ({ai_any / max(total,1):.1%})", file=w)
|
||||
|
||||
if dl:
|
||||
print("\n保留项里 allow_download=false 的原因:", file=w)
|
||||
for k, v2 in dl.most_common():
|
||||
print(f" {k:24s} {v2:7d}", file=w)
|
||||
for k, v2 in reasons.items():
|
||||
print(f" {k}: {v2}", file=w)
|
||||
|
||||
print(f"\n最终保留 {kept} 条，总时长 {dur_ms/3600000:.2f} 小时", file=w)
|
||||
if out_t:
|
||||
print(f"写出 {args.out}.txt" + (f" 和 {args.out}.jsonl" if out_j else ""), file=w)
|
||||
|
||||
if args.samples:
|
||||
for t in TIERS:
|
||||
if not samples[t]:
|
||||
continue
|
||||
print(f"\n----- {t} 样例 -----", file=w)
|
||||
for aid, d, s in samples[t]:
|
||||
print(f" {aid} {d}\n 信号: {s}", file=w)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
228
scripts/export_sample_xlsx.py
Normal file
228
scripts/export_sample_xlsx.py
Normal file
@@ -0,0 +1,228 @@
|
||||
#!/usr/bin/env python
|
||||
"""把 records.jsonl 导成交付用的样本 xlsx。
|
||||
|
||||
列是甲方给的字段表（序号跟着他们的编号走，中间 4/5/9 他们没要）：
|
||||
|
||||
1 样本 ID 必填 aweme_id
|
||||
2 视频标题 必填 title，为空时退回 desc 首行
|
||||
3 平台 必填 固定「抖音」
|
||||
6 生成方式 AI 生成 / AI 辅助 / 疑似 AI / 未标注
|
||||
7 生成工具 / 模型 从文案标签里认出来的工具名，逗号分隔
|
||||
8 题材 / 内容类别 按标签归类
|
||||
10 分辨率 / 清晰度 必填 1024x576（540p）
|
||||
11 发布时间 必填 2026-08-06 14:12:47
|
||||
12 是否有 AI 标识/水印 必填 抖音 risk_infos 打的标，原文照抄
|
||||
|
||||
用法:
|
||||
python export_sample_xlsx.py # 全量
|
||||
python export_sample_xlsx.py --ai-only # 只要判定为 AI 的
|
||||
python export_sample_xlsx.py --limit 500 -o 样本.xlsx
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from douyin.aiclass import _fields, _scan, classify
|
||||
|
||||
# risk_infos.type 里属于 AIGC 的那几个码，别的都是跟 AI 无关的免责声明
|
||||
# （45=虚构演绎、336/351=个人观点、54=来源于网络…）。见 README §4.5。
|
||||
_AIGC_TYPES = {44, 47, 76, 328}
|
||||
|
||||
# 生成方式：抖音标签的措辞本身就分了轻重，直接用它，别自己发明
|
||||
_HOW_BY_TYPE = {
|
||||
44: "AI 生成（作者声明）",
|
||||
76: "AI 生成（平台判定）",
|
||||
328: "AI 生成（平台判定）",
|
||||
47: "疑似 AI 生成（平台检测）",
|
||||
}
|
||||
|
||||
# 题材归类。**顺序即优先级**，第一个命中的赢 —— 一条片子既是「漫剧」又带
|
||||
# 「剧情」很常见，先匹配到的那个更具体。
|
||||
#
|
||||
# 词要挑「只在这个题材里出现」的。踩过的坑：「古风/国风/画质」放进
|
||||
# 写真/摄影，结果把一整个仙侠短片系列（169/300 条）全吸过去了 ——
|
||||
# 那些是叙事片，只是标签带古风。宽词一律不要，宁可落进「其他」。
|
||||
_GENRES = [
|
||||
("漫剧 / 动画", "漫剧|动漫|动画|二次元|国漫|番剧|漫画"),
|
||||
("短剧 / 剧情", "短剧|剧情|微短剧|电影|影视|短片|反转|结局|"
|
||||
"第[一二三四五六七八九十\\d]+集|合集"),
|
||||
("教程 / 科普", "教程|教学|技巧|入门|保姆级|科普|干货|工作流|提示词"),
|
||||
("资讯 / 新闻", "新闻|资讯|报道|快讯|媒体精选|发布会|财经|股市"),
|
||||
("带货 / 电商", "好物|同款|下单|优惠|源头工厂|旗舰店|带货|种草|清仓"),
|
||||
("户外 / 露营", "露营|帐篷|户外|野营|钓鱼|自驾|徒步|登山"),
|
||||
("音乐 / 舞蹈", "音乐|歌曲|翻唱|舞蹈|跳舞"),
|
||||
("写真 / 摄影", "写真|摄影|人像|壁纸|头像"),
|
||||
("游戏", "手游|端游|原神|王者荣耀|电竞|无畏契约|崩坏"),
|
||||
("生活 / 日常", "日常|vlog|美食|旅行|萌宠|搞笑"),
|
||||
]
|
||||
_GENRE_RE = [(name, re.compile(pat, re.I)) for name, pat in _GENRES]
|
||||
|
||||
|
||||
def _risk(rec: dict) -> tuple:
|
||||
"""(type, content)。抖音把 AI 标识放在 raw.risk_infos 里。"""
|
||||
ri = (rec.get("raw") or {}).get("risk_infos") or {}
|
||||
if not isinstance(ri, dict):
|
||||
return None, ""
|
||||
return ri.get("type"), (ri.get("content") or "").strip()
|
||||
|
||||
|
||||
def how_generated(rec: dict, verdict: dict) -> str:
|
||||
t, _ = _risk(rec)
|
||||
if t in _HOW_BY_TYPE:
|
||||
return _HOW_BY_TYPE[t]
|
||||
if verdict["tier"] == "strong":
|
||||
return "AI 生成（作品自述）"
|
||||
if verdict["tier"] in ("topic", "author"):
|
||||
return "疑似 AI 相关"
|
||||
return "未标注"
|
||||
|
||||
|
||||
def tools_used(verdict: dict) -> str:
|
||||
"""从判定信号里把工具名摘出来。信号形如 `desc:工具:即梦`。"""
|
||||
names = []
|
||||
for s in verdict["signals"]:
|
||||
parts = s.split(":")
|
||||
if len(parts) >= 3 and parts[1] in ("工具", "漫剧"):
|
||||
n = parts[2]
|
||||
if n not in names:
|
||||
names.append(n)
|
||||
return "、".join(names)
|
||||
|
||||
|
||||
def genre(rec: dict) -> str:
|
||||
text = " ".join([rec.get("title") or "", rec.get("desc") or ""]
|
||||
+ list(rec.get("tags") or []))
|
||||
for name, rx in _GENRE_RE:
|
||||
if rx.search(text):
|
||||
return name
|
||||
return "其他"
|
||||
|
||||
|
||||
def resolution(rec: dict) -> str:
|
||||
v = rec.get("video") or {}
|
||||
w, h = v.get("width"), v.get("height")
|
||||
ratio = ((rec.get("raw") or {}).get("video") or {}).get("ratio") or ""
|
||||
if w and h:
|
||||
return f"{w}x{h}" + (f"（{ratio}）" if ratio else "")
|
||||
return ratio or ""
|
||||
|
||||
|
||||
def ai_mark(rec: dict) -> str:
|
||||
"""是否有 AI 标识/水印 —— 抖音的原文照抄，没有就写「无」。
|
||||
只认 _AIGC_TYPES；45/336/351 那些是虚构演绎、个人观点，跟 AI 无关。"""
|
||||
t, content = _risk(rec)
|
||||
if t in _AIGC_TYPES and content:
|
||||
return f"有：{content}"
|
||||
return "无"
|
||||
|
||||
|
||||
def title_of(rec: dict) -> str:
|
||||
"""标题 + 标签拼一格。
|
||||
|
||||
records.jsonl 里的 title 是 split_desc 把标签摘掉之后的正文，标签单独放
|
||||
在 tags 里。甲方只要一列，所以拼回去 —— 顺序按原文案的习惯：正文在前，
|
||||
`#标签` 跟在后面。
|
||||
"""
|
||||
t = (rec.get("title") or "").strip()
|
||||
if not t:
|
||||
d = (rec.get("desc") or "").strip().splitlines()
|
||||
t = d[0][:120] if d else ""
|
||||
tags = " ".join(f"#{x}" for x in (rec.get("tags") or []) if x)
|
||||
return (t + " " + tags).strip() if tags else t
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description="records.jsonl -> 交付样本 xlsx")
|
||||
ap.add_argument("-i", "--input", default="records/records.jsonl")
|
||||
ap.add_argument("-o", "--out", default="exports/AI视频样本.xlsx")
|
||||
ap.add_argument("--ai-only", action="store_true",
|
||||
help="只导出判定为 AI 的（platform/strong 档）")
|
||||
ap.add_argument("--limit", type=int, default=0)
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
try:
|
||||
from openpyxl import Workbook
|
||||
except ImportError:
|
||||
print("需要 openpyxl：pip install openpyxl", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
src = Path(args.input)
|
||||
if not src.exists():
|
||||
ap.error(f"找不到 {src}")
|
||||
|
||||
# write_only：11 万行常规模式会把内存吃光
|
||||
wb = Workbook(write_only=True)
|
||||
ws = wb.create_sheet("样本")
|
||||
# 前 9 列是甲方字段表要的，后面几列是核对/取数用的：题材判得对不对，
|
||||
# 光看标题看不出来，得有标签和链接才能点开验；COS 路径是拿原始文件的地址。
|
||||
# 作者插在「平台」后面 —— 甲方编号 1/2/3 的三列保持原位原序，
|
||||
# 作者正好落进他们跳过的 4/5 那两个空号里。
|
||||
ws.append(["样本 ID", "视频标题", "平台", "作者",
|
||||
"生成方式", "生成工具 / 模型",
|
||||
"题材 / 内容类别", "分辨率 / 清晰度", "发布时间",
|
||||
"是否有 AI 标识 / 水印",
|
||||
"时长(秒)", "视频链接",
|
||||
"COS 视频路径", "COS 音频路径"])
|
||||
|
||||
n = written = bad = 0
|
||||
with open(src, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
n += 1
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except ValueError:
|
||||
bad += 1
|
||||
continue
|
||||
v = classify(rec)
|
||||
if args.ai_only and v["tier"] not in ("platform", "strong"):
|
||||
continue
|
||||
# 每格都过一遍 str()：openpyxl 会把 None 原样写成 "None"
|
||||
ws.append([
|
||||
str(rec.get("aweme_id") or ""),
|
||||
title_of(rec),
|
||||
"抖音",
|
||||
(rec.get("author") or {}).get("nickname") or "",
|
||||
# 生成方式 / 生成工具：留空，人工填。
|
||||
# 自动推的那版（how_generated / tools_used）只能看抖音的标和
|
||||
# 文案里写没写工具名，作者不写就抓不到 —— 实测 44.8% 的工具
|
||||
# 列是空的，而"生成方式"里 81.8% 全都压在"作者声明"一个值上，
|
||||
# 区分度太低，不如空着让人填。两个函数留着，要用加回来即可。
|
||||
"",
|
||||
"",
|
||||
genre(rec),
|
||||
resolution(rec),
|
||||
rec.get("create_time_str") or "",
|
||||
ai_mark(rec),
|
||||
round(float(rec.get("duration_sec") or 0), 1),
|
||||
rec.get("source_url")
|
||||
or f"https://www.douyin.com/video/{rec.get('aweme_id')}",
|
||||
rec.get("cos_mp4") or "",
|
||||
rec.get("cos_mp3") or "",
|
||||
])
|
||||
written += 1
|
||||
if args.limit and written >= args.limit:
|
||||
break
|
||||
if written % 20000 == 0:
|
||||
print(f" 已写 {written} 行…", flush=True)
|
||||
|
||||
wb.save(args.out)
|
||||
print(f"读入 {n} 行（坏行 {bad}），写出 {written} 行 -> {args.out}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
222
scripts/jsonl_to_xlsx.py
Normal file
222
scripts/jsonl_to_xlsx.py
Normal file
@@ -0,0 +1,222 @@
|
||||
#!/usr/bin/env python3
|
||||
"""把 jsonl 清洗并导出为 xlsx。
|
||||
|
||||
同时认两种输入：
|
||||
|
||||
records.jsonl 流水线产物，字段全（标题/作者/标签/时长/COS 地址/raw）
|
||||
manifest.jsonl 老格式，只有 cos_mp4 / cos_mp3 / duration
|
||||
|
||||
records.jsonl 内嵌了 raw，单行可能几十 KB、整个文件几百 MB，所以这里
|
||||
一律逐行流式读，任何时候都不把整个文件读进内存。选中的列也只留需要的
|
||||
那几个字段，raw 不会进内存驻留。
|
||||
|
||||
清洗动作（每一项都会报告剔除了多少）：
|
||||
1. 丢弃无法解析的行（被 Ctrl-C 打断可能留下半行）
|
||||
2. 丢弃缺字段 / 字段类型不对的行
|
||||
3. 丢弃时长非正数的行
|
||||
4. 丢弃 URL 前缀不符合预期的行
|
||||
5. 按 mp4 地址去重，保留首次出现
|
||||
|
||||
用法:
|
||||
python jsonl_to_xlsx.py
|
||||
python jsonl_to_xlsx.py -i records.jsonl -o "抖音-四川话.xlsx"
|
||||
python jsonl_to_xlsx.py -i records.jsonl --columns cos_mp4,cos_mp3,duration_sec
|
||||
python jsonl_to_xlsx.py --list-columns
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, Iterator, List
|
||||
|
||||
EXPECTED_PREFIX = "cos://asr-dig-data-1253181912/0005/zh/"
|
||||
|
||||
# 列名 -> (表头, 取值函数)。取值函数只碰需要的字段，raw 不会被带进来。
|
||||
COLUMNS: Dict[str, tuple] = {
|
||||
"aweme_id": ("视频ID", lambda r: str(r.get("aweme_id") or "")),
|
||||
"title": ("标题", lambda r: r.get("title") or ""),
|
||||
"desc": ("原始描述", lambda r: r.get("desc") or ""),
|
||||
"tags": ("标签", lambda r: " ".join(r.get("tags") or [])),
|
||||
"author": ("作者", lambda r: (r.get("author") or {}).get("nickname", "")),
|
||||
"author_uid": ("作者UID", lambda r: (r.get("author") or {}).get("uid", "")),
|
||||
"create_time": ("发布时间", lambda r: r.get("create_time_str") or ""),
|
||||
"duration_sec": ("时长(秒)", _dur := (lambda r: r.get("duration_sec",
|
||||
r.get("duration")))),
|
||||
"duration": ("时长(秒)", _dur),
|
||||
"cos_mp4": ("cosmp4地址", lambda r: r.get("cos_mp4") or ""),
|
||||
"cos_mp3": ("cosmp3地址", lambda r: r.get("cos_mp3") or ""),
|
||||
"size_mp4": ("mp4字节数", lambda r: r.get("size_mp4") or 0),
|
||||
"size_mp3": ("mp3字节数", lambda r: r.get("size_mp3") or 0),
|
||||
"play": ("播放数", lambda r: (r.get("stats") or {}).get("play", 0)),
|
||||
"digg": ("点赞数", lambda r: (r.get("stats") or {}).get("digg", 0)),
|
||||
"comment": ("评论数", lambda r: (r.get("stats") or {}).get("comment", 0)),
|
||||
"source_url": ("原始链接", lambda r: r.get("source_url") or ""),
|
||||
}
|
||||
|
||||
DEFAULT_COLUMNS = ["cos_mp4", "cos_mp3", "duration_sec"]
|
||||
FULL_COLUMNS = ["aweme_id", "title", "author", "create_time", "tags",
|
||||
"duration_sec", "cos_mp4", "cos_mp3"]
|
||||
|
||||
NUMERIC = {"duration_sec", "duration", "size_mp4", "size_mp3",
|
||||
"play", "digg", "comment"}
|
||||
|
||||
|
||||
def iter_lines(path: Path) -> Iterator[str]:
|
||||
with open(path, "r", encoding="utf-8", errors="replace") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if line:
|
||||
yield line
|
||||
|
||||
|
||||
def clean(path: Path, prefix: str, columns: List[str]):
|
||||
"""流式清洗，只保留选中的列。"""
|
||||
rows: List[List[Any]] = []
|
||||
seen: set = set()
|
||||
dropped = {"解析失败": 0, "字段缺失": 0, "时长非法": 0, "地址异常": 0, "重复": 0}
|
||||
total = 0
|
||||
dur_idx = next((i for i, c in enumerate(columns)
|
||||
if c in ("duration_sec", "duration")), None)
|
||||
|
||||
for line in iter_lines(path):
|
||||
total += 1
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except ValueError:
|
||||
dropped["解析失败"] += 1
|
||||
continue
|
||||
|
||||
mp4, mp3 = r.get("cos_mp4"), r.get("cos_mp3")
|
||||
dur = r.get("duration_sec", r.get("duration"))
|
||||
if not isinstance(mp4, str) or not isinstance(mp3, str) or dur is None:
|
||||
dropped["字段缺失"] += 1
|
||||
continue
|
||||
if not isinstance(dur, (int, float)) or isinstance(dur, bool) or dur <= 0:
|
||||
dropped["时长非法"] += 1
|
||||
continue
|
||||
if not mp4.startswith(prefix) or not mp3.startswith(prefix):
|
||||
dropped["地址异常"] += 1
|
||||
continue
|
||||
if mp4 in seen:
|
||||
dropped["重复"] += 1
|
||||
continue
|
||||
|
||||
seen.add(mp4)
|
||||
row = [COLUMNS[c][1](r) for c in columns]
|
||||
if dur_idx is not None:
|
||||
row[dur_idx] = float(row[dur_idx])
|
||||
rows.append(row)
|
||||
|
||||
return rows, dropped, total, dur_idx
|
||||
|
||||
|
||||
def write_xlsx(rows, headers, out: Path, sheet: str, dur_idx) -> None:
|
||||
from openpyxl import Workbook
|
||||
from openpyxl.styles import Alignment, Font, PatternFill
|
||||
from openpyxl.utils import get_column_letter
|
||||
|
||||
wb = Workbook()
|
||||
ws = wb.active
|
||||
ws.title = sheet
|
||||
|
||||
ws.append(headers)
|
||||
head_font = Font(bold=True, color="FFFFFF")
|
||||
head_fill = PatternFill("solid", fgColor="4472C4")
|
||||
for c in range(1, len(headers) + 1):
|
||||
cell = ws.cell(row=1, column=c)
|
||||
cell.font = head_font
|
||||
cell.fill = head_fill
|
||||
cell.alignment = Alignment(horizontal="center", vertical="center")
|
||||
|
||||
for r in rows:
|
||||
ws.append(r)
|
||||
|
||||
# 时长保留 3 位小数，与 ffprobe 输出精度一致
|
||||
if dur_idx is not None:
|
||||
for row in ws.iter_rows(min_row=2, min_col=dur_idx + 1, max_col=dur_idx + 1):
|
||||
for cell in row:
|
||||
cell.number_format = "0.000"
|
||||
|
||||
for i, head in enumerate(headers):
|
||||
widest = max((len(str(r[i])) for r in rows), default=10)
|
||||
width = min(max(len(head) + 4, widest + 2), 60)
|
||||
ws.column_dimensions[get_column_letter(i + 1)].width = width
|
||||
|
||||
ws.freeze_panes = "A2" # 表头固定
|
||||
ws.auto_filter.ref = ws.dimensions # 开筛选
|
||||
wb.save(out)
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="jsonl -> xlsx")
|
||||
ap.add_argument("-i", "--input", default="records/records.jsonl")
|
||||
ap.add_argument("-o", "--output", default="exports/抖音-四川话.xlsx")
|
||||
ap.add_argument("--sheet", default="抖音-四川话")
|
||||
ap.add_argument("--prefix", default=EXPECTED_PREFIX)
|
||||
ap.add_argument("--columns", default="",
|
||||
help=f"逗号分隔的列，默认 {','.join(DEFAULT_COLUMNS)}；"
|
||||
"用 full 展开为常用全字段")
|
||||
ap.add_argument("--list-columns", action="store_true", help="列出可用列名后退出")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
if args.list_columns:
|
||||
print("可用列：")
|
||||
for k, (head, _) in COLUMNS.items():
|
||||
print(f" {k:<14} {head}")
|
||||
return 0
|
||||
|
||||
if not args.columns:
|
||||
columns = list(DEFAULT_COLUMNS)
|
||||
elif args.columns.strip() == "full":
|
||||
columns = list(FULL_COLUMNS)
|
||||
else:
|
||||
columns = [c.strip() for c in args.columns.split(",") if c.strip()]
|
||||
bad = [c for c in columns if c not in COLUMNS]
|
||||
if bad:
|
||||
print(f"[错误] 未知列名 {bad}，用 --list-columns 看可选项", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
src = Path(args.input)
|
||||
if not src.exists():
|
||||
print(f"[错误] 找不到 {src}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
rows, dropped, total, dur_idx = clean(src, args.prefix, columns)
|
||||
|
||||
print(f"读入 {total} 行")
|
||||
if sum(dropped.values()):
|
||||
for k, v in dropped.items():
|
||||
if v:
|
||||
print(f" 剔除 {k}: {v}")
|
||||
else:
|
||||
print(" 清洗检查全部通过，无需剔除")
|
||||
print(f"保留 {len(rows)} 行")
|
||||
|
||||
if not rows:
|
||||
print("[错误] 清洗后没有数据", file=sys.stderr)
|
||||
return 3
|
||||
|
||||
headers = [COLUMNS[c][0] for c in columns]
|
||||
out = Path(args.output)
|
||||
write_xlsx(rows, headers, out, args.sheet, dur_idx)
|
||||
|
||||
print(f"\n已写入 {out}")
|
||||
print(f" {len(rows)} 行 × {len(headers)} 列：{' | '.join(headers)}")
|
||||
if dur_idx is not None:
|
||||
ds = [r[dur_idx] for r in rows]
|
||||
print(f" 总时长 {sum(ds) / 3600:.2f} 小时，平均 {sum(ds) / len(ds):.1f} 秒，"
|
||||
f"最短 {min(ds):.3f}，最长 {max(ds):.3f}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
276
tests/test_aiclass.py
Normal file
276
tests/test_aiclass.py
Normal file
@@ -0,0 +1,276 @@
|
||||
"""douyin.aiclass 的离线测试。用例全部取自 40087 条真实数据里的真实文案。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
from douyin.aiclass import classify
|
||||
|
||||
|
||||
def rec(desc="", tags=(), author="", music="", is_aigc=False, label=""):
|
||||
return {
|
||||
"desc": desc,
|
||||
"hashtags": [{"name": t} for t in tags],
|
||||
"author": {"nickname": author},
|
||||
"music": {"title": music},
|
||||
"ai": {"is_aigc": is_aigc, "label": label},
|
||||
}
|
||||
|
||||
|
||||
def tier(**kw):
|
||||
return classify(rec(**kw))["tier"]
|
||||
|
||||
|
||||
# ---------- platform：唯一能抓到"作者没标但确实是 AI"的信号 ----------
|
||||
|
||||
def test_platform_label_wins_even_with_no_ai_text():
|
||||
"""实测 1744 条：平台标了 AIGC，文案标签里一个 ai 字都没有。"""
|
||||
assert tier(desc="#趣味脑筋急转弯 #创作者中心 #创作灵感",
|
||||
tags=["趣味脑筋急转弯", "创作者中心", "创作灵感"],
|
||||
author="申哥爱剪辑", is_aigc=True,
|
||||
label="作者声明：内容由 AI 生成") == "platform"
|
||||
assert tier(desc="澳洲的野狗 #中华田园犬 #历史", tags=["中华田园犬", "历史"],
|
||||
author="胖嘟嘟历史", is_aigc=True) == "platform"
|
||||
|
||||
|
||||
# ---------- strong：作品自己声明是 AI 做的 ----------
|
||||
|
||||
def test_strong_ai_compound():
|
||||
assert tier(desc="AI花絮：我很快就会驯服AI演员啦 #AI短剧#AI电影",
|
||||
tags=["AI短剧", "AI电影"]) == "strong"
|
||||
|
||||
|
||||
def test_strong_tool_name_without_ai_letters():
|
||||
"""即梦 / seedance / libtv 这些名字里没有 ai，只能靠名单。"""
|
||||
assert tier(desc="干啥呀即梦，这是掀桌子不玩了吗？", tags=["即梦"]) == "strong"
|
||||
assert tier(desc="", tags=["seedance2.0"]) == "strong"
|
||||
assert tier(desc="本视频由libtv制作生成", tags=["Libtv"]) == "strong"
|
||||
assert tier(desc="", tags=["runninghub"]) == "strong"
|
||||
|
||||
|
||||
def test_strong_aigc_token():
|
||||
assert tier(desc="自制AIGC《银骸》第0集！") == "strong"
|
||||
|
||||
|
||||
def test_strong_handles_capital_A_lowercase_L():
|
||||
"""`抖音al` 在这批数据里出现 129 次，`#Al创作` 也有。不归一化就整片漏掉。"""
|
||||
assert tier(desc="他种下几千棵橡树 #Al创作 #爱情", tags=["Al创作", "爱情"]) == "strong"
|
||||
assert tier(desc="", tags=["抖音al"]) == "topic" # 没有创作语境词，降到 topic
|
||||
|
||||
|
||||
# ---------- topic：在讲 AI，不一定是 AI 做的 ----------
|
||||
|
||||
def test_topic_pure_ai_mention():
|
||||
assert tier(desc="布布讲ai之如何写提示词 #ai", tags=["ai"]) == "strong" # 提示词是创作语境
|
||||
assert tier(desc="今天聊聊 AI 会不会取代人类 #ai", tags=["ai"]) == "topic"
|
||||
|
||||
|
||||
def test_topic_chat_models():
|
||||
assert tier(desc="deepseek 又更新了") == "topic"
|
||||
assert tier(desc="用 ChatGPT 写周报") == "topic"
|
||||
assert tier(desc="#人工智能 的一点思考", tags=["人工智能"]) == "topic"
|
||||
|
||||
|
||||
# ---------- author：最弱，作品本身没信号 ----------
|
||||
|
||||
def test_author_only():
|
||||
r = classify(rec(desc="研究生偷吃外卖被毒S #湘潭大学 #普法教育",
|
||||
tags=["湘潭大学", "普法教育"], author="牛牛Ai"))
|
||||
assert r["tier"] == "author", r
|
||||
|
||||
|
||||
def test_author_never_upgrades_a_scored_item():
|
||||
"""作者昵称只能给 none 兜底，不能把 topic 顶成 strong。"""
|
||||
r = classify(rec(desc="聊聊 AI 的伦理问题", author="星幕AIGC"))
|
||||
assert r["tier"] == "topic", r
|
||||
|
||||
|
||||
def test_default_original_sound_is_not_a_work_signal():
|
||||
"""音乐标题默认是 `@<昵称>创作的原声`，40087 条里有 32555 条是这个壳。
|
||||
昵称带 AI 会连"创作"一起命中，曾让 771 条从 music 这条路混进 strong
|
||||
（@冠宇Aigc 的猴子炒菜、@牛牛Ai 的大案纪实）。只能落到 author 档。"""
|
||||
r = classify(rec(desc="#猴子炒菜#猴子成精#原创视频",
|
||||
tags=["猴子炒菜", "猴子成精"],
|
||||
author="冠宇Aigc(AI剧场)",
|
||||
music="@冠宇Aigc(AI剧场)创作的原声"))
|
||||
assert r["tier"] == "author", r
|
||||
# 作者昵称也不带 AI 时，默认原声不能凭空造出信号
|
||||
assert tier(desc="今天天气不错", author="老王",
|
||||
music="@老王创作的原声") == "none"
|
||||
|
||||
|
||||
def test_real_track_name_still_counts_but_only_weakly():
|
||||
assert tier(desc="随便发发", music="即梦AI生成的配乐") == "author"
|
||||
|
||||
|
||||
# ---------- 误报：这些绝不能收 ----------
|
||||
|
||||
def test_english_words_containing_ai_are_not_signals():
|
||||
for d in ["email me for detail", "again and again", "hair salon",
|
||||
"wait a minute", "chairman", "trained model"]:
|
||||
assert tier(desc=d) == "none", d
|
||||
|
||||
|
||||
def test_editing_jargon_is_not_ai():
|
||||
"""剪映 375 / 关键帧 314 / 蒙版 203 —— 剪辑术语，不是 AI。"""
|
||||
assert tier(desc="导入一张照片即可制作 #剪映", tags=["剪映"]) == "none"
|
||||
assert tier(desc="#关键帧 #蒙版 #转场 #剪辑教程",
|
||||
tags=["关键帧", "蒙版", "转场", "剪辑教程"]) == "none"
|
||||
|
||||
|
||||
def test_generic_chinese_words_alone_are_not_ai():
|
||||
"""动画/绘画/分身/动漫 单独出现不算 —— 否则 830 条 #原创动画 全被误收。"""
|
||||
assert tier(desc="第33 #原创动画", tags=["原创动画"]) == "none"
|
||||
assert tier(desc="跟小同桌回老家 #二次元 #原创动画 #恋爱番",
|
||||
tags=["二次元", "原创动画", "恋爱番"]) == "none"
|
||||
assert tier(desc="他的分身术真牛") == "none"
|
||||
|
||||
|
||||
def test_ordinary_content_is_none():
|
||||
assert tier(desc="户外露营过夜用的这款帐篷也太实用啦！ #露营装备",
|
||||
tags=["露营装备"], author="蚁巢户外直营店") == "none"
|
||||
assert tier(desc="4月16日，重庆，张雪陪同合作伙伴参观工厂 #张雪机车",
|
||||
tags=["张雪机车"], author="第一现场") == "none"
|
||||
|
||||
|
||||
def test_signals_are_recorded_for_audit():
|
||||
r = classify(rec(desc="#即梦AI #AI短剧", tags=["即梦ai", "ai短剧"]))
|
||||
assert r["tier"] == "strong"
|
||||
assert any("工具:即梦" in s for s in r["signals"]), r["signals"]
|
||||
|
||||
|
||||
def test_empty_record():
|
||||
assert classify(rec())["tier"] == "none"
|
||||
|
||||
|
||||
# ---------- 常用词工具名：必须跟 ai 同现，见 aiclass 模块开头第 6 条 ----------
|
||||
|
||||
def test_ambiguous_tool_alone_is_not_ai():
|
||||
"""87110 条里实测的误判，全部取自真实文案。"""
|
||||
# 三峰的"海螺"帐篷，撞上 MiniMax 的海螺 AI
|
||||
assert tier(desc="三峰出-海螺。重量超级轻的一款帐篷，有tpu，有烟囱口。",
|
||||
tags=["户外装备", "帐篷", "露营"]) == "none"
|
||||
# 元宝勺，撞上腾讯元宝
|
||||
assert tier(desc="边缘光滑才是好勺子！316不锈钢元宝勺。",
|
||||
tags=["辅食勺", "勺子", "元宝勺"]) == "none"
|
||||
# 豆包是只狗 / 一种包子，不是那个大模型
|
||||
assert tier(desc="豆包监督学生自习这一块 豆包：交给我你就闹心吧！",
|
||||
tags=["豆包", "搞笑", "晚自习"]) == "none"
|
||||
# 《龙神万相》，撞上阿里万相
|
||||
assert tier(desc="龙神万相之龙魂定档 迫不及待想沉浸式感受龙族的热血与传奇了！",
|
||||
tags=["龙神万相之龙魂", "西行纪"]) == "none"
|
||||
# "明星流量"里含"星流"；"起舞弄清影"里含"清影"
|
||||
assert tier(desc="商家别乱蹭明星流量！4 名被告擅自使用张雨绮肖像及姓名商用",
|
||||
tags=["张雨绮", "侵权"]) == "none"
|
||||
assert tier(desc="柳下闻瑶琴起舞弄清影 ～好久没跳了") == "none"
|
||||
assert tier(desc="做白日梦从拥有弹丝云床开始",
|
||||
tags=["户外露营", "自动充气床"]) == "none"
|
||||
|
||||
|
||||
def test_ambiguous_tool_counts_when_ai_present():
|
||||
"""真命中的无一例外都同时带 AI 字样，所以这道门槛不掉召回。"""
|
||||
r = classify(rec(desc="海螺AI生成的短片 #ai视频", tags=["ai视频"]))
|
||||
assert r["tier"] == "strong"
|
||||
assert any("工具:海螺" in s for s in r["signals"]), r["signals"]
|
||||
assert tier(desc="用豆包写的脚本，AI 生成 #ai", tags=["ai"]) == "strong"
|
||||
|
||||
|
||||
def test_digital_rmb_is_not_a_digital_human():
|
||||
"""数字人民币含"数字人"，微信分身含"分身"，穿搭也说"换脸"。
|
||||
实测放开这三个词只进误判，一条真的都捞不到。"""
|
||||
assert tier(desc="一地宣布：5月起，公务员工资用数字人民币发放",
|
||||
tags=["公务员", "工资", "数字人民币"]) == "none"
|
||||
assert tier(desc="抖音双开 小米华为苹果手机 微信怎么多开",
|
||||
tags=["微信分身双开", "抖音多开分身"]) == "none"
|
||||
assert tier(desc="救命！这套穿搭真的能换脸！",
|
||||
tags=["今天穿什么", "冬季穿搭"]) == "none"
|
||||
|
||||
|
||||
def test_sound_wrapper_csv_form_is_stripped():
|
||||
"""csv 导出的原声标题多一截 ` - 昵称`，也得剥掉，
|
||||
否则作者昵称又会从 music 这条路偷偷升档。"""
|
||||
assert tier(desc="今天去钓鱼", music="@牛牛Ai创作的原声 - 牛牛Ai") == "none"
|
||||
assert tier(desc="今天去钓鱼", music="@牛牛Ai创作的原声") == "none"
|
||||
|
||||
|
||||
def test_minimax_is_a_tool():
|
||||
assert tier(desc="minimax 生成的片子", tags=["minimax"]) == "strong"
|
||||
|
||||
|
||||
def test_roman_numeral_impersonates_ai():
|
||||
"""`AⅠ漫剧` 用罗马数字 Ⅰ(U+2160) 冒充字母 I，跟 `抖音al` 同一类花招。"""
|
||||
r = classify(rec(desc="AⅠ漫剧你就这样崛起吧！", tags=["AⅠ漫剧", "漫剧推荐"]))
|
||||
assert r["tier"] == "strong", r
|
||||
|
||||
|
||||
def test_voice_cloning_is_generative():
|
||||
assert tier(desc="克隆音色怎么弄 声音可以克隆吗",
|
||||
tags=["克隆音色", "声音克隆", "剪映教程"]) == "strong"
|
||||
|
||||
|
||||
def test_veo3_beats_a_weak_author_signal():
|
||||
"""原先这批只靠作者昵称算 author 档，认出 veo3 后该升到 strong。"""
|
||||
assert tier(desc="#猴子成精 #猴子炒菜 #veo3", tags=["猴子炒菜", "veo3"],
|
||||
author="冠宇Aigc") == "strong"
|
||||
|
||||
|
||||
def test_rejected_candidates_stay_none():
|
||||
"""这些词看着像 AI，实测全是子串陷阱，加进词表会污染结果。"""
|
||||
assert tier(desc="全国生猪价格已连续16周下跌，行业亏损幅度加大") == "none" # 度加
|
||||
assert tier(desc="女生相亲要求要30万彩礼，听到要求男士都拒绝") == "none" # 万彩
|
||||
assert tier(desc="中国男篮硬刚美国梦八队，诞生数副名场面！") == "none" # 生数
|
||||
assert tier(desc="决定全年经济走势关键节点上，国务院召开重要会议") == "none" # 节点
|
||||
assert tier(desc="卡池都快结束了，肝了个大保底怎么还不出货",
|
||||
tags=["抽卡日常", "崩坏星穹铁道"]) == "none" # 抽卡
|
||||
assert tier(desc="圈外人不语，只是一味求捏脸", tags=["燕云十六声捏脸"]) == "none" # 捏脸
|
||||
assert tier(desc="杀神降临！82-36！全力帕到底有多恐怖？",
|
||||
tags=["无畏契约", "valorant"]) == "none" # lora
|
||||
assert tier(desc="剑来!!!", tags=["国产动漫", "国漫超燃", "剑来动画"]) == "none" # 国漫
|
||||
|
||||
|
||||
def test_manju_names_stand_alone():
|
||||
"""漫剧专名单独成立，不要求跟 ai 同现（用户要求：出现关键词就算）。"""
|
||||
r = classify(rec(desc="究竟是什么样的小说仅是漫改，就能瞬间火爆全网破30亿的热度",
|
||||
tags=["万妖图录传", "二次元", "充能计划"]))
|
||||
assert r["tier"] == "strong", r
|
||||
assert any("漫剧:万妖图录传" in x for x in r["signals"]), r["signals"]
|
||||
assert tier(desc="晚夏小梦1-27：128分钟大合集看爽",
|
||||
tags=["二次元", "原创动画", "晚夏小梦"]) == "strong"
|
||||
assert tier(desc="小溪废片合集~", tags=["二次元", "月落小溪"]) == "strong"
|
||||
|
||||
|
||||
def test_hongguo_platform_name_is_not_an_ai_signal():
|
||||
"""红果主体是真人短剧，AI 漫剧只是其中一个赛道，平台名不能当信号。"""
|
||||
assert tier(desc="把生活的爽感拉满！看《家里家外2》要备好纸巾",
|
||||
tags=["红果短剧", "短剧推荐", "家里家外2", "治愈系短剧"]) == "none"
|
||||
assert tier(desc="《冷御失守》后续", tags=["红果短剧"]) == "none"
|
||||
|
||||
|
||||
def test_manju_topic_words_are_not_ai_markers():
|
||||
"""漫剧圈里 100% 是 AI 的题材词，放全库就炸 —— 它们是题材不是 AI 标记。"""
|
||||
assert tier(desc="玄奘故里，也就是西游记里唐僧的出生地，位于偃师缑氏镇") == "none"
|
||||
assert tier(desc="宫崎骏的夏天大概就是这个样子吧！", tags=["宫崎骏的夏天", "龙猫"]) == "none"
|
||||
assert tier(desc="《狼来了》第一集：还记得小时候听的《狼来了》吗？",
|
||||
tags=["狼来了", "民间故事", "原创"]) == "none"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
|
||||
failed = 0
|
||||
for t in tests:
|
||||
try:
|
||||
t()
|
||||
print(f" ok {t.__name__}")
|
||||
except AssertionError as exc:
|
||||
failed += 1
|
||||
print(f" FAIL {t.__name__}: {exc}")
|
||||
print(f"\n{len(tests) - failed}/{len(tests)} 通过")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
217
tests/test_meta.py
Normal file
217
tests/test_meta.py
Normal file
@@ -0,0 +1,217 @@
|
||||
"""douyin.meta 的离线测试：标题/标签拆分、记录追加与幂等。不联网。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import json
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from douyin.meta import RecordLog, split_desc
|
||||
|
||||
|
||||
def te(desc, *names):
|
||||
"""按真实位置构造 text_extra —— 偏移量算出来，别手数。"""
|
||||
out = []
|
||||
for n in names:
|
||||
i = desc.index("#" + n)
|
||||
out.append({"type": 1, "hashtag_name": n, "hashtag_id": "1",
|
||||
"start": i, "end": i + len(n) + 1})
|
||||
return out
|
||||
|
||||
|
||||
def te_broken(*names):
|
||||
"""偏移量是垃圾（越界）的 text_extra。"""
|
||||
return [{"type": 1, "hashtag_name": n, "hashtag_id": "1",
|
||||
"start": 999, "end": 1200} for n in names]
|
||||
|
||||
|
||||
def test_split_basic():
|
||||
desc = "是谁教他这样平账的？？？ #沙雕动画 #原创动画"
|
||||
title, tags = split_desc(desc, te(desc, "沙雕动画", "原创动画"))
|
||||
assert title == "是谁教他这样平账的？？？", repr(title)
|
||||
assert tags == ["沙雕动画", "原创动画"], tags
|
||||
|
||||
|
||||
def test_split_adjacent_no_space():
|
||||
# 实测抖音有 "#四川话#方言" 这种连着写的
|
||||
title, tags = split_desc("老板说话#四川话#方言", None)
|
||||
assert title == "老板说话", repr(title)
|
||||
assert tags == ["四川话", "方言"], tags
|
||||
|
||||
|
||||
def test_split_tag_missing_from_text_extra():
|
||||
"""text_extra 会漏标签（实测 12 条里有 2 条），tags 必须以 desc 为准。"""
|
||||
desc = "太搞笑了 #搞笑配音 #专治不开心 #非洲搞笑视频"
|
||||
title, tags = split_desc(desc, te(desc, "搞笑配音", "专治不开心"))
|
||||
assert tags == ["搞笑配音", "专治不开心", "非洲搞笑视频"], tags
|
||||
assert title == "太搞笑了", repr(title)
|
||||
|
||||
|
||||
def test_split_tag_only_in_text_extra():
|
||||
"""反过来，desc 里没写但 text_extra 有的也不能丢。"""
|
||||
_, tags = split_desc("光秃秃的标题", te_broken("隐藏标签"))
|
||||
assert tags == ["隐藏标签"], tags
|
||||
|
||||
|
||||
def test_split_tag_in_middle():
|
||||
title, tags = split_desc("前面 #中间 后面", None)
|
||||
assert title == "前面 后面", repr(title)
|
||||
assert tags == ["中间"], tags
|
||||
|
||||
|
||||
def test_split_no_tags():
|
||||
title, tags = split_desc(" 纯标题没有标签 ", None)
|
||||
assert title == "纯标题没有标签", repr(title)
|
||||
assert tags == [], tags
|
||||
|
||||
|
||||
def test_split_empty():
|
||||
assert split_desc("", None) == ("", [])
|
||||
assert split_desc(None, None) == ("", [])
|
||||
|
||||
|
||||
def test_split_dedupes_tags():
|
||||
_, tags = split_desc("标题 #方言 #方言 #四川话", None)
|
||||
assert tags == ["方言", "四川话"], tags
|
||||
|
||||
|
||||
def test_split_bad_offsets_dont_corrupt():
|
||||
"""偏移量对不上时必须退回正则，绝不能把标题切坏。"""
|
||||
title, tags = split_desc("标题正文 #标签", te_broken("标签"))
|
||||
assert title == "标题正文", repr(title)
|
||||
assert tags == ["标签"], tags
|
||||
|
||||
|
||||
def test_split_keeps_hash_inside_word():
|
||||
"""偏移量可信时不该误伤正文里的 #（如 C# 教程）——这正是用偏移的理由。"""
|
||||
desc = "C# 入门教程 #编程"
|
||||
title, tags = split_desc(desc, te(desc, "编程"))
|
||||
assert title == "C# 入门教程", repr(title)
|
||||
assert tags == ["编程"], tags
|
||||
|
||||
|
||||
def test_split_lone_hash_is_not_a_tag():
|
||||
"""# 后面必须跟非空白才算标签，所以 "C# 教程" 里的 # 不会被当标签。"""
|
||||
title, tags = split_desc("C# 入门教程 #编程", None)
|
||||
assert tags == ["编程"], tags
|
||||
assert title == "C# 入门教程", repr(title)
|
||||
|
||||
|
||||
def test_split_repeated_tagname_in_body_is_lossy():
|
||||
"""没有可信偏移、且标签名以 #名字 的形式也出现在正文里时，兜底会连正文
|
||||
一起删。这是正则兜底的已知代价，钉住它免得以后误以为无损。"""
|
||||
title, tags = split_desc("聊聊#教程 这件事 #教程", None)
|
||||
assert tags == ["教程"], tags
|
||||
assert title == "聊聊 这件事", repr(title) # 正文里那个 #教程 也没了
|
||||
|
||||
# 有可信偏移时就不会误伤——只挖服务端标注的那一处
|
||||
desc = "聊聊#教程 这件事 #教程"
|
||||
i = desc.rindex("#教程")
|
||||
ok = [{"type": 1, "hashtag_name": "教程", "start": i, "end": i + 3}]
|
||||
title2, _ = split_desc(desc, ok)
|
||||
assert title2 == "聊聊#教程 这件事", repr(title2)
|
||||
|
||||
|
||||
def test_split_case_insensitive_dedupe():
|
||||
"""抖音把 text_extra 里的标签归一化成小写，desc 里却是原始大小写。
|
||||
并集必须按大小写不敏感去重，否则同一个标签会存两份（实测踩到）。"""
|
||||
desc = "教学 #一站式AI创作工作台 #国产AI"
|
||||
te = [{"type": 1, "hashtag_name": "一站式ai创作工作台", "start": -1, "end": -1},
|
||||
{"type": 1, "hashtag_name": "国产ai", "start": -1, "end": -1}]
|
||||
title, tags = split_desc(desc, te)
|
||||
# 保留 desc 里的原始大小写，不是 text_extra 的小写版
|
||||
assert tags == ["一站式AI创作工作台", "国产AI"], tags
|
||||
assert title == "教学", repr(title)
|
||||
|
||||
|
||||
def test_split_drops_at_mention_pseudo_tag():
|
||||
"""正文里手打的 # 会被正则捕到，但带 @ 的明显是提及不是标签。
|
||||
实测：desc 里的 "#(提示词出自@青木"，抖音自己也没把它当标签。"""
|
||||
desc = "示例#正经标签 #(提示词出自@青木 文本#另一个"
|
||||
title, tags = split_desc(desc, None)
|
||||
assert tags == ["正经标签", "另一个"], tags
|
||||
# 它不进 tags，但仍然从标题里剥掉——作者本来就是当标签打的
|
||||
assert title == "示例 文本", repr(title)
|
||||
|
||||
|
||||
def test_split_strips_edge_punctuation():
|
||||
title, tags = split_desc("标题 #（括号标签） #正常", None)
|
||||
assert tags == ["括号标签", "正常"], tags
|
||||
# 标题里那对括号也得跟着标签一起剥掉——剥标题用的是 desc 原文，
|
||||
# 不是清洗过的标签名，否则会匹配不上而留在标题里
|
||||
assert title == "标题", repr(title)
|
||||
|
||||
|
||||
def test_recordlog_roundtrip_and_resume():
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "records.jsonl"
|
||||
|
||||
log = RecordLog(p)
|
||||
assert len(log) == 0
|
||||
assert log.write({"aweme_id": "111", "title": "甲", "raw": {"x": 1}}) is True
|
||||
assert log.write({"aweme_id": "222", "title": "乙", "raw": {"x": 2}}) is True
|
||||
assert "111" in log and "222" in log and "333" not in log
|
||||
|
||||
# 重开：应该扫出已有的两条
|
||||
log2 = RecordLog(p)
|
||||
assert len(log2) == 2, len(log2)
|
||||
assert "111" in log2 and "222" in log2
|
||||
|
||||
# 重复写入被幂等挡掉
|
||||
assert log2.write({"aweme_id": "111", "title": "改过的"}) is False
|
||||
assert len(p.read_text(encoding="utf-8").strip().splitlines()) == 2
|
||||
|
||||
# aweme_id 必须是每行第一个字段（续跑扫描的契约）
|
||||
for line in p.read_text(encoding="utf-8").splitlines():
|
||||
assert line.startswith('{"aweme_id":'), line[:40]
|
||||
|
||||
rows = [json.loads(x) for x in p.read_text(encoding="utf-8").splitlines()]
|
||||
assert rows[0]["title"] == "甲" and rows[0]["raw"] == {"x": 1}
|
||||
|
||||
|
||||
def test_recordlog_survives_broken_tail():
|
||||
"""进程被 Ctrl-C 打断可能留下半行，扫描不能因此崩掉。"""
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "records.jsonl"
|
||||
p.write_text('{"aweme_id":"111","title":"完整"}\n{"aweme_id":"222","tit',
|
||||
encoding="utf-8")
|
||||
log = RecordLog(p)
|
||||
assert "111" in log and "222" in log # 半行的 id 也认，照样跳过
|
||||
|
||||
log.write({"aweme_id": "333", "title": "新的"})
|
||||
lines = p.read_text(encoding="utf-8").splitlines()
|
||||
# 补写前必须先断行，不能跟半行黏成一行
|
||||
assert lines[-1].startswith('{"aweme_id":"333"'), lines[-1][:40]
|
||||
assert json.loads(lines[-1])["title"] == "新的"
|
||||
|
||||
|
||||
def test_recordlog_no_ascii_escaping():
|
||||
"""中文必须原样存，不能转成 \\uXXXX——否则文件体积翻几倍且不可读。"""
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
p = Path(d) / "r.jsonl"
|
||||
RecordLog(p).write({"aweme_id": "1", "title": "四川方言"})
|
||||
assert "四川方言" in p.read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
|
||||
failed = 0
|
||||
for t in tests:
|
||||
try:
|
||||
t()
|
||||
print(f" ok {t.__name__}")
|
||||
except AssertionError as exc:
|
||||
failed += 1
|
||||
print(f" FAIL {t.__name__}: {exc}")
|
||||
print(f"\n{len(tests) - failed}/{len(tests)} 通过")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
335
tests/test_pipeline.py
Normal file
335
tests/test_pipeline.py
Normal file
@@ -0,0 +1,335 @@
|
||||
"""pipeline.run_one 的端到端测试。不联网、不碰 COS。
|
||||
|
||||
网络（fetch_detail / download）和 COS（CosUploader）都打桩，但 ffmpeg 抽取、
|
||||
ffprobe 时长闸门、记录写入、本地删除全都真跑——这几步才是会悄悄丢数据的地方。
|
||||
测试用的 mp4 由 ffmpeg 现造。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
from pathlib import Path
|
||||
|
||||
import pipeline
|
||||
from douyin.client import Aweme
|
||||
from douyin.app import DeviceRegisterError
|
||||
from douyin.cos import SizeMismatch
|
||||
from douyin.download import DownloadResult
|
||||
from douyin.meta import RecordLog
|
||||
|
||||
DUR = 3.0
|
||||
|
||||
|
||||
def make_mp4(path: Path, seconds: float = DUR) -> None:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
|
||||
"-f", "lavfi", "-i", f"testsrc=duration={seconds}:size=320x240:rate=10",
|
||||
"-f", "lavfi", "-i", f"sine=frequency=440:duration={seconds}",
|
||||
"-c:v", "libx264", "-c:a", "aac", "-shortest", str(path)],
|
||||
check=True, capture_output=True)
|
||||
|
||||
|
||||
def fake_aweme(aweme_id="7000000000000000001"):
|
||||
raw = {
|
||||
"aweme_id": aweme_id,
|
||||
"desc": "测试标题 #方言 #四川话",
|
||||
"text_extra": [{"type": 1, "hashtag_name": "方言", "hashtag_id": "9",
|
||||
"start": 4, "end": 7}],
|
||||
"author": {"nickname": "某作者", "uid": "123", "sec_uid": "MS4w"},
|
||||
"statistics": {"digg_count": 10, "comment_count": 2, "collect_count": 1,
|
||||
"share_count": 3, "play_count": 999},
|
||||
"music": {"title": "原声", "author": "某作者"},
|
||||
"video": {"duration": 3050, "width": 320, "height": 240},
|
||||
}
|
||||
return Aweme(aweme_id=aweme_id, desc=raw["desc"], author="某作者",
|
||||
create_time=1582635538, video_urls=["http://x/v.mp4"],
|
||||
cover_urls=["http://x/c.jpg"], music_url="http://x/m.mp3",
|
||||
uri="v0200fg10000test", raw=raw)
|
||||
|
||||
|
||||
class StubPool:
|
||||
def current(self):
|
||||
return object(), 0
|
||||
|
||||
def report_dead(self, gen):
|
||||
pass
|
||||
|
||||
|
||||
class StubUploader:
|
||||
"""记下传了什么，不真传。"""
|
||||
|
||||
def __init__(self, fail_on=None):
|
||||
self.put_calls = []
|
||||
self.fail_on = fail_on
|
||||
|
||||
def key_for(self, name):
|
||||
return f"0005/zh/{name}"
|
||||
|
||||
def put(self, local, key):
|
||||
if self.fail_on and self.fail_on in key:
|
||||
raise SizeMismatch(f"{key} 大小不符：COS=1 本地=2")
|
||||
size = Path(local).stat().st_size
|
||||
self.put_calls.append((key, size))
|
||||
return f"cos://bucket/{key}", size
|
||||
|
||||
|
||||
def make_args(work, keep_local=False):
|
||||
return argparse.Namespace(work_dir=str(work), bitrate="96k", ffprobe="ffprobe",
|
||||
keep_local=keep_local)
|
||||
|
||||
|
||||
def run(tmp, *, src_mp4, keep_local=False, uploader=None, aweme=None,
|
||||
patch_download=None):
|
||||
"""跑一次 run_one，返回 (records, failures, uploader, work_dir)。"""
|
||||
work = Path(tmp) / "work"
|
||||
work.mkdir(exist_ok=True)
|
||||
aweme = aweme or fake_aweme()
|
||||
up = uploader or StubUploader()
|
||||
log = RecordLog(Path(tmp) / "records.jsonl")
|
||||
fails = []
|
||||
|
||||
def dl(session, urls, dest, **kw):
|
||||
shutil.copy(src_mp4, dest)
|
||||
return DownloadResult(dest, dest.stat().st_size, resumed=False)
|
||||
|
||||
orig_dl, orig_fetch = pipeline.download, pipeline.AppClient
|
||||
pipeline.download = patch_download or dl
|
||||
pipeline.AppClient = lambda dev: type("C", (), {"fetch_detail": lambda s, i: aweme})()
|
||||
try:
|
||||
pipeline.run_one(
|
||||
aweme.aweme_id, "http://src", pool=StubPool(), up=up, log=log,
|
||||
args=make_args(work, keep_local), ffmpeg="ffmpeg",
|
||||
ff_sem=threading.Semaphore(2), stats=pipeline.Stats(),
|
||||
fail=lambda i, s, m: fails.append((i, s, m)),
|
||||
abort=threading.Event())
|
||||
finally:
|
||||
pipeline.download, pipeline.AppClient = orig_dl, orig_fetch
|
||||
|
||||
recs = [json.loads(l) for l in
|
||||
(Path(tmp) / "records.jsonl").read_text(encoding="utf-8").splitlines() if l.strip()] \
|
||||
if (Path(tmp) / "records.jsonl").exists() else []
|
||||
return recs, fails, up, work
|
||||
|
||||
|
||||
def test_happy_path_writes_record_and_deletes_local():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src)
|
||||
recs, fails, up, work = run(tmp, src_mp4=src)
|
||||
|
||||
assert not fails, fails
|
||||
assert len(recs) == 1, recs
|
||||
r = recs[0]
|
||||
assert r["aweme_id"] == "7000000000000000001"
|
||||
assert r["title"] == "测试标题", repr(r["title"])
|
||||
assert r["tags"] == ["方言", "四川话"], r["tags"]
|
||||
assert r["author"]["uid"] == "123"
|
||||
assert r["stats"]["play"] == 999
|
||||
assert abs(r["duration_sec"] - DUR) < 0.5, r["duration_sec"]
|
||||
assert r["duration_ms_api"] == 3050 # API 声明值原样留着
|
||||
assert r["cos_mp4"].endswith(".mp4") and r["cos_mp3"].endswith(".mp3")
|
||||
assert r["raw"]["author"]["sec_uid"] == "MS4w"
|
||||
|
||||
# 两个都传了
|
||||
assert len(up.put_calls) == 2, up.put_calls
|
||||
# 本地已删干净
|
||||
assert list(work.iterdir()) == [], list(work.iterdir())
|
||||
|
||||
|
||||
def test_keep_local_keeps_files():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src)
|
||||
recs, fails, up, work = run(tmp, src_mp4=src, keep_local=True)
|
||||
assert not fails, fails
|
||||
names = sorted(p.suffix for p in work.iterdir())
|
||||
assert names == [".mp3", ".mp4"], names
|
||||
|
||||
|
||||
def test_upload_failure_keeps_local_and_writes_no_record():
|
||||
"""上传失败必须保留本地文件——不然重跑就得重新下载。"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src)
|
||||
recs, fails, up, work = run(tmp, src_mp4=src,
|
||||
uploader=StubUploader(fail_on=".mp3"))
|
||||
assert recs == [], recs
|
||||
assert len(fails) == 1 and fails[0][1] == "upload", fails
|
||||
kept = sorted(p.suffix for p in work.iterdir())
|
||||
assert kept == [".mp3", ".mp4"], kept
|
||||
|
||||
|
||||
def test_truncated_mp3_is_caught_before_delete():
|
||||
"""时长闸门：mp3 被截断时必须拦下，且不删本地、不写记录。"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src, seconds=10.0)
|
||||
|
||||
orig = pipeline.extract_mp3
|
||||
|
||||
def truncating(ffmpeg, s, dst, bitrate, timeout=600.0):
|
||||
orig(ffmpeg, s, dst, bitrate, timeout)
|
||||
data = dst.read_bytes()
|
||||
dst.write_bytes(data[: len(data) // 5]) # 砍掉 80%
|
||||
|
||||
pipeline.extract_mp3 = truncating
|
||||
try:
|
||||
recs, fails, up, work = run(tmp, src_mp4=src)
|
||||
finally:
|
||||
pipeline.extract_mp3 = orig
|
||||
|
||||
assert recs == [], recs
|
||||
assert len(fails) == 1 and fails[0][1] == "probe", fails
|
||||
assert "截断" in fails[0][2], fails[0][2]
|
||||
assert up.put_calls == [], "截断的文件绝不该被上传"
|
||||
assert len(list(work.iterdir())) == 2 # 本地保留待查
|
||||
|
||||
|
||||
def test_register_exhausted_aborts_batch():
|
||||
"""IP 级注册配额耗尽时必须中止整批，不能把剩余任务逐条烧成失败。
|
||||
上一批就是这么烧掉 678 条的。"""
|
||||
class DeadPool:
|
||||
def current(self):
|
||||
raise DeviceRegisterError("所有 host 都返回 device_id=0")
|
||||
|
||||
def report_dead(self, gen):
|
||||
pass
|
||||
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
work = Path(tmp) / "work"
|
||||
work.mkdir()
|
||||
log = RecordLog(Path(tmp) / "r.jsonl")
|
||||
fails = []
|
||||
abort = threading.Event()
|
||||
|
||||
for i in range(3):
|
||||
pipeline.run_one(f"700000000000000000{i}", "u", pool=DeadPool(),
|
||||
up=StubUploader(), log=log, args=make_args(work),
|
||||
ffmpeg="ffmpeg", ff_sem=threading.Semaphore(2),
|
||||
stats=pipeline.Stats(), fail=lambda a, b, c: fails.append((a, b, c)),
|
||||
abort=abort)
|
||||
|
||||
assert abort.is_set(), "第一条就该拉起中止信号"
|
||||
# 只有触发中止的那条记一次失败，后两条直接返回，不重复撞注册
|
||||
assert len(fails) == 1, fails
|
||||
assert fails[0][1] == "abort", fails
|
||||
|
||||
|
||||
def test_download_failure_is_reported():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src)
|
||||
|
||||
def boom(session, urls, dest, **kw):
|
||||
raise OSError("连接被重置")
|
||||
|
||||
recs, fails, up, work = run(tmp, src_mp4=src, patch_download=boom)
|
||||
assert recs == [] and len(fails) == 1
|
||||
assert fails[0][1] == "download" and "连接被重置" in fails[0][2], fails
|
||||
|
||||
|
||||
def test_resume_skips_completed():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
src = Path(tmp) / "src.mp4"
|
||||
make_mp4(src)
|
||||
run(tmp, src_mp4=src)
|
||||
recs2, fails2, up2, _ = run(tmp, src_mp4=src) # 再跑一次同一条
|
||||
assert len(recs2) == 1, "重复跑不该写出第二行"
|
||||
assert not fails2, fails2
|
||||
|
||||
|
||||
def test_sweep_removes_files_already_in_records():
|
||||
"""work/ 里凡是已经在 records.jsonl 里的文件都是废物：早就传上去了，
|
||||
留着只会撑爆磁盘。实测一次 66411 条的批跑到 2.7% 就攒了 11 GB。
|
||||
但**失败项的文件必须留下**——那是重跑续传的本钱，见
|
||||
test_upload_failure_keeps_local_and_writes_no_record。"""
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
work = Path(tmp) / "work"
|
||||
work.mkdir()
|
||||
recs = Path(tmp) / "records.jsonl"
|
||||
recs.write_text(
|
||||
json.dumps({"aweme_id": "111", "cos_mp4": "cos://x/111.mp4"}) + chr(10)
|
||||
+ json.dumps({"aweme_id": "222", "cos_mp4": "cos://x/222.mp4"}) + chr(10),
|
||||
encoding="utf-8")
|
||||
for name in ("111.mp4", "111.mp3", "222.mp4", "333.mp4", "333.mp3"):
|
||||
(work / name).write_bytes(b"x" * 10)
|
||||
(work / "444.mp4.part").write_bytes(b"x" * 10)
|
||||
|
||||
freed = pipeline.sweep_work_dir(work, recs)
|
||||
|
||||
left = sorted(p.name for p in work.iterdir())
|
||||
assert left == ["333.mp3", "333.mp4"], left # 333 没入库，留着重跑
|
||||
# 111.mp4 + 111.mp3 + 222.mp4 已入库，加上 444.mp4.part
|
||||
assert freed == 4, freed
|
||||
|
||||
|
||||
def test_sweep_tolerates_missing_paths():
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
work = Path(tmp) / "work"
|
||||
assert pipeline.sweep_work_dir(work, Path(tmp) / "nope.jsonl") == 0
|
||||
work.mkdir()
|
||||
assert pipeline.sweep_work_dir(work, Path(tmp) / "nope.jsonl") == 0
|
||||
|
||||
|
||||
def test_plan_batch_splits_long_runs():
|
||||
"""长跑必须切段：单个进程跑到 1.5 万条上下，Windows 会话的桌面堆就被
|
||||
几万个 ffmpeg/ffprobe 子进程耗光，之后 CreateProcess 一律失败
|
||||
（rc=3221225794 / STATUS_DLL_INIT_FAILED）。表现是完成数冻结、失败数
|
||||
猛涨、work/ 暴涨。实测两次：三进程并跑时撞过一次，单进程跑到 16399 条
|
||||
又撞一次 —— 所以是累积量的问题，不是并发数。切段后每段跑完重启进程，
|
||||
桌面堆跟着重置。"""
|
||||
pending = [(str(i), f"u{i}") for i in range(100)]
|
||||
batch, more = pipeline.plan_batch(pending, 30)
|
||||
assert len(batch) == 30 and more is True
|
||||
batch, more = pipeline.plan_batch(pending, 100)
|
||||
assert len(batch) == 100 and more is False, "刚好跑完不该再重启"
|
||||
batch, more = pipeline.plan_batch(pending, 500)
|
||||
assert len(batch) == 100 and more is False
|
||||
batch, more = pipeline.plan_batch(pending, 0)
|
||||
assert len(batch) == 100 and more is False, "0 = 不切段"
|
||||
|
||||
|
||||
def test_next_abort_wait_backs_off():
|
||||
"""撞 IP 级注册限流后要等冷却再续跑。连着撞说明还没解封，等得越来越久；
|
||||
一旦有一段跑成了就把等待打回底价。上限别太长，不然白白空转。"""
|
||||
assert pipeline.next_abort_wait(0, base=1800, cap=14400) == 1800
|
||||
assert pipeline.next_abort_wait(1, base=1800, cap=14400) == 3600
|
||||
assert pipeline.next_abort_wait(2, base=1800, cap=14400) == 7200
|
||||
assert pipeline.next_abort_wait(3, base=1800, cap=14400) == 14400
|
||||
assert pipeline.next_abort_wait(9, base=1800, cap=14400) == 14400, "要封顶"
|
||||
assert pipeline.next_abort_wait(0, base=0, cap=14400) == 0, "0=不等，直接退出"
|
||||
|
||||
|
||||
def main() -> int:
|
||||
if not shutil.which("ffmpeg") or not shutil.which("ffprobe"):
|
||||
print("跳过：需要 ffmpeg/ffprobe")
|
||||
return 0
|
||||
tests = [v for k, v in sorted(globals().items()) if k.startswith("test_")]
|
||||
failed = 0
|
||||
for t in tests:
|
||||
try:
|
||||
t()
|
||||
print(f" ok {t.__name__}")
|
||||
except AssertionError as exc:
|
||||
failed += 1
|
||||
print(f" FAIL {t.__name__}: {exc}")
|
||||
except Exception as exc:
|
||||
failed += 1
|
||||
print(f" ERR {t.__name__}: {type(exc).__name__}: {exc}")
|
||||
print(f"\n{len(tests) - failed}/{len(tests)} 通过")
|
||||
return 1 if failed else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
77
tests/test_smoke.py
Normal file
77
tests/test_smoke.py
Normal file
@@ -0,0 +1,77 @@
|
||||
"""不联网的冒烟测试：验证 crypto 正确性、链接解析、文件名清洗、去重记录。"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
# 移到 scripts/ tests/ 后，直接运行时脚本目录成了 sys.path[0]，
|
||||
# 把项目根加回来，import douyin / pipeline 才找得到。
|
||||
import sys as _sys
|
||||
from pathlib import Path as _Path
|
||||
_sys.path.insert(0, str(_Path(__file__).resolve().parent.parent))
|
||||
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from douyin import Registry, extract_aweme_id, safe_filename
|
||||
from douyin.crypto import self_test
|
||||
from douyin.urls import ParseError
|
||||
|
||||
|
||||
def test_crypto():
|
||||
self_test()
|
||||
|
||||
|
||||
def test_extract_id():
|
||||
cases = {
|
||||
"7123456789012345678": "7123456789012345678",
|
||||
"https://www.douyin.com/video/7123456789012345678": "7123456789012345678",
|
||||
"https://www.douyin.com/note/7123456789012345678": "7123456789012345678",
|
||||
"https://www.douyin.com/?modal_id=7123456789012345678": "7123456789012345678",
|
||||
"https://www.iesdouyin.com/share/video/7123456789012345678/": "7123456789012345678",
|
||||
"看看这个 https://www.douyin.com/video/7123456789012345678?a=1 好玩":
|
||||
"7123456789012345678",
|
||||
}
|
||||
for raw, want in cases.items():
|
||||
got = extract_aweme_id(raw)
|
||||
assert got == want, f"{raw!r} -> {got}, 期望 {want}"
|
||||
|
||||
for bad in ["随便一段没有链接的文字", "https://example.com/foo"]:
|
||||
try:
|
||||
extract_aweme_id(bad)
|
||||
except ParseError:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError(f"{bad!r} 本应报 ParseError")
|
||||
print("extract_aweme_id OK")
|
||||
|
||||
|
||||
def test_safe_filename():
|
||||
assert safe_filename('a/b\\c:d*e?f"g<h>i|j') == "a_b_c_d_e_f_g_h_i_j"
|
||||
assert safe_filename(" ") == "untitled"
|
||||
assert len(safe_filename("x" * 200)) == 80
|
||||
assert safe_filename("多个 空格") == "多个 空格"
|
||||
print("safe_filename OK")
|
||||
|
||||
|
||||
def test_registry():
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d)
|
||||
reg = Registry(root / ".downloaded.json")
|
||||
f = root / "v.mp4"
|
||||
f.write_bytes(b"x" * 10)
|
||||
assert "111" not in reg
|
||||
reg.add("111", f, 10, "标题")
|
||||
assert "111" in reg
|
||||
# 重新加载后仍在
|
||||
assert "111" in Registry(root / ".downloaded.json")
|
||||
# 文件被删则视为未下载
|
||||
f.unlink()
|
||||
assert "111" not in Registry(root / ".downloaded.json")
|
||||
print("Registry OK")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
test_crypto()
|
||||
test_extract_id()
|
||||
test_safe_filename()
|
||||
test_registry()
|
||||
print("\n全部冒烟测试通过")
|
||||
Reference in New Issue
Block a user