{"componentChunkName":"component---src-templates-acg-portal-new-template-tsx","path":"/emqeuzjvz","result":{"data":{"markdownRemark":{"html":"

简介

\n

句相似度语义切分 - 基于句子相似度的智能文本切分解决方案

\n

功能描述

\n\n

算子参数

\n

输入

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
输入含义
texts待处理的文本数组，要求元素类型为字符串。
\n

输出

\n\n\n\n\n\n\n\n\n\n\n\n\n\n
输出含义
resultpyarrow.Array: 切分后的文本块，元素为List[str]类型。
\n

参数

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
参数名称类型默认值描述
chunk_sizeint500chunk长度 描述：chunk的长度，单位为字符 默认值：500
chunk_overlapint50chunk重叠长度 描述：切分文本时chunk之间重叠的最大长度 默认值：50
model_pathstr'/opt/aihc/models'模型路径 描述：嵌入模型的根目录路径 默认值：\"/opt/aihc/models\"
embedding_model_namestr'BAAI/bge-m3'嵌入模型名称 描述：用于计算句子相似度的嵌入模型名称 默认值：\"BAAI/bge-m3\"
breakpoint_percentile_thresholdint80断点百分位阈值 描述：确定语义断点的百分位阈值，值越高，切分越保守 默认值：80
\n

调用示例

\n\n
\n
\n
\n Python\n \n
\n
\n 
1from __future__ import annotations\n2\n3import os\n4\n5import daft\n6from daft import col\n7\n8from daft.aihc.common.udf import aihc_udf\n9from daft.aihc.functions.text.chunk_text_sentence_similarity import ChunkTextSentenceSimilarity\n10\n11if __name__ == \"__main__\":\n12    if os.getenv(\"DAFT_RUNNER\", \"native\") == \"ray\":\n13        import ray\n14        ray.init(dashboard_host=\"0.0.0.0\", ignore_reinit_error=True)\n15        daft.set_runner_ray()\n16    daft.set_execution_config(actor_udf_ready_timeout=6000, min_cpu_per_task=0)\n17\n18    # TODO: 根据实际场景准备样本数据\n19    samples = {\"texts\": [...]}\n20    ds = daft.from_pydict(samples)\n21    constructor_kwargs = {\n22        \"chunk_size\": 500,\n23        \"chunk_overlap\": 50,\n24        \"model_path\": '/opt/aihc/models',\n25        \"embedding_model_name\": 'BAAI/bge-m3',\n26        \"breakpoint_percentile_threshold\": 80,\n27    }\n28    ds = ds.with_column(\n29        \"result\",\n30        aihc_udf(\n31            ChunkTextSentenceSimilarity,\n32            construct_args=constructor_kwargs,\n33            num_cpus=1,\n34            concurrency=4,\n35            batch_size=8,\n36        )(col(\"texts\")),\n37    )\n38    ds.show()
\n
\n
\n
\n ","fields":{"slug":"emqeuzjvz","title":"句相似度语义切分","date":"2026-06-15","extractedHeadings":[]},"headings":[{"value":"简介","depth":2},{"value":"功能描述","depth":3},{"value":"算子参数","depth":2},{"value":"输入","depth":3},{"value":"输出","depth":3},{"value":"参数","depth":3},{"value":"调用示例","depth":2}]}},"pageContext":{"isCreatedByStatefulCreatePages":false,"slug":"emqeuzjvz","prev":{"id":"Ampb4id5y","name":"文本长度计算器","path":"Ampb4id5y","filePath":"操作指南/AI数据处理/算子列表/文本/文本长度计算器.md","seo":null,"parentIds":["ilib2qygp","Ymo88m8hi","Imob3m6so","fmob4536z"],"parents":[{"id":"ilib2qygp","documentId":"bfa43a8b-968a-41a1-8c9d-906507eeaed9","name":"操作指南","repoName":"AIHC","filePath":"操作指南","disabled":false,"path":"ilib2qygp","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":null},{"id":"Ymo88m8hi","documentId":"c8cb5e38-f8c5-40f4-a424-b0c7895f0c0a","name":"AI数据处理","repoName":"AIHC","filePath":"操作指南/AI数据处理","disabled":false,"path":"Ymo88m8hi","lastMergeTime":"2026-04-21 14:23:10","isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"Imob3m6so","documentId":"fe548e34-6659-4ff5-86f6-eee2c43aec90","name":"算子列表","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表","disabled":false,"path":"Imob3m6so","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"fmob4536z","documentId":"cd84d7f5-def9-4cfd-a487-db445fbdf371","name":"文本","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表/文本","disabled":false,"path":"fmob4536z","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""}]},"next":{"id":"Xmqeuzjz2","name":"文本安全评分","path":"Xmqeuzjz2","filePath":"操作指南/AI数据处理/算子列表/文本/文本安全评分.md","seo":null,"parentIds":["ilib2qygp","Ymo88m8hi","Imob3m6so","fmob4536z"],"parents":[{"id":"ilib2qygp","documentId":"bfa43a8b-968a-41a1-8c9d-906507eeaed9","name":"操作指南","repoName":"AIHC","filePath":"操作指南","disabled":false,"path":"ilib2qygp","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":null},{"id":"Ymo88m8hi","documentId":"c8cb5e38-f8c5-40f4-a424-b0c7895f0c0a","name":"AI数据处理","repoName":"AIHC","filePath":"操作指南/AI数据处理","disabled":false,"path":"Ymo88m8hi","lastMergeTime":"2026-04-21 14:23:10","isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"Imob3m6so","documentId":"fe548e34-6659-4ff5-86f6-eee2c43aec90","name":"算子列表","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表","disabled":false,"path":"Imob3m6so","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"fmob4536z","documentId":"cd84d7f5-def9-4cfd-a487-db445fbdf371","name":"文本","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表/文本","disabled":false,"path":"fmob4536z","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""}]},"parents":[{"id":"ilib2qygp","documentId":"bfa43a8b-968a-41a1-8c9d-906507eeaed9","name":"操作指南","repoName":"AIHC","filePath":"操作指南","disabled":false,"path":"ilib2qygp","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":null},{"id":"Ymo88m8hi","documentId":"c8cb5e38-f8c5-40f4-a424-b0c7895f0c0a","name":"AI数据处理","repoName":"AIHC","filePath":"操作指南/AI数据处理","disabled":false,"path":"Ymo88m8hi","lastMergeTime":"2026-04-21 14:23:10","isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"Imob3m6so","documentId":"fe548e34-6659-4ff5-86f6-eee2c43aec90","name":"算子列表","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表","disabled":false,"path":"Imob3m6so","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""},{"id":"fmob4536z","documentId":"cd84d7f5-def9-4cfd-a487-db445fbdf371","name":"文本","repoName":"AIHC","filePath":"操作指南/AI数据处理/算子列表/文本","disabled":false,"path":"fmob4536z","lastMergeTime":null,"isApiDoc":null,"httpMethod":null,"seo":null,"sourceOrgName":null,"sourceRepoName":null,"sourceDocumentId":""}],"specificSeo":null}}}