feat(upload): add chunked upload support for form mode. form模式下分片传输。 #1928

LusiyAvA · 2026-01-06T20:41:42Z

Description / 描述

This PR implements chunked upload support for the form upload mode, improving reliability for large file transfers.
此 PR 为 Form 上传模式实现了分片上传支持，提高了大文件传输的可靠性。

Key implementation details:
主要实现细节：

Chunking Strategy: Uploads are split according to the chunk size configured in the backend settings.
分片策略：根据后台设置的分片大小进行上传。
Integrity Verification:
完整性校验：
- Uses CRC32 to verify the integrity of each individual chunk.
  使用 CRC32 校验每个单独分片的完整性。
- Uses xxHash to verify the integrity of the complete file after merging.
  使用 xxHash 校验合并后完整文件的完整性。

Motivation and Context / 背景

Standard multipart form uploads are prone to interruptions when transferring large files over unstable networks. Introducing chunked uploads with strong verification mechanisms ensures file integrity and improves the success rate of uploads.
标准的多部分 Form 上传在不稳定网络下传输大文件时容易中断。引入带有强校验机制的分片上传可以确保文件完整性并提高上传成功率。

How Has This Been Tested? / 测试

Manually tested uploading large files (>500MB) using the chunked form mode.
手动测试了使用分片 Form 模式上传大文件 (>500MB)。
Verified that the CRC32 check correctly rejects corrupted chunks.
验证了 CRC32 校验能正确拒绝损坏的分片。
Verified that the final xxHash check matches the source file.
验证了最终的 xxHash 校验与源文件匹配。
Code formatted with go fmt.
代码已使用 go fmt 格式化。

Checklist / 检查清单

I have read the CONTRIBUTING document.
我已阅读 CONTRIBUTING 文档。
I have formatted my code with go fmt or prettier.
我已使用 go fmt 或 prettier 格式化提交的代码。
I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
(Maintainers please add feat and backend labels / 请维护者添加 feat 和 backend 标签)
I have requested review from relevant code authors using the "Request review" feature when applicable.
我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
I have updated the repository accordingly (If it’s needed).
我已相应更新了相关仓库（若适用）。
[x] feat(upload): add chunked upload support for form mode. 添加分片逻辑。 OpenList-Frontend#356

在form模式下，会按照后台设置好的分片大小进行上传，CRC校验分片+xxhash校验完整文件保证可靠性。

This reverts commit d89fc20.

在form模式下，会按照后台设置好的分片大小进行上传，CRC校验分片+xxhash校验完整文件保证可靠性。

xrgzs · 2026-01-07T06:48:14Z

server/router.go

 	setting.POST("/set_transmission", handles.SetTransmission)
 	setting.POST("/set_115", handles.Set115)
 	setting.POST("/set_115_open", handles.Set115Open)
-	setting.POST("/set_123_pan", handles.Set123Pan)


It should not be removed here.

xrgzs · 2026-01-07T06:50:32Z

internal/bootstrap/data/setting.go

+
+		// HTTP Server 超时配置（支持大文件传输）
+		{Key: conf.HTTPServerReadTimeout, Value: "0", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP读取请求超时(秒)，0表示无限制，建议设为0以支持大文件上传"},
+		{Key: conf.HTTPServerWriteTimeout, Value: "0", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP写入响应超时(秒)，0表示无限制，建议设为0以支持大文件下载"},
+		{Key: conf.HTTPServerIdleTimeout, Value: "120", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP空闲连接超时(秒)，建议120秒允许网络短暂波动"},
+		{Key: conf.HTTPServerReadHeaderTimeout, Value: "30", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP读取Header超时(秒)，防止慢速攻击，建议30秒"},
+		{Key: conf.HTTPServerMaxHeaderBytes, Value: "1048576", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP Header最大字节数，默认1MB(1048576)"},
+		// 分片上传配置（绕过 Cloudflare CDN 限制）
+		{Key: conf.ChunkedUploadChunkSize, Value: "95", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: "分片上传阈值(MB)，超过此大小的文件将自动分片上传，建议设为95以绕过Cloudflare 100MB限制"},


Please fill in the help field in English and add the necessary i18n to the frontend.

thank you for your review. I have applied the requested changes and improved the code completeness:

Route Restoration: Restored the /set_123_pan route in
server/router.go
that was accidentally removed.
Backend Documentation: Translated all Help fields for the new HTTP Timeout and Chunked Upload settings to English in
internal/bootstrap/data/setting.go
.
Frontend i18n: Added the corresponding translation keys to the frontend English language file (
settings.json
).
Additional Fixes for Build Success: While addressing the feedback, I identified and fixed several issues that would otherwise cause compilation or runtime errors:

Defined Missing Constant: Added Pan123TempDir to
internal/conf/const.go
to resolve "undefined" errors during build.
Implemented Cleanup Logic: Implemented the
CleanStaleChunks
function in
internal/bootstrap/config.go
. This function was being called by the task manager but lacked a definition, which would lead to a linker error. It also ensures that stale chunk fragments are cleaned up to prevent disk exhaustion.
Build Verified: I have verified the entire project with a full build.

KirCute · 2026-01-07T10:47:22Z

internal/conf/const.go

 	// 123 open offline download
 	Pan123OpenOfflineDownloadCallbackUrl = "123_open_callback_url"
 	Pan123OpenTempDir                    = "123_open_temp_dir"
+	Pan123TempDir                        = "123_temp_dir"


补的位置不对，Pan123TempDir这个配置项是123破解驱动的离线下载，不是123开放平台驱动的离线下载，应该空一行并写注释

// 123

KirCute · 2026-01-07T10:50:28Z

server/handles/fsup.go

 		t, err = fs.PutAsTask(c.Request.Context(), dir, s)
 	} else {
-		err = fs.PutDirectly(c.Request.Context(), dir, s)
+		err = fs.PutDirectly(c.Request.Context(), dir, s, true)


这里为什么skipHook

KirCute · 2026-01-07T10:50:41Z

server/handles/fsup.go

 		t, err = fs.PutAsTask(c.Request.Context(), dir, s)
 	} else {
-		err = fs.PutDirectly(c.Request.Context(), dir, s)
+		err = fs.PutDirectly(c.Request.Context(), dir, s, true)


KirCute · 2026-01-07T11:17:19Z

server/handles/fsup.go

+	}
+
+	// For as_task=true (large files), immediately return and process in background
+	if req.AsTask {


从此往下有以下几个问题：

不应将分块文件合并为一个，这会占用两倍的存储空间，应当使用io.MultiReader。

在 merge 阶段校验哈希耗时太长，更建议在分片上传阶段 Write，在 merge 阶段 Sum。

goroutine里面和外面有大量重复代码，建议将公共部分放在函数里。

AsTask开启时返回的id应从PutAsTask的返回值中得到，考虑到PutAsTask在goroutine里调用，你可以直接像AsTask为false时那样返回空的id。（其实不采取建议2的情况下，我更建议你在上传任务里进行整个哈希校验流程）

KirCute · 2026-01-07T11:19:44Z

server/handles/fsup.go

+		return
+	}
+
+	user := c.Request.Context().Value(conf.UserKey).(*model.User)


我没找到开始分片上传在哪个位置，但总之应该在开始分片上传时校验用户是否具有写权限。

在分片上传函数和合并函数里应当校验用户尝试处理的分片组是不是同一个用户创建的。

KirCute · 2026-01-07T11:20:50Z

pkg/utils/hash.go

+	// CRC32 indicates CRC-32 support (IEEE polynomial)
+	CRC32 = RegisterHash("crc32", "CRC-32", 8, func() hash.Hash { return crc32.NewIEEE() })
+
+	// CRC64 indicates CRC-64 support (ECMA polynomial)
+	CRC64 = RegisterHash("crc64", "CRC-64", 16, func() hash.Hash { return crc64.New(crc64.MakeTable(crc64.ECMA)) })
+
+	// XXH64 indicates xxHash64 support
+	XXH64 = RegisterHash("xxh64", "XXH64", 16, func() hash.Hash { return xxhash.New() })


不是项目里所有用到的哈希算法都要在这个位置注册，这个位置注册的是网盘可能返回的用于秒传的哈希值。

KirCute · 2026-01-07T11:26:22Z

internal/bootstrap/task.go

+	// Start background task to clean stale chunks every 10 minutes
+	go func() {
+		c := cron.NewCron(10 * time.Minute)
+		c.Do(CleanStaleChunks)
+	}()


清理无用分片跟任务系统无关，应该将其放在init函数或bootstrap.Init中，此外，cron.Do函数不阻塞，所以不需要将其放在单独的goroutine中

KirCute · 2026-01-07T11:28:49Z

internal/bootstrap/data/setting.go

+		// HTTP Server configuration (for large file transfers)
+		{Key: conf.HTTPServerReadTimeout, Value: "0", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP read request timeout (seconds), 0 means no limit. Recommended to set to 0 to support large file uploads."},
+		{Key: conf.HTTPServerWriteTimeout, Value: "0", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP write response timeout (seconds), 0 means no limit. Recommended to set to 0 to support large file downloads."},
+		{Key: conf.HTTPServerIdleTimeout, Value: "120", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP idle connection timeout (seconds). Recommended to set to 120 seconds to allow for brief network fluctuations."},
+		{Key: conf.HTTPServerReadHeaderTimeout, Value: "30", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "HTTP read header timeout (seconds) to prevent slow attacks. Recommended to set to 30 seconds."},
+		{Key: conf.HTTPServerMaxHeaderBytes, Value: "1048576", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PRIVATE, Help: "Maximum bytes for HTTP header, defaults to 1MB (1048576)."},
+		// Chunked upload configuration (to bypass Cloudflare CDN limits)
+		{Key: conf.ChunkedUploadChunkSize, Value: "95", Type: conf.TypeNumber, Group: model.TRAFFIC, Flag: model.PUBLIC, Help: "Chunked upload threshold (MB). Files exceeding this size will be uploaded in chunks. Recommended to set to 95 to bypass Cloudflare's 100MB limit."},


我没有找到这些配置项是在哪用到的，如果是前端用到的，应该将Flag全部设为model.PUBLIC，否则前端看不到这个配置项的值。

KirCute · 2026-01-07T11:34:14Z

这个是不是跟 #1877 撞了

LusiyAvA added 3 commits January 7, 2026 04:34

form模式下分片传输。

d89fc20

在form模式下，会按照后台设置好的分片大小进行上传，CRC校验分片+xxhash校验完整文件保证可靠性。

Revert "form模式下分片传输。"

360bf73

This reverts commit d89fc20.

form模式下分片传输。

61d51ac

在form模式下，会按照后台设置好的分片大小进行上传，CRC校验分片+xxhash校验完整文件保证可靠性。

LusiyAvA mentioned this pull request Jan 6, 2026

feat(upload): add chunked upload support for form mode. 添加分片逻辑。 OpenListTeam/OpenList-Frontend#356

Open

9 tasks

xrgzs requested changes Jan 7, 2026

View reviewed changes

Find the lost codes back.

398955e

KirCute requested changes Jan 7, 2026

View reviewed changes

Uh oh!

feat(upload): add chunked upload support for form mode. form模式下分片传输。 #1928

Are you sure you want to change the base?

feat(upload): add chunked upload support for form mode. form模式下分片传输。 #1928

Uh oh!

Conversation

LusiyAvA commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description / 描述

Motivation and Context / 背景

How Has This Been Tested? / 测试

Checklist / 检查清单

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KirCute Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KirCute commented Jan 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LusiyAvA commented Jan 6, 2026 •

edited

Loading

KirCute Jan 7, 2026 •

edited

Loading