上传文件

在MSS中,用户操作的基本数据单元是文件(Object)。MSS Java SDK提供了丰富的文件上传接口,可以通过以下方式上传文件:

  • 流式上传
  • 文件上传
  • 分片上传

流式上传、文件上传的文件(Object)最大不能超过5GB。当文件较大时,请使用分片上传。

上传字符串

public void putObjectExample(String bucketName, String objectName, String content){
    try{
        //bucketName指定上传文件所在的桶名
        //objectName指定上传的文件名
        //content指定上传的文件内容
        s3client.putObject(bucketName,objectName,new ByteArrayInputStream(content.getBytes()),null);
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

上传网络流

public void pubObjectInputStreamExample(String bucketName, String objectName){
    try{
        //bucketName指定上传文件所在的桶名
        //objectName指定上传的文件名
        //content指定上传的文件内容
        InputStream inputStream = new URL("https://mtyun.com/").openStream();
        s3client.putObject(bucketName,objectName,inputStream,null);
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }catch(IOException e)
    {
        e.printStackTrace();
    }
}

上传本地文件

public void putObjectFileExample(String bucketName, String objectName){
    try{
        String uploadFileName = "/Users/junechen/learningDir/s3-mirrorserver/src/store/index.html";
        FileInputStream inputStream = new FileInputStream(uploadFileName);
        s3client.putObject(bucketName,objectName,inputStream,null);
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }catch(IOException e)
    {
        e.printStackTrace();
    }
}

设置元信息

文件元信息(Object Meta),是对用户上传到MSS的文件的属性描述,分为两种:HTTP标准属性(HTTP Headers)和User Meta(用户自定义元信息)。 文件元信息可以在各种方式上传(流上传、文件上传、追加上传、分片上传、断点续传),或拷贝文件时进行设置。元信息的名称大小写不敏感。
设定http header

名称 描述 默认值
Content-MD5 文件数据校验,设置了该值后MSS会启用文件内容MD5校验,把您提供的MD5与文件的MD5比较,不一致会抛出错误
Content-Type 文件的MIME,定义文件的类型及网页编码,决定浏览器将以什么形式、什么编码读取文件。如果用户没有指定则根据Key或文件名的扩展名生成,如果没有扩展名则填默认值 application/octet-stream
Content-Disposition 指示MINME用户代理如何显示附加的文件,打开或下载,及文件名称
Expires 缓存过期时间,MSS未使用,格式是格林威治时间(GMT)
Cache-Control 指定该Object被下载时的网页的缓存行为
Content-Encoding 传输内容编码 -
public void putObjectWithMetaExample(String bucketName,String objectName){
    try{
        String objectContent = "hello world.";
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.setContentType("text/html");
        metadata.setContentDisposition("testContentDisposition");
        s3client.putObject(new PutObjectRequest(
                bucketName, objectName, new ByteArrayInputStream(objectContent.getBytes()), metadata));
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

用户自定义元信息

MSS支持用户自定义Object的元信息,对Object进行描述。

public void putObjectWithSelfDefineMetaExample(String bucketName,String objectName){
    try{
        String objectContent = "hello world.";
        ObjectMetadata metadata = new ObjectMetadata();
        metadata.addUserMetadata("filename","test.txt");
        s3client.putObject(new PutObjectRequest(
                bucketName, objectName, new ByteArrayInputStream(objectContent.getBytes()), metadata));
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

分片上传

对于大文件上传,可以切分成片上传。用户可以在如下的应用场景内(但不仅限于此),使用分片上传(Multipart Upload)模式:

  • 需要支持断点上传。
  • 上传超过100MB大小的文件。
  • 网络条件较差,和MSS的服务器之间的链接经常断开。

分片上传(Multipart Upload)分为如下3个步骤:

  • 初始化一个分片上传任务(InitiateMultipartUpload)
  • 逐个或并行上传分片(UploadPart)
  • 完成分片上传(CompleteMultipartUpload)或取消分片上传(AbortMultipartUpload)

分步完成Multipart Upload

使用Multipart Upload模式传输数据前,必须先通知MSS初始化一个Multipart Upload事件。该操作会返回一个MSS服务器创建的全局唯一的Upload ID,用于标识本次Multipart Upload事件。用户可以根据这个ID来发起相关的操作,如中止Multipart Upload、查询Multipart Upload等。
调用MSSClient.initiateMultipartUpload初始化一个分片上传事件:

public void initailMultipartExample(String bucketName, String objectName){
    try{
        InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
                bucketName, objectName);
        InitiateMultipartUploadResult initResponse =
                s3client.initiateMultipartUpload(initRequest);
        System.out.println("UploadId " + initResponse.getUploadId());
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

上传分片

初始化一个Multipart Upload之后,可以根据指定的Object名和Upload ID来分片(Part)上传数据。每一个上传的Part都有一个标识它的号码——分片号。对于同一个Upload ID,该分片号不但唯一标识这一块数据,也标识了这块数据在整个文件内的相对位置。如果你用同一个分片号码,上传了新的数据,那么MSS上已有的这个分片的数据将被覆盖。除了最后一块Part以外,其他的part最小为5M;最后一块Part没有大小限制。每个分片不需要按顺序上传,甚至可以在不同进程、不同机器上上传,MSS会按照分片号排序组成大文件。
调用MSSClient.uploadPart上传分片:

public void uploadMultipartExample(String bucketName, String objectName, String uploadID, int partNum){
    try{
        // Create request to upload a part.
        String partContent = "part content";
        UploadPartRequest uploadRequest = new UploadPartRequest()
                .withBucketName(bucketName)
                .withKey(objectName)
                .withUploadId(uploadID)
                .withPartNumber(partNum)
                .withInputStream(new ByteArrayInputStream(partContent.getBytes()))
                .withPartNumber(partContent.length());

        // Upload part.
        s3client.uploadPart(uploadRequest);
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

完成分片上传

所有分片上传完成后,需要调用Complete Multipart Upload来完成整个文件的Multipart Upload。在执行该操作时,需要提供所有有效的分片列表(包括分片号和分片ETAG);MSS收到提交的分片列表后,会逐一验证每个分片的有效性。当所有的数据Part验证通过后,MSS将把这些分片组合成一个完整的Object。
调用MSSClient.completeMultipartUpload完成分片上传:

public void completeMultipartExample(String bucketName, String objectName){
    try{
        //partETags用于存储上传完成的每个分片的etag值,在调用分片文件上传完成接口是需要用到
        List<PartETag> partETags = new ArrayList<PartETag>();
        //指定上传的本地大文件
        String filePath = "/opt/bigfile";

        //初始化分片上传
        InitiateMultipartUploadRequest initRequest = new InitiateMultipartUploadRequest(
                bucketName, objectName);
        InitiateMultipartUploadResult initResponse =
                s3client.initiateMultipartUpload(initRequest);

        //指定分片的大小
        File file = new File(filePath);
        long contentLength = file.length();
        long partSize = 50 * 1024 * 1024; // Set part size to 50 MB.
        long filePosition = 0;

        for (int i = 1; filePosition < contentLength; i++) {
            // 最后一个分片可能小于指定的partSize
            partSize = Math.min(partSize, (contentLength - filePosition));

            // 创建一个分片上传请求
            UploadPartRequest uploadRequest = new UploadPartRequest()
                    .withBucketName(bucketName).withKey(objectName)
                    .withUploadId(initResponse.getUploadId()).withPartNumber(i)
                    .withFileOffset(filePosition)
                    .withFile(file)
                    .withPartSize(partSize);

            // 上传分片,同时将MSS返回的etag值加入数组中
            partETags.add(s3client.uploadPart(uploadRequest).getPartETag());

            filePosition += partSize;
        }

        //调用分片上传完成接口
        CompleteMultipartUploadRequest compRequest = new
                CompleteMultipartUploadRequest(bucketName,
                objectName,
                initResponse.getUploadId(),
                partETags);

        s3client.completeMultipartUpload(compRequest);
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

取消分片上传事件

该接口可以根据Upload ID中止对应的Multipart Upload事件。当一个Multipart Upload事件被中止后,就不能再使用这个Upload ID做任何操作,已经上传的Part数据也会被删除。
调用MSSClient.abortMultipartUpload取消分片上传事件:

public void abortMultipartUploadExample(String bucketName, String objectName, String uploadID){
    try{
        s3client.abortMultipartUpload(new AbortMultipartUploadRequest(
                bucketName, objectName, uploadID));
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

获取所有已上传分片

public void listMultipartUploadExample(String bucketName, String objectName, String uploadID){
    try{
        //创建获取分片的请求
        ListPartsRequest listpartreq = new ListPartsRequest(bucketName, objectName, uploadID);
        //调用获取分片接口,返回分片列表结果
        PartListing listresult = s3client.listParts(listpartreq);
        //输出分片信息
        List<PartSummary> parts_list = listresult.getParts();
        for (int i =0; i < parts_list.size(); i++) {
            PartSummary item = parts_list.get(i);
            //打印分片的etag
            System.out.println("getETag: " + item.getETag());
        }
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

获取Bucket内所有分片上传事件

列举分片上传事件可以罗列出所有执行中的分片上传事件,即已经初始化的尚未Complete或者Abort的分片上传事件。列举分片上传的可设置的参数如下:

参数 作用 方法
Prefix 限定返回的文件名(object)必须以Prefix作为前缀。注意使用Prefix查询时,返回的文件名(Object)中仍会包含Prefix。 ListMultipartUploadsRequest.setPrefix(String prefix)
Delimiter 用于对Object名字进行分组的字符。所有名字包含指定的前缀且第一次出现delimiter字符之间的object作为一组元素。 ListMultipartUploadsRequest.setDelimiter(String delimiter)
public void listMultipartUploadsExample(String bucketName){
    try{
        // 列举分片上传事件
        ListMultipartUploadsRequest listMultipartUploadsRequest = new ListMultipartUploadsRequest(bucketName);
        MultipartUploadListing multipartUploadListing = s3client.listMultipartUploads(listMultipartUploadsRequest);
        for (MultipartUpload multipartUpload : multipartUploadListing.getMultipartUploads()) {
            // Upload Id
            multipartUpload.getUploadId();
            // Key
            multipartUpload.getKey();
            // Date of initiate multipart upload
            multipartUpload.getInitiated();
        }
    }catch (AmazonServiceException ase) {
        //存储服务端处理异常
        System.out.println("Caught an ServiceException.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
        Assert.assertEquals(true, true);
    }catch (AmazonClientException ace) {
        //客户端处理异常
        System.out.println("Caught an ClientException.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}