存储服务质量保障问题

施展
武汉光电国家研究中心
光电信息存储研究部

https://shizhan.github.io/
https://shi_zhan.gitee.io/

内容大纲

  • 存储服务质量保障背景
  • 资源决策问题
  • 经典方法和实践

广泛应用的云

Source: https://www.kiwiqa.com/top-6-cloud-computing-trends-impacting-cloud-adoption-in-2020/

Source: https://www.canalys.com/newsroom/worldwide-cloud-market-q320

Pandemic boosts cloud consumption by a third in Q3 2020

Source: https://www.srgresearch.com/articles/cloud-market-growth-rate-nudges-amazon-and-microsoft-solidify-leadership

服务器整合

Source: Virtual hierarchies to support server consolidation, ISCA '07

机遇

意义

更丰富意义

  1. 规模化应用部署
  2. 扩展、可靠和安全
  3. 使能服务质量保障

Source: Beyond Server Consolidation: Server consolidation helps companies improve resource utilization, but virtualization can help in other ways, too. Queue 2008

案例

Source: Update on Virtualizing SQL

Source: Update on Virtualizing SQL

多租户存储

Source:
What is a multi-tenant system?
https://tech.asimio.net/2017/01/17/Multitenant-applications-using-Spring-Boot-JPA-Hibernate-and-Postgres.html

Source: https://www.getfilecloud.com/blog/2014/06/launching-tonido-filecloud-6-0-multi-tenant-amazon-s3-storage-ntfs-permissions-support-and-more/

服务等级协议、服务等级目标、服务等级指标

  • 服务等级协议(SLA):协议双方(服务提供者和用户)签订的一个合约或协议,这个合约规范了双方的商务关系或部分商务关系
    • SLA是服务提供者与用户之间协商并签订的一个具有法律约束力的合同,合同规定了在服务提供过程中双方所承担的商务条款。
  • 服务等级目标(SLO):指定服务所提供功能的一种期望状态
  • 服务等级指标(SLI):经过仔细定义的测量指标,它根据不同系统特点确定要测量什么。

Source: SLO(服务等级目标)与 SLA(服务等级协议)

Source: SLA vs. SLO vs. SLI: What’s the difference?

Source: The Key Differences Between SLI, SLO, and SLA in SRE

归纳一下

  • SLA是法律文书而非技术文档
  • SLO是细分后的具体目标承诺
  • SLI是监控采集的实际观测值

归纳一下…

  • SLA是法律文书而非技术文档,重在严格约束而非技术实现
  • SLO是细分后的具体目标承诺,重在明确量化而非如何测量
  • SLI是监控采集的实际观测值,需要精辟选择合适指标

归纳一下……

  • SLA是法律文书而非技术文档,重在严格约束而非技术实现,难在协商
  • SLO是细分后的具体目标承诺,重在明确量化而非如何测量,难在提炼
  • SLI是监控采集的实际观测值,需要精辟选择合适指标,难在精准观测

范例 —— Web 服务器可用性和延迟

  • 考察Web服务器可用性。指标为收到的HTTP请求数和成功响应的HTTP请求数。若要了解Web服务器是否胜任用户需求,可以分析成功请求数与请求总数的比率。
    • 用成功请求数除以请求总数,可得一个比率。乘以100得到百分比。例如,Web服务器收到100个请求,成功响应了80个,则比率为0.8,即可用性为80%。
  • 另例,考察Web服务延迟相关的度量值。需观测不到10毫秒内完成的操作与操作总数的比率。
    • 若阈值时间内返回80个请求,请求总数100,则延迟满足率80%。

Source: 服务级别指标(SLI)和服务级别目标(SLO)示例

一般评价标准

问题:过度供应

Source: A View of Cloud Computing. CACM 2010

问题:性能干扰

Source: Enforcing performance isolation across virtual machines in Xen, Middleware '06

经典机制

  • 操作系统:I/O管理器
  • 虚拟化环境:虚拟机管理器
  • 存储系统:I/O调度模块

公平排队的悠久历史

网络领域很早就开始探索……

案例1:cgroup (Linux内核)

Source: Variations on fair I/O schedulers, 2008

案例2:libvirt (KVM, Xen, VMware, QEMU)

Source: IO Throttling in QEMU, KVM Forum 2011; OpenStack - Basic volume quality of service;

案例3:Object Storage (Ceph)

Source: mClock: handling throughput variability for hypervisor IO scheduling, OSDI'10;
Using Ceph RBD as a QEMU Storage;

决策过程

  • MAPE-loop steps
    • Monitoring QoS metrics
    • Analyzing divergence of current performance from goals and updating a performance model
    • Planning decisions about actions needed to achieve performance goals
    • Executing those actions

Source: Decision-Making Approaches for Performance QoS in Distributed Storage Systems: A Survey, TPDS 2019

问题描述

  • 目标
    • (ri,lir,liw)(r_i, l^{r}_i, l^{w}_i)
    • Average latency over time wfrlir+(1fr)liw{Average\ latency\ over\ time}\ w \leq f_r\cdot l^{r}_i + (1 - f_r)\cdot l^{w}_i
  • 方法
    • 静态:任务初始资源如何分配
      • 调整资源配置
    • 动态:负载、系统变化如何动态适应
      • 流量控制

方法1:控制论

方法2:约束优化

方法3:机器学习

实践环境

实验思路

  • 调节并发数,观察性能约束与提升的空间

    with ThreadPoolExecutor(max_workers=1) as executor:
        futures = [ executor.submit(access_obs for i in range(100) ]
    
  • 尝试反馈控制并发数

    s3bench ... -numClients=8 ...
    
  • Minio API 子系统节流实践

进一步思考:如何精确控制?

  • 波动性与突发性问题
    • 比例积分控制?
    • 机器学习序列预测?
    • ……

我们的工作

Source: PID Controllers Explained

参考文献

  1. Decision-Making Approaches for Performance QoS in Distributed Storage Systems: A Survey, TPDS 2019.
  2. Server consolidation techniques in virtualized data centers of cloud environments: A systematic literature review, SPE 2018.
  3. MittOS: Supporting Millisecond Tail Tolerance with Fast Rejecting SLO-Aware OS Interface, SOSP 2017
  4. Crystal: software-defined storage for multi-tenant object stores, FAST 2017
  5. Argon: Performance Insulation for Shared Storage Servers, FAST 2007.
  6. Façade: Virtual Storage Devices with Performance Guarantees, FAST 2003.

New data from Synergy Research Group shows that Q3 enterprise spending on cloud infrastructure services were almost $33 billion, up 33% from the third quarter of 2019.

Amazon and Microsoft continue to account for over half of the worldwide market, with Amazon market share remaining at its long-standing mark of around 33%, while Microsoft’s share was over 18%. Google, Alibaba and Tencent are all growing more rapidly than the overall market and are gaining market share.

- SaaS 软件即服务,云应用 - PaaS 平台即服务,云数据库 - IaaS 基础架构即服务,云主机

Software architects have borrowed lot of concepts from civil engineering. Multi-tenancy is one such concept.

计算机网络领域上很早就开展服务质量保障方向研究 (Virtual Clock, WFQ, PGPS, WF2Q, SCFQ, Leap Forward, SFQ and Latency-rate scheduling),主要起步于公平队列

1. Algorithms that allocate throughput or bandwidth in proportion to the specified weights of the clients. Provide proportional allocation of IO resources, such as Stonehenge SFQ(D), Argon, and Aqua. Many of these algorithms are variants of weighted fair queuing mechanisms (Virtual Clock, WFQ, PGPS, WF2Q, SCFQ, Leap Forward, SFQ and Latency-rate scheduling) proposed in the networking literature, adapted to handle various storage-specific concerns such as concurrency, minimizing seek delays and improving throughput. 2. Algorithms that provide support for latency-sensitive applications along with proportional sharing. These algorithms include SMART, BVT, pClock, Avatar and service curve based techniques. 3. Algorithms that support reservation along with proportional allocation. Such as Rialto, ESX memory management and other reservation based CPU scheduling methods.