27
Irina: Accelerating DNN Inference with Efficient Online Scheduling Xiaorui Wu, Hong Xu, Yi Wang

Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

  • Upload
    others

  • View
    6

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Irina: Accelerating DNN Inference with Efficient Online SchedulingXiaorui Wu, Hong Xu, Yi Wang

Page 2: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

DNN inference in real world

Traffic Monitoring Surveillance

AI Assistant Garbage Classification

2

Page 3: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

DNN online service pipelines

QueryGeneral

PreprocessDNN-based Inference

Output Postprocess

Client Client

Distributed DNN Inference

3

Page 4: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Distributed DNN inference serving

Applications Frontend

BackendClient

Backend

Client

Client

4

Page 5: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Distributed DNN inference serving

What’s the difference?

Applications Frontend

BackendClient

Backend

Client

Client

Client Perspective Cluster Perspective

• Two Perspectives

5

Page 6: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

DNN inference serving

What’s the difference?

• Cluster: Improve the throughput of cluster and reduce the total makespan

• Client: Reduce query its response time

6

Page 7: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Different client experience?• Execute task A and then execute task B• Execute task B and then execute task A

Application A

Application B

Frontend

Frontend

Task A

Task B

7

Page 8: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Different client experience?• Execute task A and then execute task B

Task A Task B

• Execute task B and then execute task A

32

Task ATask B

3 2

8

x x

x x

Page 9: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Existing DNN serving systems focus on offline schedulingE.g., TensorFlow Serving, Cliper, Nexus • Aim to improve the throughput of cluster • Don’t consider the client experience • Rely on offline scheduling policy, online serving is FIFO

9

Page 10: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Irina: efficient DNN online inference scheduling

•Minimizes the average job completion time (JCT)•Improves the client experience

Average JCT =

Pni=1(Ti � ti)⇥BiPn

i=1 Bi<latexit sha1_base64="APdgA7ZUB7s9bWugxdh/D1tfuQo=">AAACNXicbVBNS8NAEN34WetX1aOXxSLUgyURQS8FtRcRDwr9gqaGzXZTl242YXcilNA/5cX/4UkPHhTx6l9wU3uwrQ8GHu/NMDPPjwXXYNuv1tz8wuLScm4lv7q2vrFZ2Npu6ChRlNVpJCLV8olmgktWBw6CtWLFSOgL1vT71cxvPjCleSRrMIhZJyQ9yQNOCRjJK1yfG5f0mIuvqjVcwW6gCE1dnYReyivO8E7iUs3j+BCDxw+wCzxkGl94fDjZlCl5r1C0y/YIeJY4Y1JEY9x4hWe3G9EkZBKoIFq3HTuGTkoUcCrYMO8mmsWE9s2BbUMlMcs76ejrId43ShcHkTIlAY/UvxMpCbUehL7pDAnc62kvE//z2gkEp52UyzgBJunvoiARGCKcRYi7XDEKYmAIoYqbWzG9JyY3MEFnITjTL8+SxlHZscvO7XHx7GIcRw7toj1UQg46QWfoEt2gOqLoEb2gd/RhPVlv1qf19ds6Z41ndtAErO8f0UqpqA==</latexit><latexit sha1_base64="APdgA7ZUB7s9bWugxdh/D1tfuQo=">AAACNXicbVBNS8NAEN34WetX1aOXxSLUgyURQS8FtRcRDwr9gqaGzXZTl242YXcilNA/5cX/4UkPHhTx6l9wU3uwrQ8GHu/NMDPPjwXXYNuv1tz8wuLScm4lv7q2vrFZ2Npu6ChRlNVpJCLV8olmgktWBw6CtWLFSOgL1vT71cxvPjCleSRrMIhZJyQ9yQNOCRjJK1yfG5f0mIuvqjVcwW6gCE1dnYReyivO8E7iUs3j+BCDxw+wCzxkGl94fDjZlCl5r1C0y/YIeJY4Y1JEY9x4hWe3G9EkZBKoIFq3HTuGTkoUcCrYMO8mmsWE9s2BbUMlMcs76ejrId43ShcHkTIlAY/UvxMpCbUehL7pDAnc62kvE//z2gkEp52UyzgBJunvoiARGCKcRYi7XDEKYmAIoYqbWzG9JyY3MEFnITjTL8+SxlHZscvO7XHx7GIcRw7toj1UQg46QWfoEt2gOqLoEb2gd/RhPVlv1qf19ds6Z41ndtAErO8f0UqpqA==</latexit><latexit sha1_base64="APdgA7ZUB7s9bWugxdh/D1tfuQo=">AAACNXicbVBNS8NAEN34WetX1aOXxSLUgyURQS8FtRcRDwr9gqaGzXZTl242YXcilNA/5cX/4UkPHhTx6l9wU3uwrQ8GHu/NMDPPjwXXYNuv1tz8wuLScm4lv7q2vrFZ2Npu6ChRlNVpJCLV8olmgktWBw6CtWLFSOgL1vT71cxvPjCleSRrMIhZJyQ9yQNOCRjJK1yfG5f0mIuvqjVcwW6gCE1dnYReyivO8E7iUs3j+BCDxw+wCzxkGl94fDjZlCl5r1C0y/YIeJY4Y1JEY9x4hWe3G9EkZBKoIFq3HTuGTkoUcCrYMO8mmsWE9s2BbUMlMcs76ejrId43ShcHkTIlAY/UvxMpCbUehL7pDAnc62kvE//z2gkEp52UyzgBJunvoiARGCKcRYi7XDEKYmAIoYqbWzG9JyY3MEFnITjTL8+SxlHZscvO7XHx7GIcRw7toj1UQg46QWfoEt2gOqLoEb2gd/RhPVlv1qf19ds6Z41ndtAErO8f0UqpqA==</latexit><latexit sha1_base64="APdgA7ZUB7s9bWugxdh/D1tfuQo=">AAACNXicbVBNS8NAEN34WetX1aOXxSLUgyURQS8FtRcRDwr9gqaGzXZTl242YXcilNA/5cX/4UkPHhTx6l9wU3uwrQ8GHu/NMDPPjwXXYNuv1tz8wuLScm4lv7q2vrFZ2Npu6ChRlNVpJCLV8olmgktWBw6CtWLFSOgL1vT71cxvPjCleSRrMIhZJyQ9yQNOCRjJK1yfG5f0mIuvqjVcwW6gCE1dnYReyivO8E7iUs3j+BCDxw+wCzxkGl94fDjZlCl5r1C0y/YIeJY4Y1JEY9x4hWe3G9EkZBKoIFq3HTuGTkoUcCrYMO8mmsWE9s2BbUMlMcs76ejrId43ShcHkTIlAY/UvxMpCbUehL7pDAnc62kvE//z2gkEp52UyzgBJunvoiARGCKcRYi7XDEKYmAIoYqbWzG9JyY3MEFnITjTL8+SxlHZscvO7XHx7GIcRw7toj1UQg46QWfoEt2gOqLoEb2gd/RhPVlv1qf19ds6Z41ndtAErO8f0UqpqA==</latexit>

Batchsize or the number of clients

Task finish time Task arrive time

10

Page 11: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Dynamical scheduling

1.Task-level • Preemption scheduling: provide limited resource for

the task with the highest benefit • Stacking scheduling: run multiple tasks concurrently

2.Model-level • Batching scheduling: batch examples across tasks

11

Page 12: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Preemption scheduling

Long execution tasks occupies limited computing resource

• Ongoing task occupies limited resources • New task contains a large number of queries • Minimize average JCT • satisfy the latency SLOs

Model Batch Size Latency (ms)A 8 75

B 256 10

12

Page 13: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Preemption scheduling

App A

App B

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit>

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>1

<latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit>

75<latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit>

85<latexit sha1_base64="Tx/IxqbMVGTtOTaA8V3yt27ihgM=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEsceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f65+MyQ==</latexit><latexit sha1_base64="Tx/IxqbMVGTtOTaA8V3yt27ihgM=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEsceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f65+MyQ==</latexit><latexit sha1_base64="Tx/IxqbMVGTtOTaA8V3yt27ihgM=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEsceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f65+MyQ==</latexit><latexit sha1_base64="Tx/IxqbMVGTtOTaA8V3yt27ihgM=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEsceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f65+MyQ==</latexit>

74 ms

Task A

Task B

Task B arrives

Problems: Ongoing task occupies limited resource

JCT =75⇥ 8 + 84⇥ 256

8 + 256= 83.73

<latexit sha1_base64="cKnfoWXyAJ25q6OzOSnwJxGfJHQ=">AAACIHicbVDLSgMxFM3UV62vqks3wSIIQpnpw5lNodiNuKrQh9AZSibNtKGZB0lGKEM/xY2/4saFIrrTrzHTVtDWA4Fzz7mXm3vciFEhdf1Ty6ytb2xuZbdzO7t7+wf5w6OOCGOOSRuHLOR3LhKE0YC0JZWM3EWcIN9lpOuOG6nfvSdc0DBoyUlEHB8NA+pRjKSS+nnzptGq2R5HODGr0JbUJwJa8AJalZ+qVL2cJqmUEliDVrlolvv5gl7UZ4CrxFiQAlig2c9/2IMQxz4JJGZIiJ6hR9JJEJcUMzLN2bEgEcJjNCQ9RQOkVjvJ7MApPFPKAHohVy+QcKb+nkiQL8TEd1Wnj+RILHup+J/Xi6VnOQkNoliSAM8XeTGDMoRpWnBAOcGSTRRBmFP1V4hHSKUlVaY5FYKxfPIq6ZSKhl40biuF+tUijiw4AafgHBjABHVwDZqgDTB4AE/gBbxqj9qz9qa9z1sz2mLmGPyB9vUNiLmeKw==</latexit><latexit sha1_base64="cKnfoWXyAJ25q6OzOSnwJxGfJHQ=">AAACIHicbVDLSgMxFM3UV62vqks3wSIIQpnpw5lNodiNuKrQh9AZSibNtKGZB0lGKEM/xY2/4saFIrrTrzHTVtDWA4Fzz7mXm3vciFEhdf1Ty6ytb2xuZbdzO7t7+wf5w6OOCGOOSRuHLOR3LhKE0YC0JZWM3EWcIN9lpOuOG6nfvSdc0DBoyUlEHB8NA+pRjKSS+nnzptGq2R5HODGr0JbUJwJa8AJalZ+qVL2cJqmUEliDVrlolvv5gl7UZ4CrxFiQAlig2c9/2IMQxz4JJGZIiJ6hR9JJEJcUMzLN2bEgEcJjNCQ9RQOkVjvJ7MApPFPKAHohVy+QcKb+nkiQL8TEd1Wnj+RILHup+J/Xi6VnOQkNoliSAM8XeTGDMoRpWnBAOcGSTRRBmFP1V4hHSKUlVaY5FYKxfPIq6ZSKhl40biuF+tUijiw4AafgHBjABHVwDZqgDTB4AE/gBbxqj9qz9qa9z1sz2mLmGPyB9vUNiLmeKw==</latexit><latexit sha1_base64="cKnfoWXyAJ25q6OzOSnwJxGfJHQ=">AAACIHicbVDLSgMxFM3UV62vqks3wSIIQpnpw5lNodiNuKrQh9AZSibNtKGZB0lGKEM/xY2/4saFIrrTrzHTVtDWA4Fzz7mXm3vciFEhdf1Ty6ytb2xuZbdzO7t7+wf5w6OOCGOOSRuHLOR3LhKE0YC0JZWM3EWcIN9lpOuOG6nfvSdc0DBoyUlEHB8NA+pRjKSS+nnzptGq2R5HODGr0JbUJwJa8AJalZ+qVL2cJqmUEliDVrlolvv5gl7UZ4CrxFiQAlig2c9/2IMQxz4JJGZIiJ6hR9JJEJcUMzLN2bEgEcJjNCQ9RQOkVjvJ7MApPFPKAHohVy+QcKb+nkiQL8TEd1Wnj+RILHup+J/Xi6VnOQkNoliSAM8XeTGDMoRpWnBAOcGSTRRBmFP1V4hHSKUlVaY5FYKxfPIq6ZSKhl40biuF+tUijiw4AafgHBjABHVwDZqgDTB4AE/gBbxqj9qz9qa9z1sz2mLmGPyB9vUNiLmeKw==</latexit><latexit sha1_base64="cKnfoWXyAJ25q6OzOSnwJxGfJHQ=">AAACIHicbVDLSgMxFM3UV62vqks3wSIIQpnpw5lNodiNuKrQh9AZSibNtKGZB0lGKEM/xY2/4saFIrrTrzHTVtDWA4Fzz7mXm3vciFEhdf1Ty6ytb2xuZbdzO7t7+wf5w6OOCGOOSRuHLOR3LhKE0YC0JZWM3EWcIN9lpOuOG6nfvSdc0DBoyUlEHB8NA+pRjKSS+nnzptGq2R5HODGr0JbUJwJa8AJalZ+qVL2cJqmUEliDVrlolvv5gl7UZ4CrxFiQAlig2c9/2IMQxz4JJGZIiJ6hR9JJEJcUMzLN2bEgEcJjNCQ9RQOkVjvJ7MApPFPKAHohVy+QcKb+nkiQL8TEd1Wnj+RILHup+J/Xi6VnOQkNoliSAM8XeTGDMoRpWnBAOcGSTRRBmFP1V4hHSKUlVaY5FYKxfPIq6ZSKhl40biuF+tUijiw4AafgHBjABHVwDZqgDTB4AE/gBbxqj9qz9qa9z1sz2mLmGPyB9vUNiLmeKw==</latexit>

13

Page 14: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Preemption scheduling

App A

App B

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit>

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>1

<latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit>

11<latexit sha1_base64="KOtn8d5zaeQCBaweJc/zk38nMY8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5F0DJgYxnFxEByhL3NXrJkb+/YnRPCkX9gY6GIrf/Izn/jJrlCEx8MPN6bYWZemCpp0fe/vdLa+sbmVnm7srO7t39QPTxq2yQzXLR4ohLTCZkVSmrRQolKdFIjWBwq8RiOb2b+45MwVib6ASepCGI21DKSnKGT7intV2t+3Z+DrBJakBoUaParX71BwrNYaOSKWdulfopBzgxKrsS00susSBkfs6HoOqpZLGyQzy+dkjOnDEiUGFcayVz9PZGz2NpJHLrOmOHILnsz8T+vm2F0HeRSpxkKzReLokwRTMjsbTKQRnBUE0cYN9LdSviIGcbRhVNxIdDll1dJ+6JO/Tq9u6w1qkUcZTiBUzgHClfQgFtoQgs4RPAMr/Dmjb0X7937WLSWvGLmGP7A+/wB2uyMvg==</latexit><latexit sha1_base64="KOtn8d5zaeQCBaweJc/zk38nMY8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5F0DJgYxnFxEByhL3NXrJkb+/YnRPCkX9gY6GIrf/Izn/jJrlCEx8MPN6bYWZemCpp0fe/vdLa+sbmVnm7srO7t39QPTxq2yQzXLR4ohLTCZkVSmrRQolKdFIjWBwq8RiOb2b+45MwVib6ASepCGI21DKSnKGT7intV2t+3Z+DrBJakBoUaParX71BwrNYaOSKWdulfopBzgxKrsS00susSBkfs6HoOqpZLGyQzy+dkjOnDEiUGFcayVz9PZGz2NpJHLrOmOHILnsz8T+vm2F0HeRSpxkKzReLokwRTMjsbTKQRnBUE0cYN9LdSviIGcbRhVNxIdDll1dJ+6JO/Tq9u6w1qkUcZTiBUzgHClfQgFtoQgs4RPAMr/Dmjb0X7937WLSWvGLmGP7A+/wB2uyMvg==</latexit><latexit sha1_base64="KOtn8d5zaeQCBaweJc/zk38nMY8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5F0DJgYxnFxEByhL3NXrJkb+/YnRPCkX9gY6GIrf/Izn/jJrlCEx8MPN6bYWZemCpp0fe/vdLa+sbmVnm7srO7t39QPTxq2yQzXLR4ohLTCZkVSmrRQolKdFIjWBwq8RiOb2b+45MwVib6ASepCGI21DKSnKGT7intV2t+3Z+DrBJakBoUaParX71BwrNYaOSKWdulfopBzgxKrsS00susSBkfs6HoOqpZLGyQzy+dkjOnDEiUGFcayVz9PZGz2NpJHLrOmOHILnsz8T+vm2F0HeRSpxkKzReLokwRTMjsbTKQRnBUE0cYN9LdSviIGcbRhVNxIdDll1dJ+6JO/Tq9u6w1qkUcZTiBUzgHClfQgFtoQgs4RPAMr/Dmjb0X7937WLSWvGLmGP7A+/wB2uyMvg==</latexit><latexit sha1_base64="KOtn8d5zaeQCBaweJc/zk38nMY8=">AAAB6XicbVA9SwNBEJ2LXzF+RS1tFoNgFW5F0DJgYxnFxEByhL3NXrJkb+/YnRPCkX9gY6GIrf/Izn/jJrlCEx8MPN6bYWZemCpp0fe/vdLa+sbmVnm7srO7t39QPTxq2yQzXLR4ohLTCZkVSmrRQolKdFIjWBwq8RiOb2b+45MwVib6ASepCGI21DKSnKGT7intV2t+3Z+DrBJakBoUaParX71BwrNYaOSKWdulfopBzgxKrsS00susSBkfs6HoOqpZLGyQzy+dkjOnDEiUGFcayVz9PZGz2NpJHLrOmOHILnsz8T+vm2F0HeRSpxkKzReLokwRTMjsbTKQRnBUE0cYN9LdSviIGcbRhVNxIdDll1dJ+6JO/Tq9u6w1qkUcZTiBUzgHClfQgFtoQgs4RPAMr/Dmjb0X7937WLSWvGLmGP7A+/wB2uyMvg==</latexit> 86

<latexit sha1_base64="cKPQlfoaROl+Wg48zQWWJ/UU9cg=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEtMeCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f7SOMyg==</latexit><latexit sha1_base64="cKPQlfoaROl+Wg48zQWWJ/UU9cg=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEtMeCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f7SOMyg==</latexit><latexit sha1_base64="cKPQlfoaROl+Wg48zQWWJ/UU9cg=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEtMeCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f7SOMyg==</latexit><latexit sha1_base64="cKPQlfoaROl+Wg48zQWWJ/UU9cg=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEtMeCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0xUD3Iu08ygZMtFUSaIScj8bTLkCpkRU0soU9zeStiYKsqMDadiQ/BXX/5L2hc136v5d5fVhlvEUYYTOIVz8OEaGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f7SOMyg==</latexit>

86 ms

Task B

Task A

JCT =10⇥ 256 + 86⇥ 8

8 + 256= 12.30

<latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="s+l3pR4CppGG//EUdZ6i83Mljzo=">AAACFXicbZDLSgMxFIbP1FutVatbN0ERBKFkKrbdFIRuxFWF3qAzlEyaaUMzF5KMUIY+ihtfxY0LRXTn25heBG09EPjz/zmcnM+LBVca4y8rs7G5tb2T3c3t5fcPDgtH+baKEklZi0Yikl2PKCZ4yFqaa8G6sWQk8ATreOP6LO88MKl4FDb1JGZuQIYh9zkl2lj9QuWu3qw5viQ0tTFyNA+YQqXrMrpE1fLPvTpNq8Yw9hTVkF0qXuF+4QwX8bzQurCX4gyW1egXPp1BRJOAhZoKolTPxrF2UyI1p4JNc06iWEzomAxZz8iQmMFuOl9wis6NM0B+JM0JNZq7vztSEig1CTzzMiB6pFazmflf1ku0X3VTHsaJZiFdDPITgXSEZrTQgEtGtZgYQajk5q+IjoihpQ3TnIFgr668Ltqloo2L9j2GLJzAKVyADRW4gVtoQAsoPMIzvMKb9WS9WO8LXBlrye0Y/pT18Q2Z15yI</latexit><latexit sha1_base64="s+l3pR4CppGG//EUdZ6i83Mljzo=">AAACFXicbZDLSgMxFIbP1FutVatbN0ERBKFkKrbdFIRuxFWF3qAzlEyaaUMzF5KMUIY+ihtfxY0LRXTn25heBG09EPjz/zmcnM+LBVca4y8rs7G5tb2T3c3t5fcPDgtH+baKEklZi0Yikl2PKCZ4yFqaa8G6sWQk8ATreOP6LO88MKl4FDb1JGZuQIYh9zkl2lj9QuWu3qw5viQ0tTFyNA+YQqXrMrpE1fLPvTpNq8Yw9hTVkF0qXuF+4QwX8bzQurCX4gyW1egXPp1BRJOAhZoKolTPxrF2UyI1p4JNc06iWEzomAxZz8iQmMFuOl9wis6NM0B+JM0JNZq7vztSEig1CTzzMiB6pFazmflf1ku0X3VTHsaJZiFdDPITgXSEZrTQgEtGtZgYQajk5q+IjoihpQ3TnIFgr668Ltqloo2L9j2GLJzAKVyADRW4gVtoQAsoPMIzvMKb9WS9WO8LXBlrye0Y/pT18Q2Z15yI</latexit><latexit sha1_base64="CV7o8DnjGVV+wgaQXTn4gb+lCGo=">AAACIHicbVDLSgMxFM3UV62vUZdugkUQhJJUbGdTKHYjrir0BW0pmTTThmYeJBmhDP0UN/6KGxeK6E6/xvQhaOuBwLnn3MvNPW4kuNIIfVqptfWNza30dmZnd2//wD48aqgwlpTVaShC2XKJYoIHrK65FqwVSUZ8V7CmO6pM/eY9k4qHQU2PI9b1ySDgHqdEG6lnF28rtVLHk4QmGMGO5j5TMH9VgBfQKfzUziRxjGDkCSxBnM9dop6dRTk0A1wleEGyYIFqz/7o9EMa+yzQVBCl2hhFupsQqTkVbJLpxIpFhI7IgLUNDYhZ3E1mB07gmVH60AuleYGGM/X3REJ8pca+azp9oodq2ZuK/3ntWHtON+FBFGsW0PkiLxZQh3CaFuxzyagWY0MIldz8FdIhMWlpk2nGhICXT14ljXwOoxy+Q9ny9SKONDgBp+AcYFAEZXADqqAOKHgAT+AFvFqP1rP1Zr3PW1PWYuYY/IH19Q1fOZ4P</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit><latexit sha1_base64="MhKqQEHV8WeXzrc+jlYfe5ybvXI=">AAACIHicbVDLSsNAFJ3UV62vqEs3g0UQhJJUbbMpFLsRVxX6EJpQJtNJO3TyYGYilJBPceOvuHGhiO70a5y2EbT1wMC559zLnXvciFEhDeNTy62srq1v5DcLW9s7u3v6/kFHhDHHpI1DFvI7FwnCaEDakkpG7iJOkO8y0nXHjanfvSdc0DBoyUlEHB8NA+pRjKSS+nr1ptGq2R5HODENaEvqEwHLlxV4Bq3KT22liaUEJaewBs1y6dzo60WjZMwAl4mZkSLI0OzrH/YgxLFPAokZEqJnGpF0EsQlxYykBTsWJEJ4jIakp2iA1GInmR2YwhOlDKAXcvUCCWfq74kE+UJMfFd1+kiOxKI3Ff/zerH0LCehQRRLEuD5Ii9mUIZwmhYcUE6wZBNFEOZU/RXiEVJpSZVpQYVgLp68TDrlkmmUzNuLYv0qiyMPjsAxOAUmqII6uAZN0AYYPIAn8AJetUftWXvT3uetOS2bOQR/oH19A2B5nhM=</latexit>

Challenges: • How to enable preemption?• How to process the immediate results?

Save? Or Waste?

14

Page 15: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Stacking scheduling

Model Batch Size

Latency (ms)

Avg. GPU (%)

Peak Memory (%)/On-Peak Time (ms)

C 10 200 70% 65/100

D 100 10 28% 35/5

• Underutilization of the GPU resource• Concurrent tasks meet the GPU resource limitation• Framework supports to run multiple models in the same runtime

15

Page 16: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Stacking scheduling

App C

App D

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit> 200

<latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit>

210<latexit sha1_base64="N69bwp5mM3hsCvmxeX46bYih0is=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0mKoMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0PO+ndLG5tb2Tnm3srd/cHjkHp+0TZJpxlsskYnuhtRwKRRvoUDJu6nmNA4l74ST27nfeeLaiEQ94jTlQUxHSkSCUbTSQ933Bm7Vq3kLkHXiF6QKBZoD96s/TFgWc4VMUmN6vpdikFONgkk+q/Qzw1PKJnTEe5YqGnMT5ItTZ+TCKkMSJdqWQrJQf0/kNDZmGoe2M6Y4NqveXPzP62UY3QS5UGmGXLHloiiTBBMy/5sMheYM5dQSyrSwtxI2ppoytOlUbAj+6svrpF2v+V7Nv7+qNtwijjKcwTlcgg/X0IA7aEILGIzgGV7hzZHOi/PufCxbS04xcwp/4Hz+AEqJjPk=</latexit><latexit sha1_base64="N69bwp5mM3hsCvmxeX46bYih0is=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0mKoMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0PO+ndLG5tb2Tnm3srd/cHjkHp+0TZJpxlsskYnuhtRwKRRvoUDJu6nmNA4l74ST27nfeeLaiEQ94jTlQUxHSkSCUbTSQ933Bm7Vq3kLkHXiF6QKBZoD96s/TFgWc4VMUmN6vpdikFONgkk+q/Qzw1PKJnTEe5YqGnMT5ItTZ+TCKkMSJdqWQrJQf0/kNDZmGoe2M6Y4NqveXPzP62UY3QS5UGmGXLHloiiTBBMy/5sMheYM5dQSyrSwtxI2ppoytOlUbAj+6svrpF2v+V7Nv7+qNtwijjKcwTlcgg/X0IA7aEILGIzgGV7hzZHOi/PufCxbS04xcwp/4Hz+AEqJjPk=</latexit><latexit sha1_base64="N69bwp5mM3hsCvmxeX46bYih0is=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0mKoMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0PO+ndLG5tb2Tnm3srd/cHjkHp+0TZJpxlsskYnuhtRwKRRvoUDJu6nmNA4l74ST27nfeeLaiEQ94jTlQUxHSkSCUbTSQ933Bm7Vq3kLkHXiF6QKBZoD96s/TFgWc4VMUmN6vpdikFONgkk+q/Qzw1PKJnTEe5YqGnMT5ItTZ+TCKkMSJdqWQrJQf0/kNDZmGoe2M6Y4NqveXPzP62UY3QS5UGmGXLHloiiTBBMy/5sMheYM5dQSyrSwtxI2ppoytOlUbAj+6svrpF2v+V7Nv7+qNtwijjKcwTlcgg/X0IA7aEILGIzgGV7hzZHOi/PufCxbS04xcwp/4Hz+AEqJjPk=</latexit><latexit sha1_base64="N69bwp5mM3hsCvmxeX46bYih0is=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0mKoMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemEph0PO+ndLG5tb2Tnm3srd/cHjkHp+0TZJpxlsskYnuhtRwKRRvoUDJu6nmNA4l74ST27nfeeLaiEQ94jTlQUxHSkSCUbTSQ933Bm7Vq3kLkHXiF6QKBZoD96s/TFgWc4VMUmN6vpdikFONgkk+q/Qzw1PKJnTEe5YqGnMT5ItTZ+TCKkMSJdqWQrJQf0/kNDZmGoe2M6Y4NqveXPzP62UY3QS5UGmGXLHloiiTBBMy/5sMheYM5dQSyrSwtxI2ppoytOlUbAj+6svrpF2v+V7Nv7+qNtwijjKcwTlcgg/X0IA7aEILGIzgGV7hzZHOi/PufCxbS04xcwp/4Hz+AEqJjPk=</latexit>160

<latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit>

50 ms

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>

JCT =200⇥ 10 + 50⇥ 100

10 + 100= 63.64

<latexit sha1_base64="68ZXo2MDRuZ6SZ883RJruvvFQ60=">AAACI3icbVDLSgMxFM34rPVVdekmWARBGDK1VhEKxW7EVYW+oB1KJs20oZnMkGSEMvRf3PgrblwoxY0L/8X0saitBwLnnnMvN/d4EWdKI/Rtra1vbG5tp3bSu3v7B4eZo+O6CmNJaI2EPJRNDyvKmaA1zTSnzUhSHHicNrxBeeI3nqlULBRVPYyoG+CeYD4jWBupk7l7LFeLbV9ikuQQgm3NAqqgg+AlvF4o0SiZahMGi7BwZRfynUwW2WgKuEqcOcmCOSqdzLjdDUkcUKEJx0q1HBRpN8FSM8LpKN2OFY0wGeAebRkqsNntJtMbR/DcKF3oh9I8oeFUXZxIcKDUMPBMZ4B1Xy17E/E/rxVr/9ZNmIhiTQWZLfJjDnUIJ4HBLpOUaD40BBPJzF8h6WMTmDaxpk0IzvLJq6Sesx1kO0/5bOl+HkcKnIIzcAEccANK4AFUQA0Q8ALewAf4tF6td2tsfc1a16z5zAn4A+vnF6CTnqA=</latexit><latexit sha1_base64="68ZXo2MDRuZ6SZ883RJruvvFQ60=">AAACI3icbVDLSgMxFM34rPVVdekmWARBGDK1VhEKxW7EVYW+oB1KJs20oZnMkGSEMvRf3PgrblwoxY0L/8X0saitBwLnnnMvN/d4EWdKI/Rtra1vbG5tp3bSu3v7B4eZo+O6CmNJaI2EPJRNDyvKmaA1zTSnzUhSHHicNrxBeeI3nqlULBRVPYyoG+CeYD4jWBupk7l7LFeLbV9ikuQQgm3NAqqgg+AlvF4o0SiZahMGi7BwZRfynUwW2WgKuEqcOcmCOSqdzLjdDUkcUKEJx0q1HBRpN8FSM8LpKN2OFY0wGeAebRkqsNntJtMbR/DcKF3oh9I8oeFUXZxIcKDUMPBMZ4B1Xy17E/E/rxVr/9ZNmIhiTQWZLfJjDnUIJ4HBLpOUaD40BBPJzF8h6WMTmDaxpk0IzvLJq6Sesx1kO0/5bOl+HkcKnIIzcAEccANK4AFUQA0Q8ALewAf4tF6td2tsfc1a16z5zAn4A+vnF6CTnqA=</latexit><latexit sha1_base64="68ZXo2MDRuZ6SZ883RJruvvFQ60=">AAACI3icbVDLSgMxFM34rPVVdekmWARBGDK1VhEKxW7EVYW+oB1KJs20oZnMkGSEMvRf3PgrblwoxY0L/8X0saitBwLnnnMvN/d4EWdKI/Rtra1vbG5tp3bSu3v7B4eZo+O6CmNJaI2EPJRNDyvKmaA1zTSnzUhSHHicNrxBeeI3nqlULBRVPYyoG+CeYD4jWBupk7l7LFeLbV9ikuQQgm3NAqqgg+AlvF4o0SiZahMGi7BwZRfynUwW2WgKuEqcOcmCOSqdzLjdDUkcUKEJx0q1HBRpN8FSM8LpKN2OFY0wGeAebRkqsNntJtMbR/DcKF3oh9I8oeFUXZxIcKDUMPBMZ4B1Xy17E/E/rxVr/9ZNmIhiTQWZLfJjDnUIJ4HBLpOUaD40BBPJzF8h6WMTmDaxpk0IzvLJq6Sesx1kO0/5bOl+HkcKnIIzcAEccANK4AFUQA0Q8ALewAf4tF6td2tsfc1a16z5zAn4A+vnF6CTnqA=</latexit><latexit sha1_base64="68ZXo2MDRuZ6SZ883RJruvvFQ60=">AAACI3icbVDLSgMxFM34rPVVdekmWARBGDK1VhEKxW7EVYW+oB1KJs20oZnMkGSEMvRf3PgrblwoxY0L/8X0saitBwLnnnMvN/d4EWdKI/Rtra1vbG5tp3bSu3v7B4eZo+O6CmNJaI2EPJRNDyvKmaA1zTSnzUhSHHicNrxBeeI3nqlULBRVPYyoG+CeYD4jWBupk7l7LFeLbV9ikuQQgm3NAqqgg+AlvF4o0SiZahMGi7BwZRfynUwW2WgKuEqcOcmCOSqdzLjdDUkcUKEJx0q1HBRpN8FSM8LpKN2OFY0wGeAebRkqsNntJtMbR/DcKF3oh9I8oeFUXZxIcKDUMPBMZ4B1Xy17E/E/rxVr/9ZNmIhiTQWZLfJjDnUIJ4HBLpOUaD40BBPJzF8h6WMTmDaxpk0IzvLJq6Sesx1kO0/5bOl+HkcKnIIzcAEccANK4AFUQA0Q8ALewAf4tF6td2tsfc1a16z5zAn4A+vnF6CTnqA=</latexit>

Problem: Single task cannot saturate the GPU resource

Task D arrives

Task C

Task D

16

Page 17: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Stacking scheduling

Challenge: Interference among concurrent tasks

App C

App D

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit> 170

<latexit sha1_base64="fAxyjgacAInppTlCVpQ5ppA2QA4=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqMeCF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByO/c7T6g0T+SjmaYYxHQkecQZNVZ68OvewK16NW8Bsk78glShQHPgfvWHCctilIYJqnXP91IT5FQZzgTOKv1MY0rZhI6wZ6mkMeogX5w6IxdWGZIoUbakIQv190ROY62ncWg7Y2rGetWbi/95vcxEN0HOZZoZlGy5KMoEMQmZ/02GXCEzYmoJZYrbWwkbU0WZselUbAj+6svrpH1V872af39dbbhFHGU4g3O4BB/q0IA7aEILGIzgGV7hzRHOi/PufCxbS04xcwp/4Hz+AFIhjP4=</latexit><latexit sha1_base64="fAxyjgacAInppTlCVpQ5ppA2QA4=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqMeCF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByO/c7T6g0T+SjmaYYxHQkecQZNVZ68OvewK16NW8Bsk78glShQHPgfvWHCctilIYJqnXP91IT5FQZzgTOKv1MY0rZhI6wZ6mkMeogX5w6IxdWGZIoUbakIQv190ROY62ncWg7Y2rGetWbi/95vcxEN0HOZZoZlGy5KMoEMQmZ/02GXCEzYmoJZYrbWwkbU0WZselUbAj+6svrpH1V872af39dbbhFHGU4g3O4BB/q0IA7aEILGIzgGV7hzRHOi/PufCxbS04xcwp/4Hz+AFIhjP4=</latexit><latexit sha1_base64="fAxyjgacAInppTlCVpQ5ppA2QA4=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqMeCF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByO/c7T6g0T+SjmaYYxHQkecQZNVZ68OvewK16NW8Bsk78glShQHPgfvWHCctilIYJqnXP91IT5FQZzgTOKv1MY0rZhI6wZ6mkMeogX5w6IxdWGZIoUbakIQv190ROY62ncWg7Y2rGetWbi/95vcxEN0HOZZoZlGy5KMoEMQmZ/02GXCEzYmoJZYrbWwkbU0WZselUbAj+6svrpH1V872af39dbbhFHGU4g3O4BB/q0IA7aEILGIzgGV7hzRHOi/PufCxbS04xcwp/4Hz+AFIhjP4=</latexit><latexit sha1_base64="fAxyjgacAInppTlCVpQ5ppA2QA4=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqMeCF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByO/c7T6g0T+SjmaYYxHQkecQZNVZ68OvewK16NW8Bsk78glShQHPgfvWHCctilIYJqnXP91IT5FQZzgTOKv1MY0rZhI6wZ6mkMeogX5w6IxdWGZIoUbakIQv190ROY62ncWg7Y2rGetWbi/95vcxEN0HOZZoZlGy5KMoEMQmZ/02GXCEzYmoJZYrbWwkbU0WZselUbAj+6svrpH1V872af39dbbhFHGU4g3O4BB/q0IA7aEILGIzgGV7hzRHOi/PufCxbS04xcwp/4Hz+AFIhjP4=</latexit>

200<latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit><latexit sha1_base64="58jEU73CF/uCyvevmURGJ6SQNw4=">AAAB6nicbVBNSwMxEJ3Ur1q/Vj16CRbBU8kWQY8FLx4r2g9ol5JNs21oNrskWaEs/QlePCji1V/kzX9j2u5BWx8MPN6bYWZemEphLCHfqLSxubW9U96t7O0fHB55xydtk2Sa8RZLZKK7ITVcCsVbVljJu6nmNA4l74ST27nfeeLaiEQ92mnKg5iOlIgEo9ZJD3VCBl6V1MgCeJ34BalCgebA++oPE5bFXFkmqTE9n6Q2yKm2gkk+q/Qzw1PKJnTEe44qGnMT5ItTZ/jCKUMcJdqVsnih/p7IaWzMNA5dZ0zt2Kx6c/E/r5fZ6CbIhUozyxVbLooyiW2C53/jodCcWTl1hDIt3K2YjammzLp0Ki4Ef/XlddKu13xS8++vqg2viKMMZ3AOl+DDNTTgDprQAgYjeIZXeEMSvaB39LFsLaFi5hT+AH3+AEkEjPg=</latexit>

160<latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit><latexit sha1_base64="I66A3UnJzv34GC5cjLAEcv8qu/w=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE1GPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AVCcjP0=</latexit>

100<latexit sha1_base64="y2EgjsncA9k7hkOC7HzuJLaDezE=">AAAB6nicdVDLSgNBEOyNrxhfqx69DAbB0zKrK4m3gBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dQSq40hh/WKWV1bX1jfJmZWt7Z3fP3j9oqySTlLVoIhLZDYhigsespbkWrJtKRqJAsE4wuZr7nXsmFU/iOz1NmR+RUcxDTok20q2L8cCuYufy3PPqGGEHL2CI53kXXg25hVKFAs2B/d4fJjSLWKypIEr1XJxqPydScyrYrNLPFEsJnZAR6xkak4gpP1+cOkMnRhmiMJGmYo0W6veJnERKTaPAdEZEj9Vvby7+5fUyHdb9nMdppllMl4vCTCCdoPnfaMglo1pMDSFUcnMromMiCdUmnYoJ4etT9D9pnzkudtwbr9qwizjKcATHcAou1KAB19CEFlAYwQM8wbMlrEfrxXpdtpasYuYQfsB6+wSYkY0v</latexit><latexit sha1_base64="y2EgjsncA9k7hkOC7HzuJLaDezE=">AAAB6nicdVDLSgNBEOyNrxhfqx69DAbB0zKrK4m3gBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dQSq40hh/WKWV1bX1jfJmZWt7Z3fP3j9oqySTlLVoIhLZDYhigsespbkWrJtKRqJAsE4wuZr7nXsmFU/iOz1NmR+RUcxDTok20q2L8cCuYufy3PPqGGEHL2CI53kXXg25hVKFAs2B/d4fJjSLWKypIEr1XJxqPydScyrYrNLPFEsJnZAR6xkak4gpP1+cOkMnRhmiMJGmYo0W6veJnERKTaPAdEZEj9Vvby7+5fUyHdb9nMdppllMl4vCTCCdoPnfaMglo1pMDSFUcnMromMiCdUmnYoJ4etT9D9pnzkudtwbr9qwizjKcATHcAou1KAB19CEFlAYwQM8wbMlrEfrxXpdtpasYuYQfsB6+wSYkY0v</latexit><latexit sha1_base64="y2EgjsncA9k7hkOC7HzuJLaDezE=">AAAB6nicdVDLSgNBEOyNrxhfqx69DAbB0zKrK4m3gBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dQSq40hh/WKWV1bX1jfJmZWt7Z3fP3j9oqySTlLVoIhLZDYhigsespbkWrJtKRqJAsE4wuZr7nXsmFU/iOz1NmR+RUcxDTok20q2L8cCuYufy3PPqGGEHL2CI53kXXg25hVKFAs2B/d4fJjSLWKypIEr1XJxqPydScyrYrNLPFEsJnZAR6xkak4gpP1+cOkMnRhmiMJGmYo0W6veJnERKTaPAdEZEj9Vvby7+5fUyHdb9nMdppllMl4vCTCCdoPnfaMglo1pMDSFUcnMromMiCdUmnYoJ4etT9D9pnzkudtwbr9qwizjKcATHcAou1KAB19CEFlAYwQM8wbMlrEfrxXpdtpasYuYQfsB6+wSYkY0v</latexit><latexit sha1_base64="y2EgjsncA9k7hkOC7HzuJLaDezE=">AAAB6nicdVDLSgNBEOyNrxhfqx69DAbB0zKrK4m3gBePEc0DkiXMTmaTIbMPZmaFsOQTvHhQxKtf5M2/cZKsoKIFDUVVN91dQSq40hh/WKWV1bX1jfJmZWt7Z3fP3j9oqySTlLVoIhLZDYhigsespbkWrJtKRqJAsE4wuZr7nXsmFU/iOz1NmR+RUcxDTok20q2L8cCuYufy3PPqGGEHL2CI53kXXg25hVKFAs2B/d4fJjSLWKypIEr1XJxqPydScyrYrNLPFEsJnZAR6xkak4gpP1+cOkMnRhmiMJGmYo0W6veJnERKTaPAdEZEj9Vvby7+5fUyHdb9nMdppllMl4vCTCCdoPnfaMglo1pMDSFUcnMromMiCdUmnYoJ4etT9D9pnzkudtwbr9qwizjKcATHcAou1KAB19CEFlAYwQM8wbMlrEfrxXpdtpasYuYQfsB6+wSYkY0v</latexit>

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>

JCT =200⇥ 10 + 10⇥ 100

10 + 100= 27.3

<latexit sha1_base64="aovu+JClHxTL18Jc1FpEk+eWp+4=">AAACInicbVDLSgMxFM3UV62vUZdugkUQhCFTheqiUOxGXFXoC9pSMmmmDc08SDJCGeZb3Pgrblwo6krwY8y0g2jrgYSTc+7l5h4n5EwqhD6N3Mrq2vpGfrOwtb2zu2fuH7RkEAlCmyTggeg4WFLOfNpUTHHaCQXFnsNp25nUUr99T4Vkgd9Q05D2PTzymcsIVloamFe3tUal5wpM4hJCsKeYRyW0ETxLr58nSuJMQwmswFLZOh+YRWShGeAysTNSBBnqA/O9NwxI5FFfEY6l7NooVP0YC8UIp0mhF0kaYjLBI9rV1Md6dD+erZjAE60MoRsIfXwFZ+rvjhh7Uk49R1d6WI3lopeK/3ndSLmX/Zj5YaSoT+aD3IhDFcA0LzhkghLFp5pgIpj+KyRjrPNSOtWCDsFeXHmZtEqWjSz77qJYvc7iyIMjcAxOgQ3KoApuQB00AQEP4Am8gFfj0Xg23oyPeWnOyHoOwR8YX98Ru55b</latexit><latexit sha1_base64="aovu+JClHxTL18Jc1FpEk+eWp+4=">AAACInicbVDLSgMxFM3UV62vUZdugkUQhCFTheqiUOxGXFXoC9pSMmmmDc08SDJCGeZb3Pgrblwo6krwY8y0g2jrgYSTc+7l5h4n5EwqhD6N3Mrq2vpGfrOwtb2zu2fuH7RkEAlCmyTggeg4WFLOfNpUTHHaCQXFnsNp25nUUr99T4Vkgd9Q05D2PTzymcsIVloamFe3tUal5wpM4hJCsKeYRyW0ETxLr58nSuJMQwmswFLZOh+YRWShGeAysTNSBBnqA/O9NwxI5FFfEY6l7NooVP0YC8UIp0mhF0kaYjLBI9rV1Md6dD+erZjAE60MoRsIfXwFZ+rvjhh7Uk49R1d6WI3lopeK/3ndSLmX/Zj5YaSoT+aD3IhDFcA0LzhkghLFp5pgIpj+KyRjrPNSOtWCDsFeXHmZtEqWjSz77qJYvc7iyIMjcAxOgQ3KoApuQB00AQEP4Am8gFfj0Xg23oyPeWnOyHoOwR8YX98Ru55b</latexit><latexit sha1_base64="aovu+JClHxTL18Jc1FpEk+eWp+4=">AAACInicbVDLSgMxFM3UV62vUZdugkUQhCFTheqiUOxGXFXoC9pSMmmmDc08SDJCGeZb3Pgrblwo6krwY8y0g2jrgYSTc+7l5h4n5EwqhD6N3Mrq2vpGfrOwtb2zu2fuH7RkEAlCmyTggeg4WFLOfNpUTHHaCQXFnsNp25nUUr99T4Vkgd9Q05D2PTzymcsIVloamFe3tUal5wpM4hJCsKeYRyW0ETxLr58nSuJMQwmswFLZOh+YRWShGeAysTNSBBnqA/O9NwxI5FFfEY6l7NooVP0YC8UIp0mhF0kaYjLBI9rV1Md6dD+erZjAE60MoRsIfXwFZ+rvjhh7Uk49R1d6WI3lopeK/3ndSLmX/Zj5YaSoT+aD3IhDFcA0LzhkghLFp5pgIpj+KyRjrPNSOtWCDsFeXHmZtEqWjSz77qJYvc7iyIMjcAxOgQ3KoApuQB00AQEP4Am8gFfj0Xg23oyPeWnOyHoOwR8YX98Ru55b</latexit><latexit sha1_base64="aovu+JClHxTL18Jc1FpEk+eWp+4=">AAACInicbVDLSgMxFM3UV62vUZdugkUQhCFTheqiUOxGXFXoC9pSMmmmDc08SDJCGeZb3Pgrblwo6krwY8y0g2jrgYSTc+7l5h4n5EwqhD6N3Mrq2vpGfrOwtb2zu2fuH7RkEAlCmyTggeg4WFLOfNpUTHHaCQXFnsNp25nUUr99T4Vkgd9Q05D2PTzymcsIVloamFe3tUal5wpM4hJCsKeYRyW0ETxLr58nSuJMQwmswFLZOh+YRWShGeAysTNSBBnqA/O9NwxI5FFfEY6l7NooVP0YC8UIp0mhF0kaYjLBI9rV1Md6dD+erZjAE60MoRsIfXwFZ+rvjhh7Uk49R1d6WI3lopeK/3ndSLmX/Zj5YaSoT+aD3IhDFcA0LzhkghLFp5pgIpj+KyRjrPNSOtWCDsFeXHmZtEqWjSz77qJYvc7iyIMjcAxOgQ3KoApuQB00AQEP4Am8gFfj0Xg23oyPeWnOyHoOwR8YX98Ru55b</latexit>

Task D arrives

Task C

Task D

17

Page 18: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Batching scheduling•The same model in ongoing task and new task •Relatively low latency growth •Sufficient GPU computing resource

Model Batch Size Latency (ms)

A 4 60

A 8 75

A 16 85A 32 150

18

Page 19: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Batching scheduling

App A

App A

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit> 1

<latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit>

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodC coZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodC coZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodC coZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodC coZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>

149 ms

75<latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit>

150<latexit sha1_base64="hFpxb7z/P4/hu+pR6QrCJWD60xQ=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0WPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AU8XjPw=</latexit><latexit sha1_base64="hFpxb7z/P4/hu+pR6QrCJWD60xQ=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0WPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AU8XjPw=</latexit><latexit sha1_base64="hFpxb7z/P4/hu+pR6QrCJWD60xQ=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0WPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AU8XjPw=</latexit><latexit sha1_base64="hFpxb7z/P4/hu+pR6QrCJWD60xQ=">AAAB6nicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0WPBi8eK9gPaUDbbSbt0swm7G6GE/gQvHhTx6i/y5r9x2+agrQ8GHu/NMDMvTAXXxvO+ndLa+sbmVnm7srO7t3/gHh61dJIphk2WiER1QqpRcIlNw43ATqqQxqHAdji+nfntJ1SaJ/LRTFIMYjqUPOKMGis9+Fde3616NW8Oskr8glShQKPvfvUGCctilIYJqnXX91IT5FQZzgROK71MY0rZmA6xa6mkMeogn586JWdWGZAoUbakIXP190ROY60ncWg7Y2pGetmbif953cxEN0HOZZoZlGyxKMoEMQmZ/U0GXCEzYmIJZYrbWwkbUUWZselUbAj+8surpHVR872af39ZrbtFHGU4gVM4Bx+uoQ530IAmMBjCM7zCmyOcF+fd+Vi0lpxi5hj+wPn8AU8XjPw=</latexit>

Problem: Sequential execution cause underutilization of GPU resource

JCT =75⇥ 8 + 149⇥ 8

8 + 8= 112

<latexit sha1_base64="rnMPV/G/ea8E7F048x8zPpw94KQ=">AAACG3icbVDLSgMxFM3UV62vUZdugkUQhDIpldZFodiNuKrQF7RDyaSZNjTzIMkIZeh/uPFX3LhQxJXgwr8x0w6irQcC555zLzf3OCFnUlnWl5FZW9/Y3Mpu53Z29/YPzMOjtgwiQWiLBDwQXQdLyplPW4opTruhoNhzOO04k3rid+6pkCzwm2oaUtvDI5+5jGClpYFZvK03q31XYBKXL2FfMY9KWIEXEJWufspZnCiVGaxChIoDM28VrDngKkEpyYMUjYH50R8GJPKorwjHUvaQFSo7xkIxwuks148kDTGZ4BHtaepjvdSO57fN4JlWhtANhH6+gnP190SMPSmnnqM7PazGctlLxP+8XqTcih0zP4wU9clikRtxqAKYBAWHTFCi+FQTTATTf4VkjHVQSseZ0yGg5ZNXSbtYQFYB3ZXytes0jiw4AafgHCBQBjVwAxqgBQh4AE/gBbwaj8az8Wa8L1ozRjpzDP7A+PwG9Ouc8g==</latexit><latexit sha1_base64="rnMPV/G/ea8E7F048x8zPpw94KQ=">AAACG3icbVDLSgMxFM3UV62vUZdugkUQhDIpldZFodiNuKrQF7RDyaSZNjTzIMkIZeh/uPFX3LhQxJXgwr8x0w6irQcC555zLzf3OCFnUlnWl5FZW9/Y3Mpu53Z29/YPzMOjtgwiQWiLBDwQXQdLyplPW4opTruhoNhzOO04k3rid+6pkCzwm2oaUtvDI5+5jGClpYFZvK03q31XYBKXL2FfMY9KWIEXEJWufspZnCiVGaxChIoDM28VrDngKkEpyYMUjYH50R8GJPKorwjHUvaQFSo7xkIxwuks148kDTGZ4BHtaepjvdSO57fN4JlWhtANhH6+gnP190SMPSmnnqM7PazGctlLxP+8XqTcih0zP4wU9clikRtxqAKYBAWHTFCi+FQTTATTf4VkjHVQSseZ0yGg5ZNXSbtYQFYB3ZXytes0jiw4AafgHCBQBjVwAxqgBQh4AE/gBbwaj8az8Wa8L1ozRjpzDP7A+PwG9Ouc8g==</latexit><latexit sha1_base64="rnMPV/G/ea8E7F048x8zPpw94KQ=">AAACG3icbVDLSgMxFM3UV62vUZdugkUQhDIpldZFodiNuKrQF7RDyaSZNjTzIMkIZeh/uPFX3LhQxJXgwr8x0w6irQcC555zLzf3OCFnUlnWl5FZW9/Y3Mpu53Z29/YPzMOjtgwiQWiLBDwQXQdLyplPW4opTruhoNhzOO04k3rid+6pkCzwm2oaUtvDI5+5jGClpYFZvK03q31XYBKXL2FfMY9KWIEXEJWufspZnCiVGaxChIoDM28VrDngKkEpyYMUjYH50R8GJPKorwjHUvaQFSo7xkIxwuks148kDTGZ4BHtaepjvdSO57fN4JlWhtANhH6+gnP190SMPSmnnqM7PazGctlLxP+8XqTcih0zP4wU9clikRtxqAKYBAWHTFCi+FQTTATTf4VkjHVQSseZ0yGg5ZNXSbtYQFYB3ZXytes0jiw4AafgHCBQBjVwAxqgBQh4AE/gBbwaj8az8Wa8L1ozRjpzDP7A+PwG9Ouc8g==</latexit><latexit sha1_base64="rnMPV/G/ea8E7F048x8zPpw94KQ=">AAACG3icbVDLSgMxFM3UV62vUZdugkUQhDIpldZFodiNuKrQF7RDyaSZNjTzIMkIZeh/uPFX3LhQxJXgwr8x0w6irQcC555zLzf3OCFnUlnWl5FZW9/Y3Mpu53Z29/YPzMOjtgwiQWiLBDwQXQdLyplPW4opTruhoNhzOO04k3rid+6pkCzwm2oaUtvDI5+5jGClpYFZvK03q31XYBKXL2FfMY9KWIEXEJWufspZnCiVGaxChIoDM28VrDngKkEpyYMUjYH50R8GJPKorwjHUvaQFSo7xkIxwuks148kDTGZ4BHtaepjvdSO57fN4JlWhtANhH6+gnP190SMPSmnnqM7PazGctlLxP+8XqTcih0zP4wU9clikRtxqAKYBAWHTFCi+FQTTATTf4VkjHVQSseZ0yGg5ZNXSbtYQFYB3ZXytes0jiw4AafgHCBQBjVwAxqgBQh4AE/gBbwaj8az8Wa8L1ozRjpzDP7A+PwG9Ouc8g==</latexit>

19

Page 20: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Batching scheduling

App A

App A

0<latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit><latexit sha1_base64="uMaDhDIjFX99N65Rhs1b/J8iV9I=">AAAB6HicbVBNS8NAEJ3Ur1q/oh69LBbBU0lE0GPBi8cW7Ae0oWy2k3btZhN2N0IJ/QVePCji1Z/kzX/jts1BWx8MPN6bYWZemAqujed9O6WNza3tnfJuZW//4PDIPT5p6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqegO36tW8Bcg68QtShQKNgfvVHyYsi1EaJqjWPd9LTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90GOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7Eh+Ksvr5P2Vc33an7zulp3izjKcAbncAk+3EAd7qEBLWCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNqD4yC</latexit> 1

<latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit><latexit sha1_base64="axwJ+qFsqnc6XYbeiaj5UkOo0PI=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE0GPBi8cWTFtoQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8MBVcG9f9dkobm1vbO+Xdyt7+weFR9fikrZNMMfRZIhLVDalGwSX6hhuB3VQhjUOBnXByN/c7T6g0T+SDmaYYxHQkecQZNVZqeYNqza27C5B14hWkBgWag+pXf5iwLEZpmKBa9zw3NUFOleFM4KzSzzSmlE3oCHuWShqjDvLFoTNyYZUhiRJlSxqyUH9P5DTWehqHtjOmZqxXvbn4n9fLTHQb5FymmUHJlouiTBCTkPnXZMgVMiOmllCmuL2VsDFVlBmbTcWG4K2+vE7aV3XPrXut61qDFHGU4QzO4RI8uIEG3EMTfGCA8Ayv8OY8Oi/Ou/OxbC05xcwp/IHz+QNwY4yT</latexit>

t (ms)<latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit><latexit sha1_base64="pXOFJKbLTWgcjS98OqGEOv5OWJk=">AAAB73icbVBNS8NAEJ34WetX1aOXxSLUS0lE0GPBi8cK9gPaUDbbTbt0N4m7E6GE/gkvHhTx6t/x5r9x0+agrQ8GHu/NMDMvSKQw6Lrfztr6xubWdmmnvLu3f3BYOTpumzjVjLdYLGPdDajhUkS8hQIl7yaaUxVI3gkmt7nfeeLaiDh6wGnCfUVHkQgFo2ilLvZJTZmL8qBSdevuHGSVeAWpQoHmoPLVH8YsVTxCJqkxPc9N0M+oRsEkn5X7qeEJZRM64j1LI6q48bP5vTNybpUhCWNtK0IyV39PZFQZM1WB7VQUx2bZy8X/vF6K4Y2fiShJkUdssShMJcGY5M+TodCcoZxaQpkW9lbCxlRThjaiPARv+eVV0r6se27du7+qNipFHCU4hTOogQfX0IA7aEILGEh4hld4cx6dF+fd+Vi0rjnFzAn8gfP5A3YHjsM=</latexit>86

<latexit sha1_base64="4UrMnO6HcGqdYEyqWULesiP4oUQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEtMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jps1Bqw8GHu/NMDMvSKQw6LpfTmltfWNzq7xd2dnd2z+oHh51TJxqxtsslrHuBdRwKRRvo0DJe4nmNAok7wbTm9zvPnJtRKwecJZwP6JjJULBKFrpvnFVGVZrbt1dgPwlXkFqUKA1rH4ORjFLI66QSWpM33MT9DOqUTDJ55VBanhC2ZSOed9SRSNu/Gxx6pycWWVEwljbUkgW6s+JjEbGzKLAdkYUJ2bVy8X/vH6KYcPPhEpS5IotF4WpJBiT/G8yEpozlDNLKNPC3krYhGrK0KaTh+CtvvyXdC7qnlv37i5rzWoRRxlO4BTOwYNraMIttKANDMbwBC/w6kjn2Xlz3petJaeYOYZfcD6+ASGujN4=</latexit><latexit sha1_base64="4UrMnO6HcGqdYEyqWULesiP4oUQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEtMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jps1Bqw8GHu/NMDMvSKQw6LpfTmltfWNzq7xd2dnd2z+oHh51TJxqxtsslrHuBdRwKRRvo0DJe4nmNAok7wbTm9zvPnJtRKwecJZwP6JjJULBKFrpvnFVGVZrbt1dgPwlXkFqUKA1rH4ORjFLI66QSWpM33MT9DOqUTDJ55VBanhC2ZSOed9SRSNu/Gxx6pycWWVEwljbUkgW6s+JjEbGzKLAdkYUJ2bVy8X/vH6KYcPPhEpS5IotF4WpJBiT/G8yEpozlDNLKNPC3krYhGrK0KaTh+CtvvyXdC7qnlv37i5rzWoRRxlO4BTOwYNraMIttKANDMbwBC/w6kjn2Xlz3petJaeYOYZfcD6+ASGujN4=</latexit><latexit sha1_base64="4UrMnO6HcGqdYEyqWULesiP4oUQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEtMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jps1Bqw8GHu/NMDMvSKQw6LpfTmltfWNzq7xd2dnd2z+oHh51TJxqxtsslrHuBdRwKRRvo0DJe4nmNAok7wbTm9zvPnJtRKwecJZwP6JjJULBKFrpvnFVGVZrbt1dgPwlXkFqUKA1rH4ORjFLI66QSWpM33MT9DOqUTDJ55VBanhC2ZSOed9SRSNu/Gxx6pycWWVEwljbUkgW6s+JjEbGzKLAdkYUJ2bVy8X/vH6KYcPPhEpS5IotF4WpJBiT/G8yEpozlDNLKNPC3krYhGrK0KaTh+CtvvyXdC7qnlv37i5rzWoRRxlO4BTOwYNraMIttKANDMbwBC/w6kjn2Xlz3petJaeYOYZfcD6+ASGujN4=</latexit><latexit sha1_base64="4UrMnO6HcGqdYEyqWULesiP4oUQ=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEtMeCF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jps1Bqw8GHu/NMDMvSKQw6LpfTmltfWNzq7xd2dnd2z+oHh51TJxqxtsslrHuBdRwKRRvo0DJe4nmNAok7wbTm9zvPnJtRKwecJZwP6JjJULBKFrpvnFVGVZrbt1dgPwlXkFqUKA1rH4ORjFLI66QSWpM33MT9DOqUTDJ55VBanhC2ZSOed9SRSNu/Gxx6pycWWVEwljbUkgW6s+JjEbGzKLAdkYUJ2bVy8X/vH6KYcPPhEpS5IotF4WpJBiT/G8yEpozlDNLKNPC3krYhGrK0KaTh+CtvvyXdC7qnlv37i5rzWoRRxlO4BTOwYNraMIttKANDMbwBC/w6kjn2Xlz3petJaeYOYZfcD6+ASGujN4=</latexit>

86 ms

75<latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit><latexit sha1_base64="vOrMD9X0PLkOzWy2ZxzmwoXw/Ec=">AAAB6XicbVBNS8NAEJ3Ur1q/oh69LBbBU0lEqceCF49V7Ae0oWy2k3bpZhN2N0IJ/QdePCji1X/kzX/jts1Bqw8GHu/NMDMvTAXXxvO+nNLa+sbmVnm7srO7t3/gHh61dZIphi2WiER1Q6pRcIktw43AbqqQxqHATji5mfudR1SaJ/LBTFMMYjqSPOKMGivd168GbtWreQuQv8QvSBUKNAfuZ3+YsCxGaZigWvd8LzVBTpXhTOCs0s80ppRN6Ah7lkoaow7yxaUzcmaVIYkSZUsaslB/TuQ01noah7YzpmasV725+J/Xy0x0HeRcpplByZaLokwQk5D522TIFTIjppZQpri9lbAxVZQZG07FhuCvvvyXtC9qvlfz7y6rDbeIowwncArn4EMdGnALTWgBgwie4AVenYnz7Lw578vWklPMHMMvOB/f6hqMyA==</latexit>

JCT =86⇥ 8 + 85⇥ 8

8 + 8= 85.5

<latexit sha1_base64="K2RWRiyzYMr1Is/Ny337pk4IQp0=">AAACG3icbVDLSsNAFJ34rPUVdelmsAiCEJJiNZtCsRtxVaEvaEKZTCft0MmDmYlQQv7Djb/ixoUirgQX/o3TNoi2Hhg495x7uXOPFzMqpGl+aSura+sbm4Wt4vbO7t6+fnDYFlHCMWnhiEW86yFBGA1JS1LJSDfmBAUeIx1vXJ/6nXvCBY3CppzExA3QMKQ+xUgqqa+Xb+vNquNzhFP7EjqSBkRAG55Du/JTZelMyGBVqUalr5dMw5wBLhMrJyWQo9HXP5xBhJOAhBIzJETPMmPppohLihnJik4iSIzwGA1JT9EQqa1uOrstg6dKGUA/4uqFEs7U3xMpCoSYBJ7qDJAciUVvKv7n9RLp225KwziRJMTzRX7CoIzgNCg4oJxgySaKIMyp+ivEI6SCkirOogrBWjx5mbTLhmUa1t1FqXadx1EAx+AEnAELXIEauAEN0AIYPIAn8AJetUftWXvT3uetK1o+cwT+QPv8BgsTnP8=</latexit><latexit sha1_base64="K2RWRiyzYMr1Is/Ny337pk4IQp0=">AAACG3icbVDLSsNAFJ34rPUVdelmsAiCEJJiNZtCsRtxVaEvaEKZTCft0MmDmYlQQv7Djb/ixoUirgQX/o3TNoi2Hhg495x7uXOPFzMqpGl+aSura+sbm4Wt4vbO7t6+fnDYFlHCMWnhiEW86yFBGA1JS1LJSDfmBAUeIx1vXJ/6nXvCBY3CppzExA3QMKQ+xUgqqa+Xb+vNquNzhFP7EjqSBkRAG55Du/JTZelMyGBVqUalr5dMw5wBLhMrJyWQo9HXP5xBhJOAhBIzJETPMmPppohLihnJik4iSIzwGA1JT9EQqa1uOrstg6dKGUA/4uqFEs7U3xMpCoSYBJ7qDJAciUVvKv7n9RLp225KwziRJMTzRX7CoIzgNCg4oJxgySaKIMyp+ivEI6SCkirOogrBWjx5mbTLhmUa1t1FqXadx1EAx+AEnAELXIEauAEN0AIYPIAn8AJetUftWXvT3uetK1o+cwT+QPv8BgsTnP8=</latexit><latexit sha1_base64="K2RWRiyzYMr1Is/Ny337pk4IQp0=">AAACG3icbVDLSsNAFJ34rPUVdelmsAiCEJJiNZtCsRtxVaEvaEKZTCft0MmDmYlQQv7Djb/ixoUirgQX/o3TNoi2Hhg495x7uXOPFzMqpGl+aSura+sbm4Wt4vbO7t6+fnDYFlHCMWnhiEW86yFBGA1JS1LJSDfmBAUeIx1vXJ/6nXvCBY3CppzExA3QMKQ+xUgqqa+Xb+vNquNzhFP7EjqSBkRAG55Du/JTZelMyGBVqUalr5dMw5wBLhMrJyWQo9HXP5xBhJOAhBIzJETPMmPppohLihnJik4iSIzwGA1JT9EQqa1uOrstg6dKGUA/4uqFEs7U3xMpCoSYBJ7qDJAciUVvKv7n9RLp225KwziRJMTzRX7CoIzgNCg4oJxgySaKIMyp+ivEI6SCkirOogrBWjx5mbTLhmUa1t1FqXadx1EAx+AEnAELXIEauAEN0AIYPIAn8AJetUftWXvT3uetK1o+cwT+QPv8BgsTnP8=</latexit><latexit sha1_base64="K2RWRiyzYMr1Is/Ny337pk4IQp0=">AAACG3icbVDLSsNAFJ34rPUVdelmsAiCEJJiNZtCsRtxVaEvaEKZTCft0MmDmYlQQv7Djb/ixoUirgQX/o3TNoi2Hhg495x7uXOPFzMqpGl+aSura+sbm4Wt4vbO7t6+fnDYFlHCMWnhiEW86yFBGA1JS1LJSDfmBAUeIx1vXJ/6nXvCBY3CppzExA3QMKQ+xUgqqa+Xb+vNquNzhFP7EjqSBkRAG55Du/JTZelMyGBVqUalr5dMw5wBLhMrJyWQo9HXP5xBhJOAhBIzJETPMmPppohLihnJik4iSIzwGA1JT9EQqa1uOrstg6dKGUA/4uqFEs7U3xMpCoSYBJ7qDJAciUVvKv7n9RLp225KwziRJMTzRX7CoIzgNCg4oJxgySaKIMyp+ivEI6SCkirOogrBWjx5mbTLhmUa1t1FqXadx1EAx+AEnAELXIEauAEN0AIYPIAn8AJetUftWXvT3uetK1o+cwT+QPv8BgsTnP8=</latexit>

Challenge: How to process the immediate result?Challenges: • How to enable preemption?• How to process the immediate results?

20

Page 21: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Challenges:1.How to enable preemption? 2.How to process the immediate results? 3.Interference among concurrent GPU models

21

Page 22: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Enabling preemption:•Possible solutions:

1.Killing/Canceling the thread 2.Multiple process share the same GPU driver 3.Modify Exit op in computing graph

Layers

Exit Op

22

Page 23: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Immediate results•Possible solutions:

1.Swap in/out immediate results Save computing resource

2.Recomputing immediate results Reduce IO operations

Preemption Occurs

1

2

23

Page 24: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Interference among concurrent GPU models•Two solutions:

1.Stack long running tasks and short running tasks Reduce interference influence

2.Profiling concurrent running tasks Choose models with little interference

24

Page 25: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Real data trace simulation•Data trace is collected from testbed •Simulate 4 applications with varied workloads • Improve JCT by 1.3x-2.5x over Tensorflow Serving Scheduling

Model Name Batch Size Letency

YOLOv38 6816 117

VGG-168 22

16 33

GoogleNet 32 14

AlexNet 128 9

25

Page 26: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Conclusion

• Includes three dynamic scheduling policies • Enables preemption to support preemption scheduling and batching scheduling policies

• Analyzes interference across models and frameworks • Uses JCT as the target to schedule online inference workloads

Irina serves multiple applications with efficient online scheduling at the low job completion time

26

Page 27: Irina: Accelerating DNN Inference with Efficient Online ...conferences.sigcomm.org/events/apnet2020/material/... · Irina: Accelerating DNN Inference with Efficient Online Scheduling

Thank you