Media SDK Webinar 2014

Media SDK 1.1Mikhail Mironov

AMD Media SDK Software Architect

Oct 22, 2014

MEDIA SDK 1.1 AMF WEBINAR | OCTOBER 22, 20142

MEDIA SDK USE CASES

Video Conferencing Remote Desktop Video Playback

Cloud GamingVideo Editing/Transcode Wireless Display


MEDIA SDK – VALUE TO DEVELOPERS Use AMD’s differentiated multimedia IP

• SW IP : GPU optimized video pre/post processing • HW IP : VCE video encoder• HW IP: UVD – video decoder• HW IP : AMF-DEM library in latency sensitive applications like

wireless display and remote desktop

Optimal use of Heterogeneous Compute in Multimedia Apps• DX to OpenCL inter-op• SDK samples show developers how to build optimal video

processing pipelines

Developer Friendly• Support for industry standard MFT APIs• AMF - OS and framework agnostic C++ APIs • Next – VAAPI support for Linux media developers

Windows DesktopWindows Store

DirectXOpenCLOpenGL

Media Foundation


MFTs for leveraging Media HW accelerator blocks for video encoding/decoding

Video Quality MFT for video pre/post processing

AMF: C++ API for media processing

Support for Windows 7/8.1

MEDIA SDK SUMMARY


MEDIA FOUNDATION COMPONENTS (MFTS)

Access Video Compression Engine (VCE) for video encoding

Access Universal Video Decoder (UVD) for video decoding

Use GPU compute shaders for video processing

Released with Media SDK 1.0


AMD MEDIA FRAMEWORK (AMF) C++ API and framework for media processing on AMD platforms

OS agnostic : Initial support on Windows (Desktop and Windows Store) but can be easily ported to other OSes

Can be easily adapted to multimedia framework of choice – MFT, Gstreamer, OpenMAX …

Accept input from and provide inter-op with DirectX, OpenCL and OpenGL

Replacement for OpenVideo

Targeted for Media SDKv1.1


MEDIA SDK SOFTWARE STACK

Drivers

Applications

AMD HW (UVD, VCE, GPU)

MFT

DirectX

Frameworks

OpenCL OpenGL

OMX MFTISV

ISV

Native C/C++ISV

Decoder EncoderProcessorConverter

AMD Media Framework (AMF) (C++)

DEM Encoder

AMD

AMD

AMD

AMD

AMD AMD AMD

AMD


MEDIA SDK ROADMAP

Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2

2014

Media SDKv1.0: - Video Quality MFT

for video processing- AMF-DEM library

for low latency apps- Windows 8/8.1

Store Support

2015

Media SDK v1.0

GA

A10-7XXXA10-Micro 6XXX

Media SDKv1.1: - AMF for video - encoding- decoding- Cloud Gaming

/VDI support

Planned

In Planning

AMD Radeon™R9 285

Media SDKv2.0

Media SDK v1.1

Beta

Media SDK v1.1

GA

Media SDKv2.0: - Video Processor- Linux support- VAAPI & AMF- Open & closed

source Linux driver

Now

2016

Aug Dec

Media SDKv3.0: - Android support- Audio support- Camera support- File containers (mux,

demux)

Media SDKv3.0

AMD Radeon™R9 290

AMF architecture


AMF COMPONENTS Full Decoder:

‒ UVD - H264 AVC, H264 VC1, WMV9, MPEG2, MPEG4 P2, H264 MVC, MJPEG, H265 HEVC‒ Shader based MJPEG for legacy hardware

Encoder: VCE: H264 AVC, SVC Video Converter (Shader based)

‒ Color Space Conversion, Scale, Composition Video Processor (Shader based)

‒ Steady Video, Color Space Conversion, Scaling, Brightness, Contrast, Saturation, Tint, Vibrancy, Skin Tone Correction, Gamma, Brighter Whites, De-interlacing, Super Resolution, Dynamic Contrast, Dynamic Range, Edge Enhancement, De-noise, Mosquito Noise Reduction, False Contour Reduction, De-blocking, Demo Mode, FRC

AMF DEM Screen capture encoder (VCE)‒ Desktop only

Capability Manager


AMF ARCHITECTURE - BASICS Elementary data types

All elementary data types are defined to make code portable to other OSs. Examples:typedef __int64 amf_int64;typedef __int32 amf_int32;typedef size_t amf_size;typedef void* amf_handle;

C++ interfaces

Interfaces are implemented in the form of abstract (pure) C++ classes. Each interface is derived from the AMFInterface basic interface. It exposes methods for reference counting and querying for other interfaces :class AMFInterface{public:virtual amf_long AMF_STD_CALL Acquire()=0;virtual amf_long AMF_STD_CALL Release()=0;virtual AMF_RESULT AMF_STD_CALL QueryInterface(const AMFGuid& interfaceID, void** ppInterface) = 0;};


AMF ARCHITECTURE – INTERFACE DIAGRAM

AMFInterface

AMFPropertyStorage

AMFData AMFProperySorageEx

AMFBufferAMFSurface AMFComponent

AMFContext

AMFPlane

1

1..*

ComponentSurface


AMF ARCHITECTURE – PROPERTY STORAGE INTERFACE AMFPropertyStorage implements a property map between a string ID and an AMFVariantStruct structure. There

is a set of helper classes and template functions to safely manipulate AMFVariantStruct in C++. class AMFPropertyStorage : virtual public AMFInterface{public:virtual AMF_RESULT SetProperty(const wchar_t *pName, AMFVariantStruct value)=0;virtual AMF_RESULT GetProperty(const wchar_t *pName, AMFVariantStruct* pValue)=0;virtual bool HasProperty(const wchar_t *pName)=0;virtual amf_int32 GetPropertyCount() const = 0;virtual AMF_RESULT GetPropertyAt(amf_int32 index, wchar_t* name, amf_size nameSize, AMFVariantStruct* pValue) const = 0;virtual AMF_RESULT Clear()=0;virtual AMF_RESULT AddTo(AMFPropertyStorage *dest, bool bOverwrite, bool bDeep)=0;virtual AMF_RESULT CopyTo(AMFPropertyStorage *dest, bool bDeep)=0;template<typename _T> AMF_RESULT SetProperty(const wchar_t *pName, const _T& value);template<typename _T> AMF_RESULT GetProperty(const wchar_t *pName, _T* pValue);};


AMF ARCHITECTURE – PROPERTY STORAGE EX INTERFACEAMFPropertyStorageEx adds property description and validation features to AMFPropertyStorage.class AMFPropertyStorageEx : virtual public AMFPropertyStorage{public:virtual amf_size GetPropertiesInfoCount() = 0;virtual AMF_RESULT GetPropertyInfo(amf_size szInd,const AMFPropertyInfo** pInfo) = 0;virtual AMF_RESULT GetPropertyInfo(const wchar_t *pName,const AMFPropertyInfo** pInfo)=0;virtual AMF_RESULT ValidateProperty(const wchar_t *pName, AMFVariantStruct val, AMFVariantStruct* outValidated) = 0;};


AMF ARCHITECTURE - MEMORY Memory

AMF provides two types of memory objects: AMFBuffer and AMFSurface. Each object implements the AMFData common base interface. Each object is allocated via factory methods in the AMFContext interface. In addition, the AMFContext interface provides methods to create AMFBuffer or AMFSurface wrapping existing memory objects from various APIs:

• DX9 surfaces (Win Desktop only)• DX11 Textures (Win Store and Win Desktop)• OpenCL 2D images and buffers (Win Desktop only)• OpenGL textures (Win Desktop only)• Host memory

GetNative()

AMFPlane and AMFBuffer interfaces provide the GetNative() method. It allows to access the underlying memory object allocated by one of the technologies listed above. This method always return pointer to void. This call does not change the reference count of reference-counted objects (derived from the IUnknown interface or OCL’s memobj) Thread safety

All memory objects in AMF are not thread-safe (except AMF reference counting). It is assumed that any memory object is used by only one component at a time.


AMF ARCHITECTURE – DATA INTERFACEThe AMFData interface is a base interface for the AMBuffer and AMFSurface interfaces. It is responsible for storing common properties of the video or audio data objects (presentation time, duration, type, etc). The AMFData interface is also responsible for memory type storage and memory conversion: class AMFData : virtual public AMFPropertyStorage{public:virtual AMF_MEMORY_TYPE GetMemoryType()=0;virtual AMF_RESULT Duplicate(AMF_MEMORY_TYPE eMemType,AMFData **ppData)=0;virtual AMF_ERROR Convert(AMF_MEMORY_TYPE eMemType)=0;virtual AMF_DATA_TYPE GetDataType()=0;virtual void SetPts(amf_pts pts)=0;virtual amf_pts GetPts()=0;virtual void SetDuration(amf_pts duration)=0;virtual amf_pts GetDuration()=0;};


AMF ARCHITECTURE – BUFFER INTERFACEThe AMFBuffer interface provides access to an unordered memory buffer. class AMFBuffer : virtual public AMFData{public:virtual amf_size GetSize()=0;virtual void* GetNative()=0;virtual void AddObserver( AMFBufferObserver* pObserver) = 0;virtual void RemoveObserver( AMFBufferObserver* pObserver) = 0;};


AMF ARCHITECTURE – SURFACE INTERFACEThe AMFSurface interface represents a video frame and consists of a collection of planes. It also holds the type of the frame: progressive, interlaced, stereoscopic, etc. class AMFSurface : virtual public AMFData{public:virtual AMF_SURFACE_FORMAT GetFormat() = 0;virtual amf_size GetPlanesCount() = 0;virtual AMFPlane* GetPlaneAt(amf_size index) = 0;virtual AMFPlane* GetPlane(AMF_PLANE_TYPE type) = 0;virtual AMF_FRAME_TYPE GetFrameType() = 0;virtual void SetFrameType(AMF_FRAME_TYPE type) = 0;virtual void AddObserver(AMFSurfaceObserver* pObserver) = 0;virtual void RemoveObserver(AMFSurfaceObserver* pObserver) = 0;};


AMF ARCHITECTURE – PLANE INTERFACEThe AMFPlane interface represents a single two-dimensional plane of a multi-plane image format or the whole image for packed image formatclass AMFPlane : virtual public AMFInterface{public:virtual AMF_PLANE_TYPE GetType() = 0;virtual void* GetNative() = 0;virtual amf_int32 GetPixelSizeInBytes() = 0;virtual amf_int32 GetOffsetX() = 0;virtual amf_int32 GetOffsetY() = 0;virtual amf_int32 GetWidth() = 0;virtual amf_int32 GetHeight() = 0;virtual amf_int32 GetHPitch() = 0;virtual amf_int32 GetVPitch() = 0;};


AMF ARCHITECTURE – CONTEXT INTERFACE - DIRECTXThe AMFContext interface is responsible for initializing and storing device-specific data.class AMFContext : virtual public AMFPropertyStorage{virtual AMF_RESULT AMF_STD_CALL Terminate() = 0;

virtual AMF_RESULT InitDX9(void* pDX9Device) = 0;virtual void* GetDX9Device(AMF_DX_VERSION dxVersionRequired = AMF_DX9) = 0;virtual AMF_RESULT LockDX9() = 0;virtual AMF_RESULT UnlockDX9() = 0;

virtual AMF_RESULT InitDX11(void* pDX11Device, AMF_DX_VERSION dxVersionRequired = AMF_DX11_0) = 0;virtual void* GetDX11Device(AMF_DX_VERSION dxVersionRequired = AMF_DX11_0) = 0;virtual AMF_RESULT LockDX11() = 0;virtual AMF_RESULT UnlockDX11() = 0;


AMF ARCHITECTURE – CONTEXT INTERFACE – OCL, OGLContext interface is responsible for initializing and storing API-specific data (OGL devices, OCL contexts). virtual AMF_RESULT InitOpenCL(void* pCommandQueue = NULL) = 0;virtual void* GetOpenCLContext() = 0;virtual void* GetOpenCLCommandQueue() = 0;virtual void* GetOpenCLDeviceID() = 0;virtual AMF_RESULT LockOpenCL() = 0;virtual AMF_RESULT UnlockOpenCL() = 0;virtual AMF_RESULT InitOpenGL(amf_handle hOpenGLContext, amf_handle hWindow, amf_handle hDC) = 0;virtual amf_handle GetOpenGLContext() = 0;virtual amf_handle GetOpenGLDrawable() = 0;virtual AMF_RESULT LockOpenGL() = 0;virtual AMF_RESULT UnlockOpenGL() = 0;


AMF ARCHITECTURE – CONTEXT INTERFACE - ALLOCATION The AMFContext interface allows to allocate buffers and surfaces of a particular

memory type. The released objects are internally cached for future reuse.

virtual AMF_RESULT AllocBuffer(AMF_MEMORY_TYPE type, amf_size size, AMFBuffer** ppBuffer) = 0;virtual AMF_RESULT AllocSurface(AMF_MEMORY_TYPE type, AMF_SURFACE_FORMAT format, amf_int32 width, amf_int32 height, AMFSurface** ppSurface) = 0;


AMF ARCHITECTURE – CONTEXT INTERFACE - WRAPPING The AMFContext interface allows to temporarily wrap an external memory

object(DX, OpenCL, OpenGL or Host memory) into an AMF surface or buffer object. AMF allows to set an optional callback to track the release of AMF objects.

virtual AMF_RESULT CreateBufferFromHostNative(void* pHostBuffer, amf_size size, AMFBuffer** ppBuffer, AMFBufferObserver* pObserver) = 0;virtual AMF_RESULT CreateSurfaceFromHostNative(AMF_SURFACE_FORMAT format,amf_int32 width, amf_int32 height, amf_int32 hPitch, amf_int32 vPitch, void* pData, AMFSurface** ppSurface, AMFSurfaceObserver* pObserver) = 0;

virtual AMF_RESULT CreateSurfaceFromDX9Native(void* pDX9Surface, AMFSurface** ppSurface, AMFSurfaceObserver* pObserver) = 0;virtual AMF_RESULT CreateSurfaceFromDX11Native(void* pDX11Surface, AMFSurface** ppSurface, AMFSurfaceObserver* pObserver) = 0;

virtual AMF_RESULT CreateSurfaceFromOpenGLNative( AMF_SURFACE_FORMAT format, amf_handle hGLTextureID, AMFSurface** ppSurface, AMFSurfaceObserver* pObserver) = 0;};


AMF ARCHITECTURE – COMPONENT INTERFACEEach component implements a common AMFComponent interface. The AMFComponent interface is derived from AMFPropertyStorageEx and exposes a number of properties for the application to discover and manipulate.class AMFComponent : virtual public AMFPropertyStorageEx{public:virtual AMF_RESULT Init(AMF_SURFACE_FORMAT format,amf_int32 width,amf_int32 height) = 0;virtual AMF_RESULT ReInit(amf_int32 width,amf_int32 height) = 0;virtual AMF_RESULT Terminate() = 0;virtual AMF_RESULT Drain() = 0;virtual AMF_RESULT Flush() = 0;virtual AMF_RESULT SubmitInput(AMFData* pData) = 0;virtual AMF_RESULT QueryOutput(AMFData** ppData) = 0;virtual AMFContext* GetContext() = 0;virtual AMF_RESULT SetOutputDataAllocatorCB(AMFDataAllocatorCB* callback) = 0;};


AMF TYPICAL WORKFLOW

Create context

Create DX11 Device

Set device to context

Create Component

Set Properties on component

Initiate component

Allocate input buffer

Fill input buffer

Submit Input

Query Output

DrainQuery Output

Terminate Component

Terminate Context

Encoder Sample


Thread 1AMF ONE-THREAD MODEL

Prepare Input object

Submit input

Is EOF?

Drain

Terminate

Init

Query Output

Have Data?

Process Data

Is EOF?

Start

End Yes No

Yes

No

Yes

No


ENCODER SAMPLE - STATIC PARAMETERS#include <d3d9.h>#include <d3d11.h>#include "../../../include/components/VideoEncoderVCE.h"static amf::AMF_MEMORY_TYPE memoryTypeIn = amf::AMF_MEMORY_DX9;static amf::AMF_SURFACE_FORMAT formatIn = amf::AMF_SURFACE_NV12;static amf_int32 widthIn = 1920;static amf_int32 heightIn = 1080;static amf_int32 frameRateIn = 30;static amf_int64 bitRateIn = 1000000L; // in bits, 1MBstatic amf_int32 rectSize = 50;static amf_int32 frameCount = 100;static wchar_t *fileNameOut = L"./output.h264";static bool PollAndWrite(amf::AMFComponent *encoder, FILE *f, bool bDrain);static void FillSurface(amf::AMFContext *context, amf::AMFSurface *surface);

Init


ENCODER SAMPLE - INITIALIZATIONint _tmain(int argc, _TCHAR* argv[]){ AMF_RESULT res = AMF_OK; // error checking can be added later FILE *f = _wfopen(fileNameOut, L"wb"); // open output file // initialize AMF amf::AMFContextPtr context; res = AMFCreateContext(&context); if(memoryTypeIn == amf::AMF_MEMORY_DX9) { res = context->InitDX9(NULL); // can be DX9 or DX9Ex device } if(memoryTypeIn == amf::AMF_MEMORY_DX11) { res = context->InitDX11(NULL); // can be DX11 device }

Init


ENCODER SAMPLE - ENCODERamf::AMFComponentPtr encoder;res = AMFCreateComponent(context, AMFVideoEncoderVCE_AVC, &encoder);res = encoder->SetProperty(AMF_VIDEO_ENCODER_USAGE, AMF_VIDEO_ENCODER_USAGE_TRANSCONDING);res = encoder->SetProperty(AMF_VIDEO_ENCODER_TARGET_BITRATE, bitRateIn);res = encoder->SetProperty(AMF_VIDEO_ENCODER_FRAMESIZE, ::AMFConstructSize(widthIn, heightIn));res = encoder->SetProperty(AMF_VIDEO_ENCODER_FRAMERATE, ::AMFConstructRate(frameRateIn, 1));res = encoder->Init(formatIn, widthIn, heightIn);amf::AMFSurfacePtr surfaceIn;// create input surfaceres = context->AllocSurface(memoryTypeIn, formatIn, widthIn, heightIn, &surfaceIn);

Init


ENCODER SAMPLE - MAIN LOOPbool bFillSurface = true;amf_int32 submitted = 0;while(submitted < frameCount){ if(bFillSurface){ res = surfaceIn->Convert(memoryTypeIn); FillSurface(context, surfaceIn); } res = encoder->SubmitInput(surfaceIn); // encode if(res == AMF_INPUT_FULL){ // handle full queue bFillSurface = false; amf_sleep(1); // input queue is full: wait, poll and submit again }else{ bFillSurface = true; submitted++; } PollAndWrite(encoder, f, false); // poll and write all ready output}


Submit input


ENCODER SAMPLE - FILL SURFACE

static void FillSurface(amf::AMFContext *context, amf::AMFSurface *surface){HRESULT hr = S_OK;// no reference counting - do not Release()IDirect3DDevice9 *deviceDX9 = (IDirect3DDevice9 *)context->GetDX9Device(); IDirect3DSurface9 *surfaceDX9= (IDirect3DSurface9*)surface->GetPlaneAt(0)->GetNative(); hr = deviceDX9->ColorFill(surfaceDX9, NULL, D3DCOLOR_XYUV (128, 0, 128));}



ENCODER SAMPLE - POLLINGbool PollAndWrite(amf::AMFComponent *encoder, FILE *f, bool bDrain){ while(true){ // get all ready buffers amf::AMFDataPtr data; res = encoder->QueryOutput(&data); if(res == AMF_EOF){return false;} // Drain complete if(data != NULL){ amf::AMFBufferPtr buffer(data); fwrite(buffer->GetNative(), 1, buffer->GetSize(), f); }else{ if(bDrain){ amf_sleep(1);} else { break; } } } return true;}

Query Output

Process Data


ENCODE SAMPLE - DRAIN & TERMINATEwhile(true){// Drain res = encoder->Drain(); if(res != AMF_INPUT_FULL){ // handle full queue break; } amf_sleep(1); // input queue is full: wait and try again PollAndWrite(encoder, f, false);}PollAndWrite(encoder, f, true); // poll and write the rest of outputsurfaceIn = NULL;encoder->Terminate();encoder = NULL;context->Terminate();context = NULL; fclose(f);return 0;}

Drain

Terminate


Thread 2Polling

Thread 1Submit

AMF TWO-THREAD MODEL


Submit input

Is EOF?

Drain

Terminate

Init

Query Output

Have Data?

Process Data

Is EOF?

Thread Join

Thread 2 Start

YesYes

No

No

NoYes


Q&A, REFERENCES

AMD Media SDK website for downloads:

http://developer.amd.com/tools-and-sdks/media-sdk Latest driver download:

http://support.amd.com/en-us/download Media SDK Forum:

http://devgurus.amd.com/community/media-sdk Speaker: Mikhail Mironov, AMD Media SDK Software Architect:

[email protected] Developer Central Webinar Replays:

https://www.youtube.com/user/AMDDevCentral AMD Developer Central on Twitter (@AMDDevCentral):

https://twitter.com/AMDDevCentral Subscribe to be notified of future Webinars:

http://developer.amd.com/community/newsletter/subscribe/

http://developer.amd.com/tools-and-sdks/media-sdk/#downloads

http://developer.amd.com/tools-and-sdks/media-sdk/#downloads

http://support.amd.com/en-us/download

http://support.amd.com/en-us/download

http://devgurus.amd.com/community/media-sdk

http://devgurus.amd.com/community/media-sdk

mailto:[email protected]

https://www.youtube.com/user/AMDDevCentral

https://twitter.com/AMDDevCentral

http://developer.amd.com/community/newsletter/subscribe/


DISCLAIMER The information contained herein is for informational purposes only, and is subject to change without notice. While every precaution has been taken in the preparation of this document, it may contain technical inaccuracies, omissions and typographical errors, and AMD is under no obligation to update or otherwise correct this information. Advanced Micro Devices, Inc. makes no representations or warranties with respect to the accuracy or completeness of the contents of this document, and assumes no liability of any kind, including the implied warranties of noninfringement, merchantability or fitness for particular purposes, with respect to the operation or use of AMD hardware, software or other products described herein. No license, including implied or arising by estoppel, to any intellectual property rights is granted by this document. Terms and limitations applicable to the purchase or use of AMD’s products are as set forth in a signed agreement between the parties or in AMD's Standard Terms and Conditions of Sale.

AMD, the AMD Arrow logo, and combinations thereof are trademarks of Advanced Micro Devices, Inc. Other product names used in this publication are for identification purposes only and may be trademarks of their respective companies. USE OF THIS PRODUCT IN ANY MANNER THAT COMPLIES WITH THE MPEG ACTUAL OR DE FACTO VIDEO AND/OR AUDIO STANDARDS IS EXPRESSLY PROHIBITED WITHOUT ALL NECESSARY LICENSES UNDER APPLICABLE PATENTS. SUCH LICENSES MAY BE ACQUIRED FROM VARIOUS THIRD PARTIES INCLUDING, BUT NOT LIMITED TO, IN THE MPEG PATENT PORTFOLIO, WHICH LICENSE IS AVAILABLE FROM MPEG LA, L.L.C., 6312 S. FIDDLERS GREEN CIRCLE, SUITE 400E, GREENWOOD VILLAGE, COLORADO 80111.

Windows® is a registered trademark of Microsoft Corporation. DirectX ® is a registered trademark of Microsoft Corporation. OpenCL™ is a trademark of Apple Inc. used by permission by Khronos. Other names are for informational purposes only and may be trademarks of their respective owners.

© 2014 Advanced Micro Devices, Inc. All rights reserved.

Technology

Media SDK Webinar 2014