Download pdf - Sumit& archit osi nov-2011-displays-in-mobile-devices

1

LinuxDevelopment Centre

Displays in Mobile Devices

OSI Days 2011, Bangalore

Archit Taneja ([email protected])

Sumit Semwal ([email protected])

2


Agenda• Historical look at mobile devices

– Non-phone mobile devices• Types of display panels• Display Subsystems

– In PC World

– SoC World

– PC vs SoC

– Example Data Flow• Mobile Display Interface Standards• Interesting Display Use Cases• Example of SoC DSS:

– OMAP2+ Display Subsystem

– Current software design – DSS2 framework• Linux Kernel Display Frameworks• What Next?• Q & A


Under CC License from http://xkcd.com/732/

Nokia 1011First mass producedGSM phoneMonochrome [1]

One of the first color screen phonesSony Ericsson T68i101x80, 256color[2]

All trademarks and copyrights are properties of and attributed to their respective companiesSources: in references

Google Nexus OneAMOLED: 480x800, 3.7”

Google Galaxy NexusSuper AMOLED plus720x1280; 4.65”

Apple iPhone 4“Retina” display960 x 640; 3.5” [5]

LG Optimus 3D3D Stereoscopic LCD; 800x480; 4.3” [4]

Nokia 9000 Communicator: First 'business phone'Monochrome display, Wide640x200: [3]

A Historical Look at Mobile Displays


Device Positioning Screen size

Resolution Technology

Samsung Galaxy Note

Tablet Smartphone

5.3” 1280 x 800 HD sAMOLED

Dell PhoneTablet

5”7”10.1”

800 x 480800 x 4801366 x 768

TFT LCDCapacitativeTFT

Amazon Kindle 3

eBook Reader 6” 600 x 800 eInk, 16 level grayscale

B&N Nook eBook Reader 6” 800 x 600 eInk, 16 level grayscale

Archos PMPTabletTablet

7”8”10.1”

800 x 4801024 x 7681024 x 600

TFT

Apple iPad Tablet 9.7” 1024 x 768 IPS LED

Samsung Galaxy Tab

Tablet 7”8.9”10.1”

1024 x 6001280 x 8001280 x 800

TFT

'non-phone' Mobile Displays


Only an indicative list; certainly not exhaustive!



Types of Display Panels• Retina – Apple iPhone 4

– In Plane Switching (IPS) LCD– Highest density: 326 ppi (3.5” screen, 960x640)

• AMOLED (Active Matrix Organic LED)– No Backlight needed

• Higher contrast ratios, more power efficient.– Pen Tile arrangement – RGBG arrangement of sub-pixels, where G sub-

pixel is shared by 2 pixels. So the display has only 2 sub-pixels per real pixel.

• Super AMOLED– Same as AMOLED, but touchscreen is integrated onto the same glass, thus

increasing visibility on the screen.



Types of Display Panels• Super AMOLED Plus

– Integrated touchscreen, and no Pen tile arrangement – much brighter, thinner and more efficient.

– regular RGB, so 50% more subtiles, hence lesser resolution.

• eInk Panel– Based on actual electronic ink that is coated on some ultra-thin plastic.– eInk contains microcapsules filled with ink, and pigmented, charged chips.– Each microcapsule acts as a pixel; when an electric charge is applied, either

the light or dark pigmented chip will 'come on top', thus 'switching' the pixel on or off.


Display Subsystems: PC World

GPU Display I/F

Display(s)

HDMI

Display Port

LVDS

Analog Video

Video Decoders

Display Port

2D/3D Accelerators

Overlay /Blending

Scene informationor

Encoded frames

Interconnect

Shared/Dedicated Memory

Pixel Gen 1

Pixel Gen 2

Pixel Gen M

Display Subsystem


Generic SoC Hardware Blocks

interconnect

MPU GPUVideo

AcceleratorDisplay Sub

System

DSP Imaging OtherPeripherals

EMIF

Display(s)

MemoryOnly display related blocks are shown; there are many other blocks too, which are out of scope for this discussion.


Display Subsystem in SoCs

Plane 1

Plane 2

Plane 3

Plane N

Compositor 1 &

Pixel Gen

Serial Interface

(MIPI DSI)

HDMI/Display Port

GPU composed framesor

Decoded Video frames

Interconnect

Overlay

Parallel Interface

(MIPI DPI)

Remote framebuffer

Interface(MIPI DBI)

Display I/F

Display(s)

Display Subsystem

Compositor 2 &

Pixel Gen

Compositor M &

Pixel Gen


Display Subsystems: PC vs SoC• Differences:

– PC: GPUs are generally responsible for composing different windows/layers to get one single frame. Display Subsystem only fetches and display this composed frame .

– SoC: Display Subsystem is 'also' responsible for composing layers generated by the GPU and Video accelerators.

– PC: Display Subsystem may only fetch frame data from the dedicated memory for the graphics card/controller.

– SoC: Display Subsystem fetches frame data from the main memory connected to the SoC.

• Reasons:– Legacy: GPUs and Video Accelerators have existed in SoCs as separate entities for a while. Making it

important for the Display Subsystems to compose the output of each into one frame.

– Area: SoCs (and mobile devices) can't afford dedicated memory for GPUs.

– IP reuse: Different Vendors of GPUs and Display Subsytems in SoCs, easier to interface with a standard interconnect.

– Power: Overlaying different layers is more optimal for SoCs in some cases.


Socinterconnect

MPU GPU Video Accelerator

Display SubSystem

Display

Memory

Only display related blocks are shown; there are many other blocks too, which are out of scope for this discussion.

Example data flow

1. Commands to draw UI

1. Video frames to decode

2.Done

2. Done

2.2.

3. Display frame on panel

3.

Data path

Control path


MIPI- Display Working Group

Primary (fixed) panels: - Display Serial Interface (DSI):

- serializes all pixel data, commands, and events.- low power(SLVS)- Works with Smart / Dumb Panels.

- Display Parallel Interface (DPI):- Legacy parallel interface.- Lots of lanes needed for pixel data and control events.- used with Dumb Panels.

- Display Bus Interface (DBI):- Parallel Interface for smart panels.

Pluggable panels:- HDMI- Display Port- SDTV / Composite- DVI


Mobile Display Interface Standards

14


Interesting Display Use cases• Device Dock

• Dynamic configurability, parallel usage– Switch output paths, using multiple paths simultaneously

• Handle audio interfacing additionally– HDMI support

– DSI support in future

• Support for 'virtual' pipelines – when we run out of video pipelines– Co-use of 2D/3D graphics accelerator.

• Multi-display configurations– Clone, Virtual.

● Display security

15


OMAP2+ DSS hardware: Overview

RFBI

DSI2 encoder

VENC HDMI

RFBI

SDTV

SecondaryDSI

Primary DSI

DisplayPanels

DPI

GFX

VID1

VID2

VID3

DSI1 encoder

DPI

HDMI encoder

TVmgr

LCD1mgr

TVmgr

LCD2mgr

DISPC

DisplayInterfaces

DSS hardware

OMAP2,3,4OMAP2,3,4OMAP3,4OMAP3,4OMAP4OMAP4

● Gfx, vid1-vid3 are overlays, which color-convert, scale each frame

● Managers can compose, overlay multiple streams into one.

●Not all paths are shown: All overlays connect to all overlay managers.

16


Current software design – DSS2 framework

UI / GraphicsUserspace appUI / Graphics

Userspace appVideo playbackUserspace appVideo playbackUserspace app

sysfs/dev/fb0/dev/fb1

/dev/video1/dev/video2

Framebuffer APIs V4L2 APIs

Control

omapfb driver omap_vout driver

DSS2 DSS2 gfx vid1 vid2 vid3

lcd1-mgr lcd2-mgr tv-mgr

HDMI TV LCD panelsDisplayhardware

Kernel space

User space

OMAP DSS hardware

Paneldrivers


Linux Kernel Display Frameworks• Frame Buffer

– Oldest of them all - first public release in 1999.

– Linux Kernel Abstraction for graphics hardware – represents framebuffer of some video hardware, so the software doesn't need to know the low-level (hardware) stuff.

– allows emulate text console without hardware text-mode support.

– is a possible graphic output method for X server; independent of video adapter hardware and drivers.

– allows 'direct' graphics programs without X; popular in embedded world [GTK+, Qt Extended are some examples]

– No inherent window or buffer management

• DirectFB [6]

– Specifically designed for embedded systems.

– Is a thin library providing hardware graphics acceleration.

– Integrated windowing system with support for translucent windows and multiple display layers.

– Input device handling.

– Is a complete hardware abstraction layer, with software fallbacks to every graphics operation not supported by an underlying hardware.

– Provides for X-less graphical systems: gtk+, LinuxTV, HP WebOS to name a few.


Linux Kernel Display Frameworks• Video For Linux (V4L)

– Kernel interface for streaming capture and output drivers.

– Provides framework for buffer management and streaming – very relevant for streaming kind of use-cases [video record, playback, TV, even analog radio! :)]

– Buffers are tied to each device, and allocated before streaming starts.

• Direct Rendering Manager (DRM)– The modern Linux (and *BSD) display driver framework

– The kernel component of DRI (Direct Rendering Infrastructure)

– But not just DRI:• Hotplug• XRandR/KMS• EDID parsing

– Kernel Mode Setting (KMS)• Setting screen resolution/timings• Hotplug, and retrieving EDID• Multi-display configurations

– Buffers are independent of overlays

– Security inherent


What Next?

- HDMI 3D

- eInk Color (Triton) [7]

20


Q and A?


References3430 public Technical Reference Manual

4430 public Technical Reference Manual

[1]: http://en.wikipedia.org/wiki/List_of_Nokia_products

[2]: http://www.mobiles.co.uk/mobile-phone-history-uk.html

[3]: http://royal.pingdom.com/2010/02/08/15-fantastic-firsts-on-the-internet/

[4]: http://www.lg.com/in/mobile-phones/all-phones/LG-smart-phones-P920.jsp

[5]: http://gigaom.com

[6]: http://directfb.org

[7]: http://www.eink.com/display_products_triton.html

http://focus.ti.com/pdfs/wtbu/OMAP34xx_ES3.1x_PUBLIC_TRM_vZN.pdf

http://focus.ti.com/pdfs/wtbu/OMAP4430_ES2.x_PUBLIC_TRM_vR.zip

http://en.wikipedia.org/wiki/List_of_Nokia_products

http://www.mobiles.co.uk/mobile-phone-history-uk.html

http://royal.pingdom.com/2010/02/08/15-fantastic-firsts-on-the-internet/

http://www.lg.com/in/mobile-phones/all-phones/LG-smart-phones-P920.jsp

http://gigaom.com/

http://directfb.org/

http://www.eink.com/display_products_triton.html

22


Thank You!

23


Backup

24


Acronyms• SoC: System on Chip

• DSI: Display Serial Interface: a MIPI standard interface for serial display panels.

• DPI: Display Pixel Interface: a MIPI standard interface for parallel display panels.

• RFBI: Remote Frame Buffer Interface: a MIPI standard interface for parallel display panels supporting remote frame buffer mechanism.

• HDMI: High Definition Multimedia Interface: a standard compact AV interface to transmit uncompressed digital data.

• VENC: Video ENCoder: standard interface to display content on standard definition TVs, using TV formats like NTSC / PAL.

25


Features of OMAP2+ DSS

OMAP3/4 support ARGB color formats.OMAP4 additionally has native support for NV12, configurable Z-order.

Overlay perspective

GFX overlay supports CLUT/gamma table, anti-flicker, replication logic – primarily meant for UI.

VID overlays support Multiple color formats(RGB, YUV, NV12 based on OMAP version), colorspace conversion, independent horizontal and vertical polyphase filter scaling, and VC-1 range mapping.

WB overlay, in mem-to-mem mode: colorspace conversion, rescaling, compositing processes.Capture mode: allows composited encoded pixel data to be stored back into memory.

Overlay perspectiveEach overlay supports: overlaying [fixed-order for OMAP2/3, z-order per pipeline for OMAP4], transparency color-keys [source and destination] and global and pixel alpha blending [OMAP3/4], Gamma correction, Sync buffers.

LCD overlays additionally support Active / Passive matrix dithering, color phase rotation.

* Read more about Virtual Rotated Frame Buffer (VRFB) and TILER in the 3430 public TRM and 4430 public TRM respectively

Rotation(per overlay)

Via VRFB* Via VRFB* Via TILER*

Feature OMAP2430 OMAP3430 OMAP4430

GFX overlay 1 1 1

VID overlay 2 2 3

WB overlay - - 1

LCD overlay manager

1 1 2

TV overlay manager

1 1 1

http://focus.ti.com/pdfs/wtbu/OMAP34xx_ES3.1x_PUBLIC_TRM_vZN.pdf

http://focus.ti.com/pdfs/wtbu/OMAP4430_ES2.x_PUBLIC_TRM_vR.zip