OpenGL ES on AndroidMinho Kim

Dept. of Computer Science

University of Seoul

Resources

• Short developer guide by Androidhttps://developer.android.com/guide/topics/graphics/opengl.html

• Tutorial by Android (Android framework) https://developer.android.com/training/graphics/opengl/index.html

• “OpenGL ES 2.0 Programming Guide” (2009) http://opengles-book.com/(requires an SVN client to download samples)

• Learn OpenGL ES by Kevin Brothaler (GLES ver. 2.0) http://www.learnopengles.com/archives/

• “OpenGL ES 2 for Android” by Kevin Brothaler (2013)

• OpenGL ES 2.0 course (spring, 2013) @Univ. of Seoul http://www.minho-kim.com/courses/13sp43.527/

OpenGL ES SDK

• ARM Mali http://malideveloper.arm.com/develop-for-mali/sdks/opengl-es-2-sdk-for-android/

• PowerVR SGX http://www.imgtec.com/powervr/insider/sdk/KhronosOpenGLES2xSGX.asp

• NVIDIA Tegra https://developer.nvidia.com/tegra-android-development-pack

• Qualcom Adreno https://developer.qualcomm.com/mobile-development/mobile-technologies/gaming-graphics-optimization-adreno/tools-and-resources

Environment Setup

• Install ADT bundle https://developer.android.com/sdk/index.html• JRE is required to execute Eclipse

• Samples can be installed using SDK Manager https://developer.android.com/tools/samples/index.html

• Install USB driver

GLSL Comparison

GLSL4.3 GLSL3.3 GLSL1.5 GLSL ES 3.0 GLSL ES 1.0Tess shader o o x x xGeo Shader o o o x xVAO o o o o xStorage Qualifiers (in, out) o o x o xUniform block o o x o xDirectly Accessing Buffer Data(glMapBuffer)

o o x o x

Instanced Rendering o o x o xMultisample Fragment Operations(glSampleCoverage)

o o x o x

Transform feedback o o x o x

OpenGL ES 2.0 on Android

• Supported through Android framework API (Java) and NDK (C)

• OpenGL API class – android.opengl.GLES20

OpenGL ES Development

• Required to create a view container by implementing both GLSurfaceViewand GLSurfaceView.Renderer

• GLSurfaceView• a view container for graphics drawn with OpenGL• needs to be extended to implement the touch listener to capture touch screen

events

• GLSurfaceView.Renderer• an interface that controls what is drawn within that view• Three methods are required to be implemented

• onSurfaceCreated() ≈ init()• onDrawFrame() – display()• onSurfaceChanged() – reshape()

Minimal OpenGL ES App (Java)

Minimal OpenGL ES 2.0 App from Scratch

• Create an empty Android App Project with “Create custom launcher icon” and “Create Activity” checkbox unchecked

• Minimum required SDK: 10 (Gingerbread 2.3.3)

• Create folder such as “src/com/example/android/opengl” (New Folder)

• Refresh the project

• Implement Activity, GLSurfaceView, and GLSurfaceView.Renderer as in the following slides• Implementing your GLSurfaceView is optional if no UI is required• Use “Ctrl-Shit-O” for automatic imports

• Click AndroidManifest.xml and select “Application” tab

• Add an “Activity” to “Application Nodes” and select the Activity you implemented as its “Name” (e.g. “com.example.android.opengl.MyGLActivity”)

• Add an “Intent Filter” to the Activity you added

• Add an “Action” to the Intent Filter you added with its name “android.intent.action.MAIN”

• Add an “Category” to the Intent Filter you added with its name “android.intent.category.LAUNCHER”

Implementing Activity

• New Class• Package – browse the folder (e.g. com.example.android.opengl)

• Name – choose yours (e.g. MyActivity)

• Superclass – android.app.Activity

package com.example.android.opengl;

import android.app.Activity;import android.opengl.GLSurfaceView;import android.os.Bundle;

public class MyActivity extends Activity {private GLSurfaceView mGLSurfaceView;@Override protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);mGLSurfaceView = new GLSurfaceView(this);mGLSurfaceView.setEGLContextClientVersion(2);mGLSurfaceView.setRenderer(new MyGLRenderer(this));setContentView(mGLSurfaceView);

}@Override protected void onResume() {

super.onResume();mGLSurfaceView.onResume();

}@Override protected void onPause() {

super.onPause();mGLSurfaceView.onPause();

}}

MyActivity.java

Implementing GLSurfaceView.Renderer

• New Class• Package – browse the folder (e.g. com.example.android.opengl)

• Name – choose yours (e.g. “MyGLRenderer”)

• Superclass – none

• Add interface “Renderer”

package com.example.android.opengl;import java.nio.ByteBuffer;import java.nio.ByteOrder;import java.nio.FloatBuffer;

import javax.microedition.khronos.egl.EGLConfig;import javax.microedition.khronos.opengles.GL10;

import android.opengl.GLES20;import android.opengl.GLSurfaceView.Renderer;import android.content.Context;

public class MyGLRenderer implements Renderer {private int mProgramObject;private FloatBuffer mVertices;

private final float[] mVerticesData ={ -0.90f, -0.90f , // Triangle 1

0.85f, -0.90f ,-0.90f, 0.85f ,0.90f, -0.85f , // Triangle 20.90f, 0.90f ,-0.85f, 0.90f };

public MyGLRenderer(Context context) {mVertices = ByteBuffer.allocateDirect(mVerticesData.length *

4).order(ByteOrder.nativeOrder()).asFloatBuffer();mVertices.put(mVerticesData).position(0);

}@Override public void onDrawFrame(GL10 gl) {

GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);GLES20.glUseProgram(mProgramObject);GLES20.glDrawArrays(GLES20.GL_TRIANGLES, 0, 6);

}@Override public void onSurfaceChanged(GL10 gl, int width, int

height) {GLES20.glViewport(0, 0, width, height);

}

@Override public void onSurfaceCreated(GL10 gl, EGLConfig config) {String src_vert =

"attribute vec4 vPosition; \n"+ "void main() \n"+ "{ \n"+ " gl_Position = vPosition; \n"+ "} \n";

String src_frag = "precision mediump float; \n"

+ "void main() \n"+ "{ \n"+ " gl_FragColor = vec4 ( 0.0, 0.0, 1.0, 1.0 );\n"+ "} \n";

int h_vert = GLES20.glCreateShader(GLES20.GL_VERTEX_SHADER);GLES20.glShaderSource(h_vert, src_vert);GLES20.glCompileShader(h_vert);

int h_frag = GLES20.glCreateShader(GLES20.GL_FRAGMENT_SHADER);GLES20.glShaderSource(h_frag, src_frag);GLES20.glCompileShader(h_frag);

int h_prog = GLES20.glCreateProgram();GLES20.glAttachShader(h_prog, h_vert);GLES20.glAttachShader(h_prog, h_frag);GLES20.glBindAttribLocation(h_prog, 0, "vPosition");GLES20.glLinkProgram(h_prog);

GLES20.glUseProgram(h_prog);GLES20.glVertexAttribPointer(0, 2, GLES20.GL_FLOAT, false, 0,

mVertices);GLES20.glEnableVertexAttribArray(0);

mProgramObject = h_prog;

GLES20.glClearColor(0.0f, 0.0f, 0.0f, 0.0f);}

}

Implementing GLSurfaceView

• New Class• Package – browse the folder (e.g. com.example.android.opengl)

• Name – choose yours (e.g. “MyGLSurfaceView”)

• Superclass – GLSurfaceView

• MyGLSurfaceView.javapackage com.example.android.opengl;

import com.example.android.opengl.MyGLRenderer;

import android.content.Context;import android.opengl.GLSurfaceView;

public class MyGLSurfaceView extends GLSurfaceView {public MyGLSurfaceView(Context context) {

super(context);}

}

GLSurfaceView

• Manages an EGL display, which enables OpenGL to render into a surface.• Calls EGL functions internally

(https://android.googlesource.com/platform/frameworks/base/+/master/opengl/java/android/opengl/GLSurfaceView.java)

• Renders on a dedicated thread to decouple rendering performance from the UI thread.

• UI handling• Requires to subclass GLSurfaveView• Requires cross-thread communication using queueEvent()

• EGL configuration using setEGLConfigChooser before setRenderer()• default: RGB_888 color buffer + (at least) 16-bit depth buffer

• Requires to register a renderer using setRenderer

• Rendering mode – continuous or on-demand (set by setRenderMode())

• Requires to be paused/resumed when the activity is pause/resumed

GLSurfaceView.Renderer

• Responsible for making OpenGL calls to render a frame

• Called on a separate thread than the UI thread communication by queueEvent()

• EGL context lost• happens typically when device wakes up after going to sleep

• needs to re-create any lost resources in onSurfaceCreated()

Minimal OpenGL ES App (NDK)

Building a Minimal OpenGL ES App (NDK)

• NDK Setup• Download & unzip NDK (https://developer.android.com/tools/sdk/ndk)

• Set NDK location in Eclipse• Window Preferences Android NDK NDK Location (where you unzipped the

NDK files)

• Samples

• hello-gl2 by Google (included in NDK)

MyActivity.java

onCreate()onResume()onPause()

MyGLRenderer.java

onDrawFrame(){

MyGLJNI.display();}onSurfaceChanged(){

MyGLJNI.reshape(…);}onSurfaceCreated(){

MyGLJNI.init();}

MyGLJNI.java

static {System.LoadLibrary(“my_gl_jni”);

}native init();native reshape(…);native display();

my_gl_jni.cpp

init_JNI()reshape_JNI(…)display_JNI()JNICALL Java_com_example_test_MyGLJNI_init(…){ init_JNI(); }JNICALL Java_com_example_test_MyGLJNI_reshape(…){ reshape_JNI(…); }JNICALL Java_com_example_test_MyGLJNI_display(…){ display_JNI(); }

Android.mk

LOCAL_MODULE := libmy_gl_jniLOCAL_SRC_FILES := my_gl_jni.cpp

src/com/example/test

jni

Overall Procedure

1. Create an Android Application Project (with black activity)• Don’t include “_” in the package name!

2. Implement an Activity class (onCreate(), onResume(), onPause())• Set activity properly in AndroidManifest.xml as before

3. Implement a GLSurfaceView class (optional)

4. Implement a GLSurfaceView.Renderer class (onDrawFrame(), onSurfaceChanged(), onSurfaceCreated())

5. Implement a JNI interface library

6. Add native support: Right click Android Tools Add Native Support… Library Name: my_gl_jni(“my_gl_jni.cpp” and “Android.mk” are created in “jni” folder)

7. Modify “my_gl_jni.cpp” to implement OpenGL ES source code (init_JNI(), display_JNI(), reshape_JNI())

8. Modify “Android.mk”

9. AndroidManifest.xml Manifest Manifest Extras Uses Sdk Set “Min SDK version” to “14”

MyActivity.java

private GLSurfaceView mGLSurfaceView;@Override protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);mGLSurfaceView = new GLSurfaceView(this);mGLSurfaceView.setEGLContextClientVersion(2);mGLSurfaceView.setRenderer(new MyGLRenderer());setContentView(mGLSurfaceView);

}@Override protected void onResume() {

super.onResume();mGLSurfaceView.onResume(); }

@Override protected void onPause() {super.onPause();mGLSurfaceView.onPause(); }

MyGLSurfaceView (optional)

public MyGLSurfaceView(Context context) {super(context);

}

MyGLRenderer.java

@Overridepublic void onDrawFrame(GL10 gl) {

MyGLJNI.display();}

@Overridepublic void onSurfaceChanged(GL10 gl, int width, int height) {

MyGLJNI.reshape(width, height);}

@Overridepublic void onSurfaceCreated(GL10 gl, EGLConfig config) {

MyGLJNI.init();}

MyGLJNI.java

• Implement a JNI library class• New Class

• Name – e.g. “MyGLJNI”

• No superclass

static {System.loadLibrary("my_gl_jni");

}public static native void reshape(int width, int height);public static native void display();public static native void init();

my_gl_jni.cpp

• Create “jni” folder

• Create my_gl_jni.cpp which implements init_JNI(), display_JNI(), reshape_JNI()

• Add following (e.g. package name is “com.example.test”, JNI interface java class name is “MyGLJNI”)

extern "C" {JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_init(JNIEnv * env, jobject obj);JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_reshape(JNIEnv * env, jobject obj, jint width, jint height);JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_display(JNIEnv * env, jobject obj);

};JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_reshape(JNIEnv * env, jobject obj, jint width, jint height){ reshape_JNI(width, height);}JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_display(JNIEnv * env, jobject obj){ display_JNI();}JNIEXPORT void JNICALL Java_com_example_test_MyGLJNI_init(JNIEnv * env, jobject obj){ init_JNI();}

Android.mk

LOCAL_PATH:= $(call my-dir)

include $(CLEAR_VARS)

LOCAL_MODULE := libmy_gl_jniLOCAL_CFLAGS := -WerrorLOCAL_SRC_FILES := my_gl_jni.cppLOCAL_LDLIBS := -llog -lGLESv2

include $(BUILD_SHARED_LIBRARY)

EGL

• Native Platform Interface• https://www.khronos.org/egl• https://www.khronos.org/registry/egl/sdk/docs/man/xhtml/

• GLSurfaceView calls EGL functions internally

• NDK without GLSurfaceView• https://code.google.com/p/android-native-egl-example/• Examples by Imagination (PowerVR)• EGL initialization in JNI

• FreeGLUT for Android• http://freeglut.sourceforge.net/docs/android.php

OpenGL ES 2.0 Version Checking

• Needs to check if GLES 2.0 is supported after creating GLSurfaceViewby checking

• Android emulator bug – returned value is always 0

• If supported, set GLES version to 2.0 before setRenderer()

((ActivityManager)getSystemService(Context.ACTIVITY_SERVICE)).getDeviceConfigurationInfo().reqGlEsVersion >= 0x20000

mGLSurfaceView.setEGLContextClientVersion(2);

Compiling Shaders

• Similar to the desktop OpenGL

Examples

Filling Buffers

• Use ByteBuffer class

• https://developer.android.com/training/graphics/opengl/shapes.html

Transformations

• android.opengl.Matrix class has helper functions for model-view transformations & projections

• Steps1. float[] matMVP = new float[16];

2. android.opengl.setIdentity(matMVP, 0);

3. android.opengl.perspectiveM(matMVP, …);

4. <more transformations in reverse order…>

5. android.opengl.translateM(matMVP, …);

6. GLES20.glUniformMatrix4fv(…,FloatBuffer.wrap(matMVP));

Animation

• Set setRenderMode(GLSurfaceView.RENDERMODE_WHEN_DIRTY); (e.g. in the constructor of your GLSurfaceView class)

• Call requestRender() whenever the screen needs to be refreshed

• https://developer.android.com/training/graphics/opengl/touch.html

UI

• To handle touch input, implement onTouchEvent() method in your GLSurfaceView class

• ApiDemos/Graphics/OpenGL ES/Touch Rotate

• “Responding to Touch Events” (Android tutorial)

• http://www.learnopengles.com/android-lesson-five-an-introduction-to-blending/

Texture Loading

• android.opengl.GLUtils to load texture from Bitmap class

• ApiDemos/Graphics/OpenGL ES/Textured Triangle

• http://www.learnopengles.com/android-lesson-four-introducing-basic-texturing/

Texture Compression

• ETC1• standard compression format (no alpha channel supported)• ETC1Util utility class

• ATITC(ATC)

• PVRTC

• S3TC(DXTn/DXTC)

• 3DC

• Should be declared in AndroidMenifest.xml using <supports-gl-texture> tag for Google Play filtering service

• ApiDemos/Graphics/OpenGL ES/Compressed Texture

Text Rendering

• ApiDemos/Graphics/OpenGL ES/Sprite Text