Using Apache Portable Runtime (APR)

To make cross-platform development easier

Bruce Greenblatt Sr. Principal Software Engineer Symantec Information Management Division Data Insight http://www.symantec.com/data-insight Bruce_greenblatt@symantec.com

How to Support Multiple Target OS Platforms?

• As a developer, sometimes we have to make our code work on more than one OS. Usually just these two for me: – Linux – Windows

• What techniques are available to help? The two most popular techniques seem to be: – Write on Windows and laugh as the poor Linux

developers have to port your native code. – Write on Linux and laugh as the poor Windows

developers have to port your native code.

What is APR

• A set of C Language APIs that are implemented on Linux and Windows

• The APIs provide access to OS primitives and other useful functionality, e.g. – Synchronization and Multithreading

– File System

– Time Functions

– Hashing

– Memory Mapped

– Command Line Parsing

What is APR (cont)

• Used by Apache Web Server implementers to ease porting to multiple platforms

• It provides a consistent interface to underlying platform-specific functionality

• It’s FREE to use and distribute • Source code is available • My team uses APR on:

– Windows Servers: 2003, 2008, 2012 32 and 64-bit – Windows Desktops: 7 – Linux:

• Centos (5 and 6) • RHEL 5 • Ubuntu

Who Should Use APR?

• Will you have to develop code that runs on Linux and Windows? – If so, then you should use APR.

• Will you have to develop code that runs on multiple versions of Linux? – If so, then you should probably use APR.

• Are you a Linux developer and are worried that the sales team will tell you the exciting news about the big deal they just made that requires a Linux version? – If so, then you should probably use APR.

• Do you have to work on Windows but are more comfortable with Linux tools? – If so, then you could use APR and develop and debug on Linux.

APR Approach

• Don’t use native APIs. • Write to APR APIs where available • Compile your code and link with APR on the

platforms it supports • Many APR calls look very similar to native Linux

calls, but are prefixed by “apr_”. – apr_atoi64 instead of atoi64 – apr_file_open instead of fopen

• All APIs implemented on each platform – There aren’t any “not implemented yet” stubs!

APR Keys

• Source code available – Home page is: http://apr.apache.org/ – Licensed un Apache License 2.0 – Build environment for Windows and Linux – I use cygwin and MSVC for Windows builds, and gcc for Linux

• Great Documentation available: – http://apr.apache.org/docs/apr/1.4/modules.html – http://apr.apache.org/mailing-lists.html

• Compiles to native APIs – There’s no middleware system running along with your program – There’s no APR Virtual Machine

Debugging APR

• Since you get the source code, if you compile in debug mode

– You can step into APR code

– You can set breakpoints, etc. inside APR code

– You can look at APR internal data structures

• This is a major advantage over some previously used proprietary libraries, where this was not available

Initialization and Termination

• Before making apr calls, the library must be initialized, and a memory pool allocated – apr_initialize();

– apr_pool_t *mp = NULL;

– status = apr_pool_create(&mp, NULL);

• When done, the memory pool is destroyed, and the library is shutdown: – apr_pool_destroy(mp);

– apr_terminate();

APR Memory Pools

• Mainly used by APR for internal memory allocation.

• Theoretically, they can be used for your own memory too (but I never have).

• So, many apr functions take an apr_pool_t structure as a parameter.

Simple program

apr_pool_t *mp = NULL;

apr_time_t time1 = 0, time2 = 0;

apr_initialize();

apr_pool_create(&mp, NULL);

time1 = apr_time_now();

time1 = apr_time_sec(time1);

/* do some stuff */

time2 = apr_time_now();

time2 = apr_time_sec(time2);

printf(“Did stuff in %"APR_TIME_T_FMT" seconds.\n", time2 – time1);

apr_sleep(1000000);

apr_pool_destroy(mp);

apr_terminate();

Apr_time_t is a 64 bit number representing the number of seconds since epoch start (i.e. midnight Jan 1, 1970). Apr_sleep always sleeps for the specified number of microseconds. Let’s look at the implementation on Linux and Windows to see the difference.

Windows apr_time_now()

APR_DECLARE(apr_time_t) apr_time_now(void)

{

LONGLONG aprtime = 0;

FILETIME time;

SYSTEMTIME st;

GetSystemTime(&st);

SystemTimeToFileTime(&st, &time);

FileTimeToAprTime(&aprtime, &time);

return aprtime;

}

Linux apr_time_now()

APR_DECLARE(apr_time_t) apr_time_now(void)

{

struct timeval tv;

gettimeofday(&tv, NULL);

return tv.tv_sec * APR_USEC_PER_SEC + tv.tv_usec;

}

Implementation Differences

• Notice both Linux and Windows have the same behavior

• It’s important to look at the implementation on each platform to understand if there are any differences in the behavior

• It appeared to me in some cases (especially process synchronization) that the behavior on Windows vs Linux was different

Atomic Operations

• Used to update shared integers without locking

• Always call apr_atomic_init(mp); before using any APR atomic API call

• Calls are available to read, set and increment, decrement and add 32 bit integers

– apr_atomic_inc32(&i); // increments i

– j = apr_atomic_read32(&i); // stores the value of I in j

– apr_atomic_set32(&i, j); //updates the value of i with j

Atomic Operations (cont)

• Consider apr_atomic_inc32

– Windows implementation uses a platform call InterlockedIncrement

– Some *nix implementations use Native APIs, some ia32 based implementations use assembly.

Shared Memory

• Allows one process to create a memory segment and share it with another process (or processes).

• A simple way of allowing multiple processes to communicate – E.g. A process may create a shared memory area to

store it’s current “state”, which may include certain counters, etc.

– Other processes will then access the shared memory to see how that process is doing.

• Usually requires synchronization, e.g. proc mutex, if readers and writers are involved

Shared Memory Example apr_pool_t *mp = NULL;

apr_shm_t *ctrs = NULL;

apr_proc_mutex_t *mutex_ctrs = NULL;

apr_status_t status = APR_SUCCESS;

apr_initialize();

status = apr_pool_create(&mp, NULL);

status = apr_proc_mutex_create(&mutex__ctrs, MTX_FILENAME,

APR_LOCK_DEFAULT, mp);

status = apr_shm_attach(&shm_netapp_ctrs, shmem_path, mp);

apr_proc_mutex_lock(mutex_ctrs);

p = (ctr_info_t *) apr_shm_baseaddr_get(shm_netapp_ctrs);

while (p && p->dev_id !=0) {

apr_ctime(apr_str, p->timestamp);

printf("id: %d\t latency: %d microseconds\t timestamp%s\n"

p->id, p->latency, apr_str);

i++;

if (i >= MAX_DEVICES) {

break;

}

p++;

}

apr_proc_mutex_unlock(mutex_netapp_ctrs);

apr_proc_mutex_destroy(mutex_netapp_ctrs);

status = apr_shm_detach(shm_netapp_ctrs);

apr_pool_destroy(mp);

apr_terminate();

Shared Memory Implementation

• On Windows use CreateFileMapping and MapViewOfFile

• On Linux use mmap, shmget, etc. Open APIs.

APR Multithreading support

• APR supports creating and exiting threads much like pthreads.

• apr_thread_create to start a thread

– You pass in the function to start the thread in

– The data you want to pass in to the thread

– An APR memory pool

• apr_thread_exit to terminate the thread

APR Hash Tables

• Create a hash table using one of two APIs: – hash = apr_hash_make(mp); – hash = apr_hash_make_custom(mp, callback_hash_function);

• Use the second version if you want to provide your own hash function and don’t want to use APR’s default hash function.

• Hash table lookup using apr_hash_get(hash, key_string, keylen); – keylen is usually = APR_HASH_KEY_STRING

• Hash table insert using apr_hash_set(hash, key_string, keylen, val); – Val is a void * – keylen is usually = APR_HASH_KEY_STRING

APR Hash Tables (cont)

• Clean up hash tables by removing all the rows, and then calling apr_hash_clear();

apr_hash_index_t *hi = 0;

char *key = 0;

apr_ssize_t keylen = 0;

if (hash) {

hi = apr_hash_first(mp, hash);

while (hi) {

apr_hash_this(hi,(const void**) &key, &keylen,

(void**) &e);

if (e) {

free(e->field1);

free(e->field2);

free(e);

e = NULL;

}

hi = apr_hash_next(hi);

}

apr_hash_clear(hash);

hash = NULL;

}

APR Modules

• Internal Memory Allocation • Atomic Operations • Dynamic Object Handling • Functions for manipulating the

environment • Error Codes

– APR Error Space – APR Error Values – Status Value Tests

• File Information – File Permissions flags – Stat Functions – Directory Manipulation Functions – Filepath Manipulation Functions

• File I/O Handling Functions – File Open Flags/Routines – File Seek Flags – File Attribute Flags – {_full} max iovec size – File Lock Types

• Filename Matching Functions • Miscellaneous library routines • Command Argument Parsing • Global Locking Routines

• Hash Tables • General Purpose Library Routines • MMAP (Memory Map) Routines • Network Routines

– Socket option definitions – IP Protocol Definitions for use

when creating sockets – IP Multicast

• Poll Routines • Memory Pool Functions

– Pool Cleanup Functions – Pool Debugging functions.

• Portability Routines – Thread portability Routines – DSO (Dynamic Loading) Portability

Routines

• Process Locking Routines • Random Functions • Ring Macro Implementations • Shared Memory Routines • Signal Handling • String routines

– snprintf implementations

• Internal APR support functions

• Table and Array Functions • Condition Variable Routines • Thread Mutex Routines • Threads and Process Functions

– Other Child Flags

• Reader/Writer Lock Routines • Time Routines • User and Group ID Services • Library initialization and

termination • ctype functions (e.g. apr_islower)