© 2010 InHand Electronics, Inc. – Troubleshooting Linux Performance and Battery Consumption Issues Dave Stewart Director of Software Engineering

© 2010 InHand Electronics, Inc. – www.inhand.com

Troubleshooting

Linux Performance and

Battery Consumption Issues

Dave Stewart

Director of Software Engineering

InHand Electronics

[email protected]

www.inhand.com

Embedded Systems Conference Chicago 2010 Class ESC 361

Dave Stewart, ESC Chicago – June 2010

Troubleshooting Linux Performance and Battery Consumption Issues © 2010 InHand Electronics, Inc. – www.inhand.com 2

Contents

Overview Go for Idle Performance Debugging Sleep and Shelf Modes Device Low Power Modes Dynamic Frequency and Voltage Scaling

(DFVS) Duty Cycling



Overview

Typical Characteristics of Embedded Hardware Environments that support Linux: 32-bit Environment, E.g.

• ARM (e.g. OMAP, PXA320)• x86 (e.g. Intel Atom)• PowerPC

Numerous desktop-like devices• E.g. USB, Ethernet, WiFi, Bluetooth

Human-Machine Interfaces• E.g. Display, Touchscreen, Keyboard, Keypad, Mouse• Possibly HMI is via network, e.g. Web Page

Some custom devices• E.g. camera, display, sensors, actuators, wireless radio, GPS



General Concepts

Performance vs. Battery Consumption Increased performance == more power Longer battery life == performance compromise

One Exception: Optimize code ==

Less power AND better performance

Only Power what you Need Most hardware have various modes that enable turning off

or going into low power mode Most drivers, by default, don’t make use of these modes



Battery Life

Capacity is specified in mAh• mAh = milli-Amp-Hours• E.g. 1400 mAh means the battery can drain 100mA for 14 continuous

hours• Or it can draw 1.4A for one hour

By measuring current when battery is discharging, can compute power consumption

• Power = Battery_Voltage * Discharge_Current



Battery Life

Battery capacity only guaranteed for “new” batteries• After a battery has been recharged a few dozen times, it starts to age• As it ages, capacity decreases.• The faster you discharge the batteries, typically the shorter its lifespan

• Check battery datasheet. Some batteries may have capacity reduced by 50% in just a few hundred recharge cycles.

– This means less than a year if recharged daily.



Battery Level

For Alkaline batteries, voltage is a reasonable gauge of remaining capacity

For Lithium type batteries, voltage level of battery decreases relatively slowly

• thus voltage is not indicative of remaining life



Battery Level

Li-Ion and Li-Polymer batteries typically use a “fuel gauge” to measure battery level

• The gauge counts coulombs– When charging, the count increases– When discharging, the count decreases

• Battery level is only accurate when the capacity is “calibrated”– Once a battery starts aging, a battery level algorithm may

require the battery to be fully discharged, then reset the algorithm to indicate a new battery (even though it is old), then fully charge. This allows the new capacity to be re-learned.

– Some drivers auto-calibrate anytime a full charge occurs Actual algorithm is hardware dependent

• If battery level does not seem to be accurate, verify that drivers are properly reading the fuel gauge, and that there is some form of recalibration for aging batteries



Reduce Power: Go For Idle

Idle systems can offer best power savings Sometimes, running FASTER can reduce power, by getting

back to idle quicker• E.g. SPI driver with busy-wait loops. Run SPI clock as fast as

possible, to get back to idle more quickly• For processor, running faster usually means more power

Slow down periodic threads• Read an A/D 50 times per second instead of 100 times per second,

if application allows it Eliminate unnecessary services and interrupts

• If system is not being used for 1 minute, capture every interrupt that occurs or every service that executes.

• Ask WHY they execute, if system is idle?• Coming out of idle often is costly to battery life.



Go For Idle

Idle Implementations Idle Thread

• Lowest-priority thread has an empty while(1) loop. • Anything that becomes active preempts this thread.• Hardware-independent, thus easily portable to any processor• Power-hungry; processor is still running at full tilt

CPU Idle Mode• Most modern processors have an Idle Mode• By installing hardware-dependent support to use this mode, significant

power savings can be achieved when idle Deep Idle

• Turn off the processor oscillator when idle• May take 100 usec extra to wakeup• But the power savings can be significant• Extra care needed if DMA running in background, e.g. for Display• If turning off oscillator not an option because it breaks some devices,

consider just slowing down clock when idle



Go For Idle

Look at arch/$ARCH/kernel cpu_idle() is starting point

• grep cpu_idle *.c to find which file has the function– Depending on architecture, several possibilities

• default_idle() is used if you don’t override it• arch_idle() may already utilize the CPU instruction, check what it

does pm_idle() is your hook

• To create your own idle, define an idle function, and attach it to the pm_idle() hook

• Use pm_idle() to adjust clock speed or oscillator upon enterring and exiting idle

• Refer to “idle” section of your processor architecture for power-saving options that are available

• This code might require a call to assembly language functions



Performance Debugging

Identify threads that use too much execution time On many systems, one or two threads end up using the bulk

of execution time Optimize these one or two threads, and quickly see power

and performance improvements

How to identify the culprits? If really obvious, ps command will show it Many times not so obvious, use logic analyzer

• Enables identifying issues with both threads and interrupts




Scatter logic analyzer debug statements throughout code Start and stop of each function and thread is a good starting point. Often, just keep all the prior debug statements in place, they may

provide the data needed.

Run the code, and monitor on the logic analyzer. Look for patterns of behavior. Identify patterns that use too much CPU. Verify what is happening when analyzer shows periods of

inactivity. Monitor execution time of interrupts.

Some examples on next few slides. PDEBUG_HEX8(x) sends an 8-bit code ‘x’ to the logic analyzer via

pre-defined digital outputs.• See ESC-203, “Troubleshooting Real-Time Software Issues using a Logic

Analyzer” for specific details




Green is commmunication. White/Yellow is PDEBUG_HEX8().

We can clearly see communication stopped for a period of time.

Why? Is that normal?

From ESC-203 – Troubleshooting Real-Time Software Issues using a Logic Analyzer




See the Pattern?

Four short communication packets, one long one, then none for what seems like two or three system cycles.





What line of code executes immediately after PDEBUG_HEX8(0x9F)?

Whatever it is, it takes a LONG time.

If there is a performance issue, focus on that code first.





This is an example of code executed with PDEBUG_HEX8()

What is happening right after the code 0x33 is sent?





How do we capture EVERYTHING in Linux? Go straight for the context switch and ISR code

For context switch, look at kernel/sched.c context switch code is in function schedule() After call to sched_info_switch(), send process ID of new

task to logic analyzer Often it is fine to just send lower 8 bits of ID

For ISR, look at kernel/irq/handle.c At beginning and end of handle_IRQ_event(), send the IRQ

number to logic analyzerIdentify which threads or ISRs are using the most execution time

Focus on the longest ones, and determine if the amount of execution time is reasonable

If not, optimize it!



Suspend and Shelf Modes

Suspend (aka Sleep) Turn off processor and most peripherals Keep DRAM powered State of each I/O pin controlled by software Enables resuming within a second

Shelf (aka Deep Sleep) Turn off processor and all peripherals Turn off DRAM State of each I/O pin is undefined Resuming requires rebooting Used to maintain enough power for a “soft” power-switch,

and perhaps maintain an RTC Hibernate is a form of shelf mode, except that the contents

of DRAM are first saved to the file system




Suspend Mode Power Software has significant impact on power consumption

during this mode, even though CPU not running Devices not properly turned off or put into low power

modes may continue to draw power I/O pins not placed in the right state may cause power drain

Shelf Mode Power Software has no control of this. If this power consumption is too high, hardware engineers

need to investigate




Manage entry into suspend mode Use the xxx_power_off() of the driver

• Start by saving the context In addition to saving context:

• if the device has a low-power mode, enter it; or• if the device has a power-enable, disable it• if turning off power, hardware engineers should verify there is no backfeed,

as that will unnecessarily consume power during the sleep mode Ensure that any output signal to that device is placed into its

inactive state• Note that for active-low signals that typically have a pull-up on the line, this

means set the output HI

When exiting suspend mode Do opposite steps in xxx_power_on(), in reverse order

• Exit low-power mode or re-enable power• End by restoring the context

If device was powered off instead of a low-power mode• It might be necessary to re-initialize the device, as any internal state would

have been lost.



Device Low-Power Modes

When a device is not needed, it should be placed into a low-power mode In theory, the xxx_power_off() function of the driver is what

should be called to turn off that device However, that only gets called during a suspend operation For high-powered drivers, create an IOCTL to execute the

commands of the power-off function under user control Create a second IOCTL to re-enable the power



Dynamic Frequency and Voltage Scaling (DFVS)

Higher frequency requires higher voltage The two need to be adjusted in sync, in order to achieve

best power savings

If CPU is not fully loaded, consider running at a lower frequency E.g. PXA270, CPU Core Only

• Running at 200MHz = 279mW• Running at 500MHz = 432mW• This is a 35% savings

In this case, running slower is better



Dynamic Frequency and Voltage Scaling (DFVS)

Affect on performance Sometimes no effect

• For example, if memory bus is 200MHz, but processor is 500MHz, a memory-intensive activity might not run any slower if processor slows down to 200MHz

• In this case, utilization is same regardless of processor speed, so go for lower power

Sometimes it does affect it• A CPU-bound job based completely out of cache may run 100% of

the time at 200MHz, but only 40% at 500MHz. • In this case, when running at 500MHz, task is in idle for 60% of the

time, thus power savings of idle mode can be leveraged.• However, if execution used 100% at 500MHz, performance would

be reduced if CPU clock is lowered to 200MHz

Check your processor manual or linux distribution for possible support for DFVS



DFVS Example Logic Analyzer Screen Captures Follow

Next few pages show a logic analyzer setup of a DFVS systemKey signals to look at:

PDEBUG_HEX8(): 8-bit debug port (yellow)• Indicates where in the code we are executing• See ESC-203 class for more details on this macro

DFVS 1-bit trigger (magenta) 3-bit Frequency (green)

• 1=104MHz, 2=208MHz, … 5=520MHz Instantaneous current consumption (green analog)

• Reading is inverted, scale is 0mA to 400mA, lower reading means higher current draw.

Battery voltage (red).Each page is a zoom of the prior page.

The zoom area is shown by the magenta rectangle.



Setup ExtrasExample

zoom



Setup ExtrasExample

zoom



Setup ExtrasExample

zoom



Setup ExtrasExample



Duty Cycling

Duty cycling high-powered items can lower average power consumption

A 50% Duty Cycle of a device can reduce consumption in half for that device

• This means ON half the time, then either OFF or in low-power mode the other half

Look at each individual hardware item to determine if there is an opportunity.

• Some are not obvious. A few examples given on following slides



Duty CyclingLEDs

LEDs can consume 0.1W each if on all the time• A 50% duty cycle at a fast enough rate will still light them albeit

more dimly, but consume one-fourth the power.• Essential LEDs, like the power indicator, can be more dimly lit at

night or indoors than if used outdoors Consider using a heartbeat LED instead

• Turn LED ON for 200msec every 5 seconds• This consumes only 4% of the power as compared to leaving that

LED ON all the time• Reduce that to 100msec every 10 seconds, then it consumes only

1% of the power as compared to always on



Duty CyclingNetworking and Serial

The receiver of most radios and serial or network links often consume more power than the transmitter

• This is because the transmitter knows when it transmits. It is easy to turn on transmitter, send data, then turn it off.

• It is more difficult for the receiver, because it has to be listening all the time just in case a message comes in

If “wake-on-activity” features exist in the hardware, use them!

• Drivers may support the feature, but high-level protocols don’t utilize it by default

• If a device has the feature, but driver doesn’t, then modify the driver



Duty CyclingNetworking and Serial (continued)

Consider the transmitting side’s retry mechanism• If the system could be idle for extended periods of time, then

suddenly receive data, and the transmitter retries if it doesn’t succeed, then it may be possible to turn off the receiver.

Suppose the transmitter has a 1-second timeout, and will retry 4 times before giving up

• The receiver can wakeup once every 4 seconds, stay on for 1 second, then turn off.

• It may not get first message, but it will get it at least once. If transmitter is not on battery power, then consider

increasing the retries• E.g. transmit every 0.25 seconds for 10 seconds until an Ack. In this

case, receiver only needs to wakeup for 0.5 seconds every 5 seconds, and still get redundancy in case of transmission errors.



Duty CyclingAnalog Inputs

Review every analog input, and determine how often the data is really needed Many applications throw away unnecessary samples

• If data is being sampled at 100Hz, but the application using it is only running once a second, then most the data might be unnecessary

Some data changes very slow. E.g. Battery Voltage.• Sampling it once every 20 seconds instead of every second is likely

sufficient.• Bump up rate to every second when charging and on AC power

Reducing sampling rate can enable CPU going into idle more often• As discussed earlier, this is a win

Many A/D devices consume less power when they are idle than when they are scanning

• Check datasheets; in some cases this may be automatic; in other cases the device might need to be placed into low-power mode explicitly



Summary

Find ways to make the system idle more often, for the greatest savings

Use a Logic Analyzer for Performance Debugging Leverage Sleep and Shelf Modes for extended battery life;

ensure proper device and I/O settings during these modes Use individual device low power modes when a device is

not being used Enable Dynamic Frequency and Voltage Scaling (DFVS) Look for opportunities to use duty cycling to reduce power

Documents

© 2010 InHand Electronics, Inc. – Troubleshooting Linux Performance and Battery Consumption Issues Dave Stewart Director of Software Engineering