For Review Only

Dry Electrodes-based Fully Integrated EEG/fNIRS Hybrid

Brain-monitoring System

Journal: Transactions on Biomedical Engineering

Manuscript ID Draft

Manuscript Type: Paper

Date Submitted by the Author: n/a

Complete List of Authors: Lee, Heung-No; Gwangju Institute of Science and Technology, Information and Communications Lee, Seungchan; Gwangju Institute of Science and Technology, Department of Electrical Engineering and Computer Science Shin, Younghak; Norges teknisk-naturvitenskapelige universitet, Electronics and Telecommunications

Kumar, A; PDPM -Indian Institute of Information Technology Design and Manufacturing Jabalpur, Electronic * Communication Kim, Minhee; Gwangju Institute of Science and Technology, Department of Biomedical Science and Engineering

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

1

Abstract—A portable hybrid brain monitoring (HBM) system

is proposed to perform simultaneous electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) measurements. The proposed system comprises two separate boards for EEG and fNIRS acquisition, and these boards are simultaneously controlled by the STM32L4 microcontroller. In order to facilitate miniaturization and reliable performance, ADS1299 and ADS8688A-analog frontend (AFE) integrated circuits (ICs) for 16-channel EEG and 8-channel fNIRS measurements, respectively, are employed. The noise effect generated from digital circuit components, such as the microcontroller, oscillator, and switching LEDs, is reduced through electrical isolation of the EEG and fNIRS acquisition circuits. Gel-less EEG measurement was enabled by using spring-loaded dry electrodes. Several evaluations were carried out by conducting an EEG phantom test and an arterial occlusion experiment. An alpha rhythm detection test and an experiment to measure mental arithmetic operation were conducted to determine whether the system is applicable to human subject studies. The evaluation results show that the proposed system is sufficiently capable of detecting micro-voltage EEG signals and hemodynamic responses. The results of the human subject studies enabled us to verify that the proposed system is able to detect task-related EEG spectral features such as eye-closed event-related synchronization (ERS) and mental-arithmetic event-related desynchronization (ERD) in the alpha and beta rhythm ranges. An analysis of the fNIRS measurements with an arithmetic operation task also revealed a decreasing trend in oxyhemoglobin concentration.

Index Terms—electroencephalogram (EEG), functional near-infrared spectroscopy (fNIRS), hybrid brain–computer interface, multimodal analysis, portable instrument, simultaneous measurement.

This research was supported by the Brain Research Program through the

National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning (NRF-2016M3C7A1905475). This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIP) (NRF-2015R1A2A1A05001826)

S. Lee, A. Kumar and Heung-No Lee* are with Department of Electrical Engineering and Computer Science, Gwangju Institute of Science and Technology, Gwangju, 61005, Korea (e-mail: seungchan@gist.ac.kr, heungno@gist.ac.kr, anilkdee@gmail.com). * indicates the corresponding author.

Y. Shin is with Department of Electronic Systems, Norwegian University of Science and Technology, Trondheim, Norway (e-mail: shinyh0919@ gmail.com).

M. Kim is with Department of Biomedical Science and Engineering, Institute of Integrated Technology, Gwangju Institute of Science and Technology, Gwangju, 61005, Korea (e-mail: minhee@gist.ac.kr, jaekim@gist.ac.kr).

I. INTRODUCTION he brain–computer interface (BCI) [1], [2] was originally developed to assist severely disabled people in controlling

the action of their peripheral nerves and muscles, in cases of neurological and neuromuscular disorders such as amyotrophic lateral sclerosis, brainstem strokes, and spinal cord injuries. This technology, in turn, has provided a new channel that facilitates human-machine interaction. Presently, a number of new techniques based on wearable devices and the Internet of Things (IoT) are being applied to BCIs related to the fields of healthcare, telemedicine, and clinical care [3]. Current BCI technology, however, faces several challenges, such as its limited number of controllable functional-brain signals [4], the need for recalibration, and lack of controllability, also referred to as “BCI-illiteracy” in real-world applications [5].

Multimodal analysis of brain activities—the so-called hybrid BCI [6], which can be implemented by simultaneously acquiring and analyzing two or more brain signals, has been proposed as an alternative BCI technique capable of overcoming the above challenges. Two or more complementary neurological signals can be combined and shared to maximize the amount of exploitable information, thereby enhancing the robustness of control accuracy in real-world applications.

Hybrid BCI systems could be established by selecting suitable modalities amongst various brain imaging techniques, such as electroencephalogram (EEG), magneto encephalogram (MEG), functional magnetic resonance imaging (fMRI), and functional near-infrared spectroscopy (fNIRS). Among these modalities, the disadvantages of MEG- and fMRI-based techniques is the need to install the machines in confined areas and the fact that they can only be used for short runtimes because of their high cost, large size, and the need for expert operators [7]. Contrary to this, EEG- and fNIRS-based brain-monitoring systems are advantageous because they are lightweight and compact, and involve low costs. Such hybrid systems could easily be built as portable or wearable devices and utilized in more dynamic applications, such as driver drowsiness detection [8] and seizure monitoring in epileptic patients [9].

EEG is a spatially summed up electrical potential produced by synchronous activation of dendritic branches of a large number of neurons. Given that electrical conductivity of biological tissues is similar to that of electrolytic fluids, its spatial resolution is relatively low, whereas its temporal

Dry Electrodes-based Fully Integrated EEG/fNIRS Hybrid Brain-monitoring System

Seungchan Lee, Younghak Shin, Member, IEEE, Anil Kumar, Member, IEEE, Minhee Kim, and Heung-No Lee*, Senior Member, IEEE

T

Page 1 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

2

resolution is sufficiently high in the millisecond range [2]. On the other hand, fNIRS measures the changes in the local concentration of oxygenated and deoxygenated hemoglobin in the cerebral cortex region by utilizing low-energy optical radiation from two different wavelength light sources in the near-infrared range (700-1000 nm). Although this technique demonstrates a slower response compared to EEG, it enables an investigation of metabolic and microcirculatory neuronal activation regardless of the electrically synchronized activation of neurons [10]. The simultaneous acquisition of EEG and fNIRS measurements could make more comprehensive neurodynamic information regarding neuronal metabolism and neuroelectric activities accessible. As such, several researchers have recently developed EEG–fNIRS hybrid systems for use in various applications [11].

A review of the available literature related to hybrid BCI systems indicates that a combination of individual EEG and fNIRS systems has been used in various experimental accomplishments regarding motor imageries [12] and visual and auditory stimulations [13]. In such a setup, a fully synchronized operation of the entire system is difficult, because each individual system contains its own controller that is operated at a predefined clock speed. Therefore, the measurements acquired from two systems may not be completely synchronized in the absence of a precise simultaneous control mechanism. Attempts to address this concern have resulted in the design of customized EEG–fNIRS hybrid acquisition instruments.

One of the first attempts to this end started with the design of a probe for simultaneous measurements of EEG and fNIRS data [14]. Lareau et al. [15], [16] proposed a similar hybrid system that was capable of acquiring multichannel EEG and fNIRS measurements. However, it was practically difficult to use it as a portable device because of its large size. In 2013, two more compact and advanced bimodal acquisition systems were developed by Safaie et al. [17] and Sawan et al. [18]. These systems possess a small form factor, facilitated by employment of a field-programmable gate array (FPGA) and an analog front end (AFE) integrated circuit (IC). In 2014, Zhang et al. [19] proposed a wearable recording system for hemodynamics, electrocardiography (ECG), and actigraphy measurements. Recently, Luhmann et al. [20] developed a miniaturized modular hybrid system, wherein one module is capable of simultaneously monitoring four channels of bioelectrical and bio-optical measurements. Despite the small size of the system, its practical usability might be undesirable for mobile and ambulatory applications because of its use of wet electrodes that require conductive pastes.

This paper proposes a portable hybrid brain monitoring (HBM) system that provides high-resolution, multi-channel, and fully synchronized EEG and fNIRS monitoring. To enable simultaneous measurement of 16 EEG and 8 fNIRS channels, the proposed system consists of system boards (designed as a main and a slave board), 16 dry electrodes, and an fNIRS probe set (including two near-infrared LEDs and six monolithic photodiodes). The dry electrodes [21], [22] are employed for EEG acquisition instead of conventional wet electrodes to

maximize system usability in out-of-lab applications. The proposed system is synchronously operated by a single microcontroller unit (MCU), whereas electrical and hemodynamic activities are separately monitored by dedicated AFE ICs. Each acquisition circuit comprises state-of-the-art components to facilitate miniaturization, low complexity, and reliable low-noise performance. The acquisition of EEG measurements using the dry electrodes was evaluated by performing an EEG phantom test. An arterial occlusion experiment was performed to verify the hemodynamic responses of the fNIRS measurements. Finally, human subject studies including an alpha rhythm detection test and an experiment to assess mental arithmetic operation were performed to verify the practical capabilities for EEG and fNIRS feature measurements.

The remainder of the paper is organized as follows: Section II describes the design concept, detailed system architecture, and experimental method for the capability evaluation of EEG measurements using artificially generated EEG waveform. In addition, human subject studies, including an arterial occlusion experiment, an alpha rhythm detection test, and a mental arithmetic operation experiment, are also presented. Section III summarizes several results relating to instrumental design, capability evaluation, and offline analysis of human subject studies. The hardware architecture for EEG and fNIRS acquisition is compared in Section IV. Finally, concluding remarks with a summary of system design contributions and experimental results are given in Section IV.

II. METHODOLOGY

A. Design Concepts

The integration of high-precision multi-channel EEG and fNIRS acquisition systems in a limited space and efficient suppression of the crosstalk originating from digital components, such as the MCU, oscillators, and LED driving circuits, in a mixed-signal system were identified as the major challenges facing the proposed system design. Maximizing system usability for long-term and out-of-lab applications was also considered as further requirements. The following design strategy for the proposed HBM system is adopted to address these concerns: • Use of optimized and integrated AFE ICs; • Design of fully isolated low-noise power-supply circuits;

and • Use of dry spring-loaded electrodes.

1) Use of optimized and integrated AFE ICs

As physiological signals—EEG and fNIRS—possess small amplitudes and are highly susceptible to various types of noise; therefore, the use of complicated signal-conditioning circuits: amplifiers and discrete filters, becomes necessary to achieve high-precision measurements. Recently, state-of-the-art integrated AFE ICs combined with high-resolution analog-to-digital converters (ADCs), signal-conditioning circuits, and associated peripherals were reported [23]. Such

Page 2 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

3

ICs can reduce the number of required discrete components due

to their integrated functions. By avoiding mismatch errors arising from the specification tolerance of the discrete components, these AFE ICs provide reliable performance and support for miniaturized and low-cost designs.

The proposed design employs the ADS1299 AFE IC (Texas Instruments, USA) [24] for EEG measurements. It was integrated with 8-channel, 24-bit resolution delta–sigma ADCs, programmable gain amplifiers (PGAs), and other useful peripherals, such as an input-control multiplexer and a bias-driving circuit. A sufficiently small step size of the least significant bit (LSB) (0.022 µV at a gain setting of 24) and low peak-to-peak noise performance (0.98 µV at 250-Hz sampling rate and 24 gain setting) enables precise detection of EEG signals in the µV range. The simultaneous sampling architecture of the IC instantaneously samples input measurements over all channels, and all ADCs operate synchronously without the need for sample-and-hold circuits. Moreover, no sampling skew and glitch noise exists in the converted data. The firmware of the internal registers of the AFE IC was programmable with various configuration settings such as the sampling rate (250–16 kHz) and gain (1–24) associated with each acquisition channel and the embedded functions.

The ADS8688A AFE IC (Texas Instruments, USA) [25] was employed for the acquisition of fNIRS measurements. Similar to ADS1299, this AFE IC was also integrated with such peripherals as a 16-bit successive approximation register (SAR) ADC, 8-channel input multiplexer, PGAs, and second-order low pass filters. Although the resolution of the built-in ADC was lower than that of the one integrated with ADS1299, a sufficiently small LSB step size could be achieved by manipulating the acceptable input ranges (unipolar range of 1.28–10.24 V and bipolar range of ±0.64–±10.24 V). The smaller the input range at the maximum PGA gain setting, the smaller is the LSB step size; therefore, the maximum resolution is achievable from minimum input range setting. An input range setting with a minimum LSB step size of 19.53 µV, obtained from the minimum input range (± 0.64 V), provides sufficient resolution to accurately acquire fNIRS data in the millivolt range.

2) Design of fully isolated low-noise power-supply circuits

In mixed-signal systems, the analog and digital parts are integrated into a single circuit, and high-speed current switching in digital circuits induces high-frequency noise that could be coupled to neighboring analog circuits via stray capacitances [26]. In the proposed HBM system, periodical switching operation of LEDs is necessary to acquire fNIRS measurements. It can generate oscillating noise in the digital circuits due to the instantaneously high current flow in the LED driving circuit. The noise may interfere with the analog circuits via the stray capacitances and can easily distort the small EEG and fNIRS amplitudes.

The proposed system was designed to include fully isolated power-supply circuits and digital interfaces to facilitate noise-free acquisition of EEG and fNIRS measurements. The EEG acquisition circuit is operated by an independent

Fig. 2. Power-supply circuit design employed in the proposed HBM system. Two lithium-polymer batteries supply power to the main board and the slave board, respectively. In the main board, the isolated DC–DC converter supplies power to the main control circuit (shaded area) and the isolated NIRS acquisition circuit (transparent area).

DC-DC Converter(TPS61232)

Regulator(ADM7171)

STM32L4, fNIR LEDs, Peripherals

3.3v

Digital Power Supply Circuits

fNIRS AFE(ADS8688A)

Regulator(TPS7A4701)

Regulator(ADP7142)

IsolatedDC-DC Converter

(DCP020507) 3.3v

5v

fNIRS Analog Power Supply Circuits

(1) Power Management Circuit for the Main Board

Battery Charger(BQ24073)

Li-poly Battery

5v

7v

3~4.2v

5v

Regulator(ADM7154)

Regulator(ADP7182)

Regulator(LP5907)

EEG AFEs(ADS1299)

3.3v

2.5v

-2.5v

EEG Analog Power Supply Circuits

(2) Power Management Circuit for the Slave Board

DC-DC Converter(TPS65133)

Battery Charger(BQ24073)

5v

-5v

3~4.2v

3~4.2v

Li-poly Battery

3~4.2v

Fig. 1. Simplified schematic of the proposed HBM system. Solid and dotted arrows, respectively, indicate the flow of digital logic signals and analog measurements. Likewise, the shaded and transparent regions, respectively, indicate the digital and analog circuits. The boundary between the analog and digital circuits is isolated by a digital isolator and DC–DC converter. The dedicated EEG acquisition circuits is also isolated from the main board circuits

EEG AFEs(ADS1299 x2)

16ch. EEGControl & Digitized

EEG

MCU(STM32L475)

DAC +Mux

Control

fNIRS AFE(ADS8688A)

MOSFET LED Drivers

+ Analog Multiplexer

6ch. NIRS

2ch. LED Switching

Digital Isolator(SI8662)

Control & Digitized

fNIRS

The Slave Board (EEG Acquisition)

The Main Board (fNIRS Acquisition)

Digital Isolator(SI8662)

EEG Analog Circuits

NIRS Analog Circuits

System & LEDs Control Digital Circuits

BT

Board-to-Board Connector

16 Dry Electrodes

2 fNIR LEDs(730nm+850nm)OE-MV7385-P

6 OPT101s(Photodiode +

Integrated TIA)

Page 3 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

4

lithium-polymer battery to isolate the power source from other digital-circuit components. The digital interface located between the MCU and ADS1299 was also isolated by a digital isolator IC (Silicon Labs Si8662), and the same isolator was used in the fNIRS acquisition circuit as well. Many advanced features, such as high data throughput, low propagation delay, and noise robustness of the isolator IC serve to provide a secure data path. The path utilizes an on/off keying scheme of radio frequency modulation on a semiconductor-based isolation barrier. The fNIRS acquisition circuit contains an isolated DC–DC converter (DCP020509) (Texas Instruments, USA) to provide a completely isolated power source. These isolation techniques ensure that the two acquisition circuits each have separate power sources and independently insulated ground planes to effectively suppress the effects of digital noise sources.

To provide low-noise DC power to the AFE ICs in each acquisition circuit, power supply circuits were carefully designed using a combination of low-noise linear regulators, a number of decoupling capacitors, and ferrite beads. The linear regulators offer several advantages compared to DC-DC converters, such as highly regulated output voltage, a low noise spectral density, and a high power supply rejection ratio (PSRR), thereby making them ideally suited for noise-sensitive power supply applications. In addition to these low-noise power supply circuits, an optimized printed-circuit-board (PCB) layout and advanced circuit-design techniques, such as grounding, signal routing, and decoupling [27], were applied to maintain stable and regulated DC voltages and build a low-impedance return current path.

3) Use of spring-loaded dry electrodes

Conventionally, disc-shaped Ag/AgCl electrodes have been employed in EEG measurements. These electrodes require the use of conductive gels and hair preparation during installation in order to reduce the electrical impedance to an acceptable level. These procedures are time consuming and cause irritation in most subjects, because conductive gels are sticky. Moreover, these electrodes are not suitable for long-term and ambulatory applications, because conductive gels dry over time and their adhesion is easily lost during motional vibrations. To overcome these problems, dry electrodes, which do not require conductive gels, are used in the proposed system. These electrodes comprise spring-loaded probes that maintain a constant pressure to the surface of the uneven scalp regardless of its movement. Consequently, these electrodes are capable of more stable EEG measurements even in out-of-lab applications. The dry-electrode structure is described in detail in the following section.

B. Instrumentation Design

Fig. 1 depicts a schematic of the proposed system excluding the power supply circuits. The system comprises two boards—the main board and slave board. The main board is capable of performing 8-channel bio-optical measurements, and 4-channel dual-wavelength LED emissions. The slave board was designed to perform 16-channel bio-potential

measurements. The two boards were connected using the Molex board-to-board connector, and all components were controlled by the STM32L475 low-power microcontroller (STMicroelectronics, USA) installed on the main board.

The following procedure was used to perform bio-optical measurements on the main board. Common-mode electromagnetic and radio-frequency interference noise is first filtered out from raw fNIRS measurements using a simple RC low-pass filter in the input stage. Inside the embedded ADS8688A AFE IC, the acquired NIR signal is amplified by the integrated PGA to pre-programed input ranges and subsequently filtered by an anti-aliasing low-pass filter with a 15-kHz cutoff frequency. The filtered signal is then fed to the ADC driver and multiplexer circuits, and is finally sampled by a 16-bit SAR ADC in accordance with pre-assigned channel-selection commands. The above sampling procedure and digitized data retrieval process takes approximately 12.8 ns using the serial peripheral interface (SPI) bus with a data speed of 2.5 Mbps.

As NIR LEDs consume an excessively large amount of current that exceeds the driving capacity of the MCU output port, LED driving circuits, based on MOSFETs, are essential. By using a digital-to-analog converter (DAC) embedded in the MCU to control the MOSFET gate voltage, the MCU is able to regulate the emission intensity of NIR LEDs by modulating the cathode-current flow in the driven LED. The regulated gate voltage of the MOSFET, controlled by the DAC, is buffered with an OPAMP and can be multiplexed with the analog device ADG729 to control up to eight NIR LED emission intensities. Since the NIR LEDs may exhibit radiation-power tolerance even under the same current flow, the emission intensity of all NIR LEDs was manually adjusted to 10 mW by using an optical power meter and adjustment of the MOSFET gate voltage setting.

The following procedure was also used to perform bio-potential measurements on the slave board. The bio-potential measurements acquired by the dry electrodes are filtered by the onboard input filter stage. This stage rejects the inherent electromagnetic and radio-frequency interference noise by a common- and differential-mode balanced RC filter employing X2Y capacitors [28]. These three-terminal capacitors, in which two matched capacitors are embedded via the built-in plate structure, can reduce the number of filter capacitors to build the common-mode filters. As these capacitors perform the same function as a common mode choke, they facilitate higher attenuation of electromagnetic and radio-frequency noise compared to conventional common-mode RC filters, while reducing onboard space requirements. Inside the ADS1299 AFE IC, the filtered bio-potential measurements are amplified by a built-in low noise PGA and digitized by a dedicated ADC for each channel over every sampling period. The sampled EEG data are then transmitted to the MCU via an SPI bus. Two ADS1299 AFE ICs are connected to each other via a daisy-chain configuration for 16-channel EEG monitoring. This configuration allows simultaneous control of a number of ICs with a single shared SPI data bus. The common reference mode is supported by all

Page 4 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

5

channels, four of which are also compatible with the fully differential mode in order to support other physiological measurements, such as Electromyography (EMG), Electrocardiography (ECG), and Electro-oculography (EOG) by manipulating the onboard dip switch.

Fig. 2 depicts a schematic of power-supply circuits of the proposed HBM system. The proposed system is powered by two lithium-polymer batteries—one each for the main and slave boards—which can be charged via the onboard battery management IC (Texas Instrument BQ24073) through a USB port. As the battery voltage decreases over time, boost and dual-output (integrated with boost and inverting buck-boost) DC–DC converters (Texas Instrument TPS61232, TPS65133) are used to stabilize the DC voltage. Low-noise dedicated power is supplied to the fNIRS acquisition circuits by employing an isolated DC–DC converter (Texas Instrument DCP020507) on the main board. In the final stage of the power-supply circuits, low-noise DC voltage is lastly delivered to the AFE ICs, MCU, and other peripherals through six low-noise linear regulators (Analog Devices—ADM7154, ADP7182, ADP 7142, and ADM7171; Texas Instruments—TPS7A4701 and LP5907)

C. Sensor Design

Customized sensor units were designed for the EEG and fNIRS measurements to enhance the usability and re-configurability of the proposed system. The sensor units comprise 16-channel dry electrodes, 2-channel NIR LEDs, and 6-channel optodes.

The left-most image in Fig. 3 depicts a prototype of the dry electrode for the EEG measurements, which comprises spring-loaded probes, a PCB, and housing. The electrode unit,

which is designed to remain in contact with the subject's scalp, acquires EEG potentials via the 18 spring-loaded probes (Leeno Industrial Inc., SK100R). Each probe comprises four components—(1) plunger, (2) barrel, (3) spring, and (4) probe receptacles. The plunger has a cylindrical shape and is fabricated of nickel plated with beryllium, copper, and gold. The plunger is combined with a barrel and spring to constitute the spring-loaded structure. Each spring can withstand up to 54 g of pressure in its maximally compressed state. This enables each probe to maintain a suitable contact pressure on the uneven surface of the scalp. In terms of electrical specifications, the resistance of each probe is less than 50 mΩ, which is sufficiently low for conducting bioelectrical measurements. All probes are electrically connected to each other via the PCB embedded in the electrode housing and are thereby linked to a single electrode wire. The entire electrode assembly is enclosed inside 3D-printed plastic housing.

A helmet-like bracket (named the EEGCAP) was designed using flexible rubber materials to hold the dry electrodes in position in accordance with 10–20 systems. The mesh-type EEGCAP structure was equipped with as many as 58 holes to allow electrodes to be positioned on the scalp. Each electrode was firmly engaged in the hole via an interlocking frame structure, and able to continuously push against the subject's scalp to maintain a constant pressure. This customizable structure could make a number of configuration choices available in terms of electrode positioning layout depending on the experimental paradigm.

Dual wavelength (730 and 850 nm) AlGaAs LEDs (Opto ENG OE-MV7385-P) were used for NIR emissions. Two LEDs packaged in a miniaturized plastic leaded chip carrier (PLCC) were soldered onto an emitter PCB and covered by 3D printed materials. The spectral spread of the emitted radiation (Δλ = 30–40 nm) was broader compared to that of monochromatic laser diodes (Δλ ≈ 1 nm). However, the incoherent and un-collimated characteristics of the LED light source achieve sufficient tissue penetration to enable the investigation of local hemodynamic changes. Owing to its suppressed heating and low risk of retinal damage, it can be used in direct contact with the human scalp [29].

The optode design was based on a monolithic photodiode (Texas Instruments OPT101) integrated with an on-chip trans-impedance amplifier to obtain accurate fNIRS measurements without the need for additional circuit components. Because the photodiode exhibits high spectral sensitivity in the infrared spectrum (> 0.5 A/W in the 730–850 nm wavelength), it is optimized for use in NIR detection applications. Owing to the embedded trans-impedance amplifier circuitry composed of an integrated operational amplifier and an internal feedback network, the optodes provide a sufficiently wide bandwidth of operation (14 kHz) with a fast dynamic response. The photodiode is soldered onto an optode PCB along with decoupling capacitors, and housed inside a 3D-printed-material casing.

Fig. 4 illustrates the positioning layout for NIR LEDs and optodes for placement on the subject's forehead. In this configuration, only two LEDs and six optodes are used for

Fig. 3. Dry electrode for EEG acquisition, optode unit for fNIRS acquisition, and LED unit for emission of NIR radiation.

Fig. 4. Configuration layout of NIR LEDs (L1 and L2) and optodes (O1–O6) for fNIRS measurement acquisition. To investigate hemodynamic changes at the frontal lobes, the emitter and detector units are attached using a transparent double-sided tape. Actual bio-optical measurements are acquired from 8 fNIRS channels (1–8).

O1 O3 O5

O2 O4 O6

L1 L2

1

2

3

4

5

6 8

7

10cm

3cm

Page 5 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

6

8-channel bio-optical measurements. Eight channels of bio-optical measurements requires measurements from the 3rd and 4th optodes to be shared between the 3rd, 4th, 5th, and 6th acquisition channels by exploiting a TDM-based sampling scheme. In this scheme, only the acquisition channels surrounding the turned-on NIR LEDs are instantaneously activated. The 3rd and 4th optode, therefore, contributes two channels of bio-optical measurements at different timings in sync with the operating sequence of the LEDs. The detailed operating sequence is provided in Section II-D.

D. System Operation and Hybrid Data Acquisition

The ADC basically converts analog input signals into digitized signals with consistent intervals based on an internal or external reference clock. However, the clock may have its own tolerance and frequency drift characteristics. In heterogeneous data-acquisition systems employing two or more ADCs to produce a fully synchronized data stream, the clock tolerance of individual ADCs makes accurate synchronization difficult to achieve. This problem can be solved by using a reference system clock to which all ADCs could be universally referred.

Complete synchronization is achieved between the EEG and fNIRS measurements by using the data-ready signals (referred to as DRDY in the datasheet) generated by the ADS1299 AFE IC as the reference system clock. The DRDY signal represents the logical transition of a falling edge when the digitized EEG data stream becomes valid. It, therefore, generates a pulse signal of the same period as the sampling rate of EEG acquisition. By synchronizing the emission control of NIR LEDs and data conversion of ADS8688A with the DRDY pulse cycle, the complete synchronization between EEG and fNIRS measurements can be preserved regardless of the occurrence of small timing errors in the reference clock of each ADC.

Fig. 5 depicts a single period of simultaneous EEG and fNIRS acquisition captured from the logic analyzer screen. Once ADS1299 begins to acquire EEG measurements from the internal register at a pre-programed sampling rate (250 Hz), DRDY signals begin to generate pulses with the same sampling period (4 ms) as EEG measurements. In accordance with the DRDY pulse generation, the wavelength of NIR LED emission is sequentially switched between 730 and 850 nm—L1 (730 nm)–L2 (730 nm)–L1 (850 nm)–L2 (850 nm) in that

order—over the course of 50 EEG acquisition cycles (200 ms). Each time the NIR LED is turned on by the multiplexer switch, the emission lasts for 4 ms, during which time the ADS8688A acquires NIR light intensities from the set of activated optodes surrounding the turned-on LED; i.e., when the L1-730 nm or L1-850 nm states are active, measurements from the optodes O1–O4 are sampled. This also applies to the two L2 states and sampling of optodes O3–O6. To obtain stable measurements with minimized background noise, the light intensity of each acquisition channel is repeatedly acquired 11 times with a 50-µs interval and subsequently averaged.

The aforementioned sequence of processes are used to packetize the fNIRS and EEG measurements along with a header and timing information, which are then successively sent to the host device via the SPBT3.0 DP2 Bluetooth module (STMicroelectronics, USA) every 200 ms and 4 ms, respectively. The sampling rate of fNIRS acquisition is, therefore, mechanically determined to be 5 Hz (=1/200 ms). In the host device, packets of EEG and fNIRS measurements are decoded in MATLAB 2014a (MathWorks, USA), using the instrumentation control toolbox, and monitored via real time plot updates.

MCU firmware was designed to manage these processes to perform the following closed loop operation: 1) Peripheral initialization—establishment of peripheral

interfaces (SPI interface, general purpose input/output ports, and interrupt routine) and setting up registers of all AFE ICs;

2) Launching the data-retrieval loop upon detection of trigger signals;

3) Acquisition of EEG data from ADS1299, when a DRDY pulse is generated;

4) Control of NIR LED emission in accordance with DRDY trigger signals;

5) Acquisition of fNIRS data from the predefined channel sets within ADS8688A in accordance with the LED control sequence;

6) Packetization of acquired EEG and fNIRS data along with header and timing indication and subsequent transmission of data packets to the host device via the Bluetooth module;

7) Repeating steps 3 through 6 until the stop trigger signal is detected.

Fig. 5. Logic analyzer view of one period of simultaneous EEG and NIRS acquisition and magnified view of the upper gray region (-0.5–4.5 ms).

ADS1299 DRDYNIR LED SwitchADS8688A Acq.

ADS1299 DRDYNIR LED SwitchADS8688A Acq.

4ms Interval, 250Hz EEG AcquisitionL1-730 L2-730 L1-850 L2-850 L1-730

1 Period (200ms, 50 times EEG Acquisition and one NIRS Acquisition)

1st EEG AcquisitionL1-730 NIRS LED Mux Switch On Switch Off

2nd EEG Acquisition

Optode Voltage Measurements and Averaging

Page 6 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

7

In summary, the EEG and fNIRS sampling rates of the system are to be set to 250 Hz and 5 Hz, respectively. The internal gain of the ADS1299 is set to 24 to minimize the baseline input-referred noise for EEG acquisition. The input range of the ADS8688A was set to ±0.64 V to minimize the LSB step size for fNIRS acquisition.

E. Calculations of Concentration Change of Oxy-, Deoxy-, and Total Hemoglobin

The changes in the concentration of oxy-, deoxy-, and total hemoglobin were calculated with reference to the previous studies [30], [31]. Within the range of NIR wavelengths utilized in this system, it is reasonable to assume that the background absorbance of biological tissues is negligible and that the main contribution of chromophores in human tissue is limited to oxy- and deoxy-hemoglobin (OHb and RHb, respectively). In view of the above assumptions, the change in optical density (ΔOD) at the two wavelengths (730 and 850 nm) could be related to the changes in OHb (ΔOHb) and RHb (ΔRHb) given by the following relation.

730 730 730

850 850 850

730 730

850 850,

RHb OHb

RHb OHb

RHb OHb

RHb OHb

OD RHbL

OHbOD

RHbd DPF

OHb

ε ε

ε ε

ε ε

ε ε

∆ ∆=

∆∆

∆= ⋅

∆

(1)

where 730 850 730 850, , ,OHb OHb RHb RHb

ε ε ε ε represent the extinction

coefficients of OHb and RHb at 730 and 850 nm, respectively, L is the optical path length between the light source and detector and may be approximated as L d DPF= × , where d is the source–detector distance and DPF is the differential path length factor introduced in Beer–Lambert’s law by accounting for light scattering effects [32]. As ΔOD can be estimated from the photodiode measurements using the relation given as

log( / )

Transient Baseline

Baseline Transient

OD OD OD

I I

o o o

o o

∆ = −

=, (2)

In Eq. (2), ΔOHb and ΔRHb could be estimated by using the following inverse formulation.

1730 730 730

850 850 850

1 RHb OHb

RHb OHb

RHb OD

OHb ODd DPF

ε ε

ε ε

−

∆ ∆=

∆ ∆⋅

, (3)

The change in concentration of the total hemoglobin (ΔTHb) can be estimated as the sum of ΔOHb and ΔRHb; i.e.,

THb OHb RHb∆ = ∆ + ∆ . (4)

F. System Evaluation of EEG and fNIRS Acquisition

1) EEG phantom experiment using dry electrodes

The proposed HBM system employs dry electrodes for EEG acquisition instead of the conventional wet electrodes for wide applicability and enhanced usability. Therefore, it is necessary to verify the acquisition capability of the dry electrodes at the level of micro-voltage amplitudes. In the proposed system, the

fNIRS and EEG acquisition circuits operate simultaneously. Thus, EEG signal acquisition is subjected to interference from the electric fields generated by the NIR LEDs and this effect must be examined. For this purpose, we devised an EEG phantom experiment, which is described next.

The EEG phantom experiment is designed in the following manner. First, EEG-like voltage signals were generated. Digital EEG signal samples of 1-min duration were taken from the C3 channel of a BCI competition 3-IVa dataset (motor imagery task down-sampled to 250 Hz). These EEG digital samples were then inputted to a signal generator which reproduces a voltage waveform (using the arbitrary signal generator function offered in Keysight 33220A). The reproduced voltage waveform was then passed through a voltage divider circuit (of 10000:1 ratio) to create a microvolt-level EEG signal. This voltage waveform was finally fed to the simple EEG phantom, described next.

Second, a simple EEG phantom was created using a conductive rubber pad (10 cm × 12 cm × 5 mm, 100 Ω/cm) to simulate a real human scalp. An NIR LED unit is placed at the center of the rubber pad. Then, one dry and one wet electrode (with conductive gel) were attached around the LED unit on the rubber pad to emulate the NIR interference during EEG signal measurement. The two electrodes and the NIR LED unit were connected the EEG signal port and NIR LED driving circuit of the HBM, respectably. The EEG reference input of the HBM system was connected to the ground potential of the waveform generator.

Third, the prepared voltage waveform of 60-s duration in the first step was fed to the simple EEG phantom. Measurement samples were recorded at a sampling speed of 250 samples per second from the two electrodes during the 60-s period. The two sequences of acquired samples were compared with the sequence of the samples of the prepared voltage waveform in terms of correlation coefficients. In offline analysis, three correlation coefficients were calculated and compared when the NIR LED is on and when it is off. The correlation coefficient between the dry electrode and the prepared sample is ρD; the correlation coefficient between the wet electrode and the prepared sample is ρW; and the correlation coefficient between the wet electrode and the dry electrode is ρDW. To ensure reliability of the analysis, this test was repeated thrice, and the normalized waveforms were derived.

2) Arterial occlusion experiment

The hemodynamic response of the proposed system was verified by evaluating the fNIRS responsivity using an arterial occlusion experiment [17], [18]. The experiment can be fulfilled using an inflatable arm cuff and a sphygmomanometer. The arm cuff could be shrunk to block arterial blood flow to artificially change the concentration of oxy and deoxy hemoglobin in the bloodstream on the arm. This would enable us to verify the hemodynamic behavior of the proposed system by observing this occlusion through NIRS data acquisition and offline analysis.

For the experiment, NIR LEDs and optodes were attached to a

Page 7 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

8

subject's arm in the layout shown in Fig. 4. The experiment was carried out for 5 min. Until the first minute of the experiment had elapsed, we waited for constriction of the cuff for the baseline observation. After 1 min, the pressure was increased to 200 mmHg for 6 s and maintained for 2 min, and then, the contraction was relaxed. During the experiment, the optode measurements were converted to the hemodynamic responses (ΔOHb, ΔRHb, and ΔTHb) according to (3) and (4) in Section II-E. Through offline analysis, recorded responses were filtered with a 4th order zero-phase Butterworth 0.2-Hz low-pass filter and normalized responsivities for all channel measurements were derived.

G. Human Subject Studies-Alpha Rhythm Detection Test and Mental Arithmetic Experiment

Although the evaluation and verification of EEG and fNIRS acquisition system were done through the EEG phantom and fNIRS responsivity tests, a human subject experiment was also carried out to evaluate the proposed system. To this end, an alpha rhythm detection test and a mental arithmetic experiment were used. The first is a basic level test to determine whether the system is effective for EEG acquisition. The second is a more challenging experiment to establish whether the proposed system can be used to discern the subtle difference in the EEG and fNIRS signal patterns when the brain engages in non-trivial mental activity, i.e., a mathematical subtraction operation.

The alpha rhythm is the most well-known EEG feature that can be easily detected when the user closes his or her eyes. When the eyes are closed, the spectral power of the alpha rhythm band (8~15 Hz) is amplified relative to the other spectral ranges. By comparing the spectral power when the eyes are closed and when they are not, the detection capability of real EEG features for the proposed system can be verified. This test, which consists of 10 trials, was performed on a subject. One trial consists of maintaining the eye-open state for 12.5 ± 2.5 seconds and the eye-closed state for 10 seconds. In every transition of the command, a beep sound was used to alert the subject to the change of instruction.

The mental arithmetic experiment is designed to examine the brain activation that occurs when subjects are required to carry out non-trivial mathematical operations. During a subtraction operation, brain activation appears in the form of event-related desynchronization (ERD) or event-related synchronization (ERS) [33]. ERD and ERS are known as spectral enhancement and suppression of the measured EEG signals. The brain activation also appears in the form of hemodynamic difference between the tasking and the non-tasking state. This difference appears as a hemodynamic concentration change of the oxy- and the deoxy-hemoglobin measured by the fNIRS system [32]. When mathematical operations are performed these activations can be observed by time-frequency analysis of the EEG measurements and time-course analysis of the fNIRS measurements.

Two additional subjects were recruited for the alpha rhythm detection test; thus, a total of three subjects voluntarily participated in this experiment. All subjects (three males, average age: 26.3 ± 1.7 years old) were healthy and had no

record of neurological and psychiatric disorders. Each subject was given a summary of the experiment and signed a consent form before their participation started.

The subjects were seated on a chair in front of a 24-inch LCD monitor. Prior to the experiment, pilot signal monitoring was performed to check the adhesion state of the probe set and baseline noise characteristics of the acquired EEG signals. The experiment consisted of two sessions, and each session consisted of 10 trials. In a trial, a white fixation cross was displayed while waiting for the next task period in the first 22.5 ± 2.5 s. In this resting state, the subjects were instructed to gaze at the center cross sign and to refrain from think anything to maintain a low mental load. During the next task period, the subjects were instructed to cumulatively subtract a two-digit prime number (ranging from 10 to 30) from a three-digit number in the range 500 to 999 for 20 s. For example, a problem of subtracting 13 from 700 is presented via a computer screen to the subject, i.e., “700 – 13”. The subject does the calculation inside his/her head. Once he/she has gotten the answer to the problem, 687 = 700 – 13, memorized it and continued on to subtracting another 13 from the answer, i.e., “687 – 13”. This is continued until the end of task period.

EEG measurements were conducted by attaching 16 dry electrodes to the scalp with the fabricated EEGCAP. To observe the task-related activation in the overall brain areas, 16 electrode positions covering the frontal (Fz, F3, F4, Fc1, Fc2, Fc5, and Fc6), motor/temporal (C3 and C4), and parietal (Pz, P3, P4, Cp1, Cp2, Cp5, and Cp6) regions were carefully chosen in accordance with the international 10-20 system. Reference and bias electrodes were also attached to the skin behind the left and right earlobes, respectively, using disposable wet electrodes. The EEGCAP equipped with dry electrodes was fastened to a strap on the subject’s chest. Two NIR LEDs and six optodes were also installed on the forehead using double-sided adhesive tape according to the probe layout in Fig. 4. The EEG and fNIRS measurements acquired by the installed electrodes and optodes were simultaneously recorded with event trigger in real time using MATLAB 2014a.

The acquired EEG and fNIRS datasets were analyzed offline using MATLAB 2014a and EEGLAB toolbox [34]. The EEG datasets were obtained from both the alpha rhythm detection test and mental arithmetic experiments. Each EEG dataset was bandpass filtered with a 4th order zero-phase Butterworth filter of bandpass in the range 0.5–40 Hz. From the filtered dataset, each epoch before and after task onset (-10 to +10 s for the alpha rhythm detection dataset and -15 to +15 s for the mental arithmetic experiment dataset) was extracted based on the recorded event trigger. Noisy and abnormal measurements among the analysis results were eliminated by manually inspecting the datasets and discarding four EEG channels from each dataset. An EEGLAB built-in function is utilized to investigate ERD/ERS patterns for the time-frequency analysis. To obtain the grand-averaged ERD/ERS patterns for each experiment, we averaged the time-frequency decomposition outcomes for all sessions and all subjects who participated. The grand-averaged characteristics are finally visualized for each experiment. The fNIRS datasets were only obtained from the

Page 8 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

9

mental arithmetic experiments. The fNIRS datasets, which comprise the relative concentration changes of oxy-, deoxy- and total hemoglobin (ΔOHb, ΔRHb, and ΔTHb), were calculated in real-time from the raw NIR light measurements. A 4th order zero-phase 0.01–0.2 Hz Butterworth bandpass filter was applied to the datasets and each epoch was extracted similarly to the EEG pre-processing procedure. Baseline

correction of the extracted epoch was performed by subtracting the averaged fNIRS data measured between -5s and 0s from the onset of a task. Identification of the hemodynamic trends during arithmetic operations was achieved by averaging the time courses of ΔOHb, ΔRHb, and ΔTHb to finally visualize the grand-averaged results.

III. RESULTS

A. Instrument Design

Images of the circuit boards of the proposed HBM system for EEG and fNIRS acquisition are shown in Fig. 7. Two four-layered 70 × 70 mm PCBs were fabricated for 16-channel EEG and 8-channel fNIRS acquisition. These boards are connected to each other through the board-to-board connector and are powered by two 2000mAh lithium polymer batteries. Sixteen-channel dry electrodes with 18 spring-loaded probes were installed in the EEGCAP, as shown in Fig. 8. Six-channel NIR optodes and 2-channel NIR LEDs were also fabricated as depicted in Fig 3. In the experiment involving human subjects, installation of the dry electrodes fNIRS probe set was easily accomplished by attaching the set of optodes and NIR LEDs to the subject’s forehead and by making the subject wear the EEGCAP equipped with dry electrodes.

B. Dry-electrode Evaluation

The evaluation results for each electrode comparison set (dry electrode vs. raw signal, wet electrode vs. raw signal, and dry electrode vs. wet electrode) using the EEG phantom are summarized in Table I. The ρDW close to one indicates that the dry and wet electrodes are detected almost the same waveform regardless of the activation of NIR LEDs. This confirms that the dry electrode is capable of obtaining EEG signals without the use of conductive gels and provides almost the same EEG measurement results as the wet electrode. The ρD and ρW above 0.9 indicate that the phantom measurements through the dry and wet electrodes are not significantly different from the raw signal data. The slight decrease in the correlation coefficient, compared to ρDW, is considered to be caused by the error that occurs in the waveform-reduction process using the

TABLE I CORRELATION COMPARISON FOR ARTIFICIALLY GENERATED EEG

RECORDING

NIR LED states Dρ

Wρ

DWρ

On 0.9422 0.9423 0.9995

Off 0.9433 0.9437 0.9996

Fig. 7 (a) Comparison of the recorded waveforms of artificially generated EEG signals, (b) Normalized hemodynamic responses over the six optodes with an arterial occlusion experiment.

0 50 100 150 200 250 300-0.1

-0.05

0

0.05

0.1

0.15

0.2

0.25

0.3

Time(sec)

Con

cent

ratio

n(m

M/D

PF)

NIRS Arm Cuff Test

Constriction Period ∆OHb

∆RHb∆THb

0 10 20 30 40 50 60-150

-100

-50

0

50

100Dry electrode measurement Vs. Raw EEG signal

Time(sec)

Vol

tage

(uV

)

DryRaw

(a) (b)

Fig. 8. Front and inner view of rubber EEGCAP with 16 channel dry electrodes. The dry electrodes were tightly fixed in the electrode positioning holes for correct electrode placement.

Fig. 6 Main board and slave board design of the proposed HBM system.

(1) Main Board (2) Slave Board

Page 9 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

10

voltage-divider circuit during artificial EEG generation. The raw signal and waveform generated by the dry electrode

are depicted in Fig. 6(a). The signal recorded at the dry electrode has an overall polarized offset as compared to the original signal; however, the overall flow of the waveform is not significantly different as a result of the calculated correlation coefficient.

In addition, an input-referred noise in the EEG acquisition circuit was evaluated using a spare input channel of the ADS1299 during the EEG phantom test. The bipolar input channel can be shorted internally by setting the input multiplexer of the channel to the input short state. This configuration allows investigating the noise characteristics of the input channel. When the NIR LED is activated, the input-referred noises of 0.141 μVRMS and 1.066 μVpp were calculated from the offline analysis of the input short channel data. These results verified that the proposed system closely achieved the low-noise characteristics of 0.14 μVRMS and 0.98 μVpp (at a sampling rate of 250 Hz and PGA gain of 24) as specified in the ADS1299 datasheet [24].

C. fNIRS Response Evaluation

From the offline analysis of the arterial occlusion experiment data, the normalized ΔOHb, ΔRHb, and ΔTHb levels for the six channels are depicted in Fig. 6(b). All hemodynamic responses

converge towards a baseline within ±0.02 mM / DPF during the first 60 s before contraction of the cuff and increase rapidly over 6 s when the cuff is inflated. When the contraction is complete, the inflowing arterial blood is almost blocked and therefore, the ΔOHb and ΔRHb are linearly diverged until the moment the cuff is released. The slope of the ΔOHb and ΔRHb are -0.7uM/DPF·s and +1.4uM/DPF·s, respectively. When the pressure on the cuff is released to allow the arterial blood flow to return, the ΔOHb and ΔRHb dramatically converge and overshooting occurs. After peaking to the opposite overshoot, all hemodynamic responses gradually converge to the steady state. Compared with the previous studies [17], [18], in which the same experiment was conducted, the results of this experiment demonstrate that the proposed HBM system is sufficiently responsive to analyze the changes in the hemoglobin concentration.

D. Grand-averaged Analysis for Human Subject Studies

The grand-averaged time-frequency analysis results and a comparison of the normalized spectra for the alpha rhythm detection test are depicted in Fig. 9 (a) and (c). The vertical dashed lines on the time-frequency analysis plot at 0 seconds denote the onset of the task period (eyes closed instruction for alpha rhythm detection test and cumulative subtraction instruction for mental arithmetic experiments).

Fig. 9. dB scale grand averaged time-frequency analysis results for the alpha rhythm detection test (a) and mental arithmetic experiments (b). Vertical dashed lines indicate task onset. Red and blue zones mean the time and frequency ranges associated with high event-related synchronization (ERS) and desynchronization (ERD). Spectral comparisons (c) and (d) are normalized spectra for each task states ((c) eye open states vs. eye closed states, (d) arithmetic operating states vs. resting states)

5 10 15 20 25 3010-4

10-3

Normalized Spectrum

Frequency(Hz)

Nor

mal

ized

Pow

er

Eyes ClosedEyes Open

5 10 15 20 25 30

10-4

10-3Normalized Spectrum

Frequency(Hz)

Nor

mal

ized

Pow

er

Arithmetic OperationResting States

(a)

(c)

(b)

(d)

Time(sec)

Freq

uenc

y(H

z)Time-frequency Analysis

-5 0 55

10

15

20

25

30

-5

0

5

Time(sec)

Freq

uenc

y(H

z)

Time-frequency Analysis

-10 0 105

10

15

20

25

30

-3

-2

-1

0

1

2

3

Page 10 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

11

In the alpha rhythm detection test, the event-related synchronization (ERS) pattern evoked by the instruction to close the eyes is clearly indicated with higher spectral power (red zones) in the alpha rhythm placed in the 8-13 Hz bands compared to the baseline spectral power of -7.5 to -2.5 s. The high spectral power of the beta rhythm at approximately 20 Hz also appears at the beginning of the task as a harmonic distortion related to the high spectral power of the alpha rhythm. The dB scale comparison of the normalized spectrum graphs was used to compute the first and second maximum ERS intensities as 4.17 dB at 10.3 Hz and 2.56 dB at 21.04 Hz. Based on these results, which show that the alpha rhythm associated with closure of the eyes can be detected by using spectral analysis, it is evident that the proposed system can appropriately acquire the general EEG feature signals.

The grand-averaged time-frequency analysis results and comparison of the normalized spectra during the mental arithmetic experiments are depicted in Fig. 9 (b) and (d). On the basis of the time-frequency analysis results compared with the alpha rhythm detection test, it is evident that reversed patterns of the spectral perturbation are observed, compared to the baseline spectral power during the period -15 to -5 s before task onset. Main and weaker event-related desynchronization (ERD) patterns are also observed in the alpha rhythm at approximately 10 Hz, and in the wide beta rhythm range 18–25 Hz during the arithmetic operating state. The dB scale comparison of the normalized spectrum graphs was used to compute the maximum ERD intensity of -2.74 dB at 10.58 Hz in the alpha rhythm range. The second highest ERD intensity is -2.40 dB at 19.73 Hz in the beta rhythm range.

The grand-averaged time courses of the concentration changes in oxy-, deoxy- and total hemoglobin (ΔOHb, ΔRHb, and ΔTHb) in the mental arithmetic experiments are plotted in Fig. 10. During the cumulative subtraction task, which is given to the subject to increase the workload level of the brain, we can found clear decrease trend of ΔOHb. The diminished ΔOHb level is then rapidly restored again to the resting state after the task periods. On the contrary, ΔRHb shows a weaker inverse

pattern and more delayed response compared to the ΔOHb trend. The lowest ΔOHb is recorded just before the end of the task, whereas the ΔRHb trend continues to increase slightly after the task period. This ΔRHb trend begins to decrease belatedly at 8 s after the end of the task. These analysis results show that the ΔOHb pattern much more closely reflects the mental workload level than the weaker ΔRHb response and the ΔTHb pattern also follows the more dominant ΔOHb trend.

The EEG and fNIRS responses in the mental arithmetic experiments provided the brain activation responses such as the ERD pattern on the alpha and beta rhythm bands and the decreasing trend of the ΔOHb response. These results were compared with those obtained in the previous study [35], in which similar experiments were conducted using commercial equipment. Based on our studies with human subjects, we can conclude that the proposed HBM system has sufficient capabilities to simultaneously monitor EEG and fNIRS signals.

IV. DISCUSSION The proposed HBM system includes highly integrated AFE

ICs for bioelectrical and bio-optical signal acquisition to reduce the design complexity and ensure reliable performance. A comparison of the ADS1299 and ADS8688A ICs revealed the following differences.

The ADS8688A was operated using a sequential sampling approach by employing a single multiplexed ADC. EEG signals were acquired through continuous sampling by multi-channel electrodes, whereas fNIRS measurements were acquired through instantaneous sampling of a specific optode set when the NIR LEDs were activated. This discontinuous sampling approach obviates the need to use many built-in ADCs to simultaneously sample all channels used for fNIRS acquisition. The sequential sampling approach using a single multiplexed ADC is sufficient to enable the achievement of multi-channel, high-precision measurement by averaging the results from repeated measurements.

In addition, ADS8688A is based on the SAR ADC architecture that is different from the delta–sigma architecture employed in ADS1299 [36]. The operating mechanism of the delta–sigma ADC requires a digital decimation filter for noise-shaped representation of oversampled data, thereby resulting in conversion latency known as the settling time [37], which represents the delay between the beginning of the input signal conversion and the end time at which fully settled output data are available. Because it is necessary to sample fNIRS measurements with precise timing, in accordance with the LED switching sequence, delta–sigma ADCs incapable of avoiding conversion delays are not suitable for fNIRS data acquisition. Unlike delta–sigma ADCs, the SAR ADC architecture exhibits zero operational delay, thereby producing output data by iteratively comparing the reference voltage through the sample and hold circuits, a comparator, and digital-to-analog converter (DAC). This zero cycle latency feature leads to reliable timing performance during fNIRS measurements, as digitized data are produced within a short cycle period.

In the proposed hybrid data acquisition system, the acquisition of data by ADS8688A and NIR LED control is

Fig. 10 Grand-averaged time courses of concentration changes in oxy-, deoxy- and total hemoglobin (ΔOHb, ΔRHb, and ΔTHb) for mental arithmetic experiments.

0 10 20 30 40-6

-5

-4

-3

-2

-1

0

1

Time(sec)

Con

cent

ratio

n(uM

/DP

F)

Time courses of ∆OHb, ∆RHb, ∆THb

Task Period

∆OHb∆RHb∆THb

Page 11 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960

For Review Only

Submitted to IEEE Transactions on Biomedical Engineering

12

synchronized with DRDY signal generation, which occurs at the sampling rate of ADS1299. This sequence of operations serves to maintain full synchronization at all instants regardless of the dedicated clock tolerance of AFE ICs.

V. CONCLUSION The ability of the hybrid brain monitoring system proposed in

this study to realize high-resolution, multi-channel, and fully synchronized EEG and fNIRS monitoring was qualitatively and experimentally verified. The system readily supports synchronous operation controlled by a single microcontroller unit, whereas electrical and hemodynamic activities are separately monitored by individually dedicated AFE ICs. Each acquisition circuit comprises state-of-the-art components to facilitate miniaturization, low design complexity, and reliable low-noise performance. At the same time, the use of dry electrodes maximizes system usability in out-of-lab applications. The proposed system enables simultaneous measurement of 16 EEG and 8 fNIRS channels by using 2 near-infrared LEDs and 6 monolithic photodiodes. The acquisition of EEG and fNIRS measurements was evaluated by conducting an EEG phantom test using artificially generated EEG signals and an arterial occlusion experiment. Additionally, an alpha rhythm detection test and mental arithmetic experiments were performed to assess the practical capabilities of the proposed system for human subject studies. The grand-averaged results of the time-frequency analysis for EEG measurements and time courses for NIRS measurements verified that the proposed HBM systems are suitable for use in real BCI applications.

REFERENCES [1] J. R. Wolpaw et al., “Brain-computer interfaces for communication and

control,” Clinical Neurophysiology, vol. 113, no. 6, pp. 767–791, 2002. [2] L. F. Nicolas-Alonso and J. Gomez-Gil, “Brain Computer Interfaces, a

Review,” Sensors (Basel), vol. 12, no. 2, pp. 1211–1279, Jan. 2012. [3] J. d. R. Millán et al., “Combining Brain–Computer Interfaces and

Assistive Technologies: State-of-the-Art and Challenges,” Front Neurosci, vol. 4, Sep. 2010.

[4] M. D. Murphy et al., “Current Challenges Facing the Translation of Brain Computer Interfaces from Preclinical Trials to Use in Human Patients,” Front. Cell. Neurosci., vol. 9, 2016.

[5] C. Vidaurre and B. Blankertz, “Towards a Cure for BCI Illiteracy,” Brain Topogr, vol. 23, no. 2, pp. 194–198, 2010.

[6] G. Pfurtscheller et al., “The Hybrid BCI,” Front Neurosci, vol. 4, Apr. 2010.

[7] N. Weiskopf, “Real-time fMRI and its application to neurofeedback,” NeuroImage, vol. 62, no. 2, pp. 682–692, Aug. 2012.

[8] T. Nguyen et al., “Utilization of a combined EEG/NIRS system to predict driver drowsiness,” Scientific Reports, vol. 7, p. srep43933, Mar. 2017.

[9] A. Machado et al., “Detection of hemodynamic responses to epileptic activity using simultaneous Electro-EncephaloGraphy (EEG)/Near Infra Red Spectroscopy (NIRS) acquisitions,” NeuroImage, vol. 56, no. 1, pp. 114–125, May 2011.

[10] N. Naseer and K.-S. Hong, “fNIRS-based brain-computer interfaces: a review,” Front. Hum. Neurosci., vol. 9, 2015.

[11] S. Amiri et al., “A Review of Hybrid Brain-Computer Interface Systems,” Advances in Human-Computer Interaction, vol. 2013, p. e187024, 2013.

[12] S. Fazli et al., “Enhanced performance by a hybrid NIRS–EEG brain computer interface,” NeuroImage, vol. 59, no. 1, pp. 519–529, Jan. 2012.

[13] Y. Tomita et al., “Bimodal BCI using simultaneously NIRS and EEG,” IEEE Trans Biomed Eng, vol. 61, no. 4, pp. 1274–1284, Apr. 2014.

[14] R. J. Cooper et al., “Design and evaluation of a probe for simultaneous EEG and near-infrared imaging of cortical activation,” Phys. Med. Biol., vol. 54, no. 7, p. 2093, 2009.

[15] E. Lareau et al., “Near infrared spectrometer combined with multichannel EEG for functional brain imaging,” in 2011 5th International Symposium on Medical Information and Communication Technology, 2011, pp. 122–126.

[16] E. Lareau et al., “Multichannel wearable system dedicated for simultaneous electroencephalography/near-infrared spectroscopy real-time data acquisitions,” J. Biomed. Opt, vol. 16, no. 9, pp. 096014-096014–14, 2011.

[17] J. Safaie et al., “Toward a fully integrated wireless wearable EEG-NIRS bimodal acquisition system,” J. Neural Eng., vol. 10, no. 5, p. 056001, 2013.

[18] M. Sawan et al., “Wireless Recording Systems: From Noninvasive EEG-NIRS to Invasive EEG Devices,” IEEE Transactions on Biomedical Circuits and Systems, vol. 7, no. 2, pp. 186–195, Apr. 2013.

[19] Q. Zhang et al., “Twenty-four-hour ambulatory recording of cerebral hemodynamics, systemic hemodynamics, electrocardiography, and actigraphy during people’s daily activities,” J Biomed Opt, vol. 19, no. 4, p. 47003, Apr. 2014.

[20] A. von Lühmann et al., “M3BA: A Mobile, Modular, Multimodal Biosignal Acquisition Architecture for Miniaturized EEG-NIRS-Based Hybrid BCI and Monitoring,” IEEE Transactions on Biomedical Engineering, vol. 64, no. 6, pp. 1199–1210, Jun. 2017.

[21] S. Lee et al., “Dry electrode design and performance evaluation for EEG based BCI systems,” in 2013 International Winter Workshop on Brain-Computer Interface (BCI), 2013, pp. 52–53.

[22] L.-D. Liao et al., “Design, Fabrication and Experimental Validation of a Novel Dry-Contact Sensor for Measuring Electroencephalography Signals without Skin Preparation,” Sensors, vol. 11, no. 6, pp. 5819–5834, May 2011.

[23] S. Karthik and B. Mark, “Analog Front-End Design for ECG Systems Using Delta-Sigma ADCs,” Texas Instruments, Application Report SBAA160A, Mar. 2009.

[24] ADS1299 Datasheet. Texas Instruments. [25] ADS8688 Datasheet. Texas Instruments. [26] S. Pithadia and S. More, “Grounding in mixed-signal systems

demystified, Part 1,” Analog Applications, 2013. [27] W. Alexander and P. Alexander, “High-Speed Layout Guidelines,”

Texas Instruments, Application Report SCAA082, Nov. 2006. [28] H. Zumbahlenas and Analog Devices, inc, Eds., Linear circuit design

handbook. Amsterdam ; Boston: Elsevier/Newnes Press, 2008. [29] G. Strangman et al., “Non-invasive neuroimaging using near-infrared

light,” Biol. Psychiatry, vol. 52, no. 7, pp. 679–693, Oct. 2002. [30] J. G. Kim and H. Liu, “Variation of haemoglobin extinction

coefficients can cause errors in the determination of haemoglobin concentration measured by near-infrared spectroscopy,” Phys. Med. Biol., vol. 52, no. 20, p. 6295, 2007.

[31] L. Kocsis et al., “The modified Beer–Lambert law revisited,” Phys. Med. Biol., vol. 51, no. 5, p. N91, 2006.

[32] Y. Fukui et al., “Monte Carlo prediction of near-infrared light propagation in realistic adult and neonatal head models,” Appl. Opt., AO, vol. 42, no. 16, pp. 2881–2887, Jun. 2003.

[33] G. Pfurtscheller and F. H. Lopes da Silva, “Event-related EEG/MEG synchronization and desynchronization: basic principles,” Clinical Neurophysiology, vol. 110, no. 11, pp. 1842–1857, Nov. 1999.

[34] A. Delorme and S. Makeig, “EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis,” Journal of Neuroscience Methods, vol. 134, no. 1, pp. 9–21, Mar. 2004.

[35] J. Shin et al., “Evaluation of a Compact Hybrid Brain-Computer Interface System,” BioMed Research International, vol. 2017, p. e6820482, 2017.

[36] W. Kester and Analog Devices, inc, Eds., Data conversion handbook. Amsterdam ; Boston: Elsevier ; Newnes, 2005.

[37] B. C. Baker, “Conversion latency in delta-sigma converters.” Texas Instruments, 2007.

Page 12 of 12

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960