Development of Control Architectures

for Multi-Robot Agricultural Field Production Systems

Santosh K. Pitla, Ph.D.

spitla2@unl.edu

Assistant Professor

Department of Biological Systems Engineering

University of Nebraska-Lincoln

1

IEEE RAS Agricultural Robotics & Automation

Technical Committee

Webinar #013

December 2013

Overview

Education and Research Background

Control Architectures

Individual Robot Control Architecture (IRCA)

Multi Robot System Control Architecture (MRSCA)

Autonomous Vehicle Platform (AVP) Development

Validation of Control Architectures

Post-Doctoral Research

Future Work at UNL

Q&A

2

Education and Research Background

2000 to 2004 – BS, Mechanical Engineering, Osmania University, Hyderabad, India

Aug 2004 to May 2007 - MS, Mechanical Engineering and Biosystems and Agricultural Engineering University of Kentucky, Lexington, KY

Thesis Title: Development of an Electro-Mechanical System to Identify and Map Soil Compaction

March 2007 to April 2012 – Research Engineer, Machine Systems and Automation, BAE, University of Kentucky

Jan 2008 to Jan 2012- Ph.D. Program

May 2012 to Present, Post-Doc, The Ohio State University

OCT 2013 to Present, Assistant Professor and Advanced Machinery Engineer, University of Nebraska-Lincoln

3

Control Architectures

4

Weeding Robot (Madsen

and Jakobsen, 2001)

Autonomous Harvester

(Pilarski et al., 2002)

Citrus Fruit Harvesting Robot

(Hannan et al., 2004)

Mato Grosso,Brazil

Un-manned agricultural machines

Road blocks Cost

Safety

Intelligence

Control architecture work is underway (Brooks, 1986; Arkin, 1990; Yavuz and Bradshaw, 2002;

Blackmore et al., 2002; Torrie et al., 2002; and Mott et al.,

2009)

Individual Robot Control Architecture

Multi-Robot System Control Architecture

Control

Architectures

Agricultural Machines

Agricultural Robots

Paradigm Shift (Big Vs. Small)

Paradigm Shift (Big Vs. Small)

6

Spray material Application Variation (Luck and Pitla , 2010)

Individual Robot Control Architecture (IRCA)

7

Individual Robot Control Architecture (IRCA)

Intelligence

IRCA (continued)

Sensing Layer (SL)

Sensor Stack

Array of sensors that aid the robot in learning

about the unknown environment

Wireless Communication Module (WCM)

Processes the information obtained wirelessly

from remote or off-machine entities

The SL receives the environment data obtained from the sensor stack (on-machine) and

the WCM (off-machine) and passes the information to the BL for further processing and

filtering

IRCA (continued) Behavior Layer (BL)

Deliberative Behavior

High level decision making processes that

require planning and algorithm execution

Reactive Behavior

Low level processes that do not require considerable

computation but are crucial for safety and operability

By-Pass of Control

The switching of the control command generation source in

response to the changing environment

Individual Robot Control Architecture (IRCA)

10

FSM I

Desired Control Commands

FSM IIFSM III

FSM IV

IRCA (continued)

FSML Simulation (MATLAB)

Inputs Scenario I Scenario II Scenario III Scenario IV Scenario IV

dob (m) 5 5 3 5 5

TuF 0 1 1 0 0

SAd (deg) ±0.125 ±2.5 - ±0.125 ±0.125

SAr (deg) - - ± 5 - -

Eflag 0 0 0 0 1

ReF 0 0 0 0 0

TrF 0 0 0 0 0

Five scenarios for FSM simulation

Input signals created using Signal Builder (MATLAB)

corresponding to the five scenarios

11

FSML created using the StateFlowChart tool

IRCA (continued)FSML simulation

SIMULINK model created for FSML simulation

Internal States Trigger conditions

FSM I Cruise

TuF~=1 && dob>=4 && Eflag

~=1

Slow TuF==1 && dob>=4

Safe Speed

TuF>=0 && dob<4 &&

dob>=2

Dead dob<2 || Eflag==1

FSM II Navigate TuF>=0 && dob>=4

Safe Navigate TuF>=0 && dob<4

FSM III Lower (1)

ReF~=1 &&

Eflag~=1&&TrF~=1&& dob>=4

Raise (0)

ReF==1 || Eflag==1||

TrF==1|| dob<4

FSM IV ON (1)

ReF~=1&&Eflag~=1&&TrF~=

1&&dob>=4

OFF (0)

ReF==1||Eflag==1||TrF==1||d

ob<4 12

Mutually Exclusive States

Parallel States

IRCA (continued)FSM Simulation Results and Discussion

Active states of FSML (I to IV) in the default state Actives states in Scenario I

Actives states in Scenario III 13

IRCA (continued)FSM Simulation Results and Discussion

Active states of FSML (I to IV)

in Scenario I

FSM simulation outputs (desired control commands)

Scenarios Simulation

time (s)

Active Internal States

(FSM I to IV)

Desired Control

Commands

Scenario I 0-10 Cruise, Navigate, Lower, ON 4 km/h, ±0.125o, 1, 1

Scenario II 10-20 Slow, Navigate, Lower, ON 2 km/h, ± 2.5 o, 1, 1

Scenario III 20-30 Safe Speed, Safe Navigate, Raise, OFF 1 km/h, ± 5o, 0, 0

Scenario IV 30-40 Cruise, Navigate, Lower, ON 4 km/h, ±0.125o, 1, 1

Scenario V 40-50 Dead, Navigate, Raise, OFF 0 km/h, ±0.125o, 0, 0

14

Multi-Robot System Control Architecture (MRSCA)

15

MRS planting in unique

work zones (WZ I to WZ III)

No Cooperation

Coordination Strategy

(No Cooperation)

Control Variables (m, r, c)

Stand Alone Behavior (0,0,0)

Control Variables:

m = Mode (values: 0,1,2)

r = Role (values: 0,1,2)

c = Communication (values: 0,1)

No Cooperation Mode

Stand Alone Role Transmit

MRSCA (continued)

16

MRS performing baling and retrieval operations

Modest Cooperation

Coordination

Strategy (Modest

Cooperation)

Control

Variables (m, r, c)

Leader (Baler) (1,1,0)

Follower (Bale Spear) (1,2, 1)

MRSCA (continued)

17

MRS performing Harvesting Operation

Absolute Cooperation

Coordination

Strategy (Absolute

Cooperation)

Control

Variables (m, r, c)

Leader (Combine) (2,1, 0)

Follower (GC-I, GC-II) (2,2, 1)

MRSCA (continued)

18

MRSCA (continued)

19

Coordination Strategy Control Variables

(m, r, c)

No Cooperation (0,0,0)

Modest Cooperation (1,1,0), (1,2, 1)

Absolute Cooperation (2,1, 0), (2,2, 1)

Global Information

Module

MRSCA (continued)

20

GIM

Role

CoordinationWireless

Communication

StandAlone Leader

Follower

No Cooperation Modest

Cooperation

Absolute

CooperationTx Rx

Wait UnloadGo To/

Retrieve

Follow/

Load

Unload

Parallel Finite States

Internal Finite States – Level I

Internal Finite States – Level II

Hierarchy of the FSMs in GIM.

MRSCA (continued)

Status Flag Definition High Low

SA Raised when role of the robot is Stand Alone 1 0

L1Raised when the robot is performing a leader role with

the task Wait is active 1 0

L2Raised when the robot is performing a leader role with

the Unload task active 1 0

F1Raised when the robot is performing a follower role

and the task Goto is active 1 0

F2Raised when the robot is performing a follower role

and the task Follow/Load is active1 0

F3Raised when the robot is performing a follower role

and the task Unload is active 1 0

21

Active States during Simulation in Stand Alone Mode

MRSCA (Continued)

External Wireless Input

StatMsgF.mat

To File1

StatMsgL.mat

To File

ScopeLeader

ScopeFollower

m

r

c

BL

Inputs-IRCA (Leader)

m

r

c

BD

FE

BW

Inputs-IRCA (Follower)

m

r

c

BL

SA

L1

L2

F1

F2

F3

Global Information Module

Baler (Leader)

m

r

c

BD

FE

BW

BLf

SA

L1

L2

F1

F2

F3

Global Information Module

Bale Retriever (Follower)

SA

L1

L2

F1

F2

F3

SAf

L1f

L2f

F1f

F2f

F3f

BL

22

Generic Flags Definition High Low

BL, BLf

Raised when the bale is ready to be

retrieved1 0

BDRaised when the Bale Retriever is closer

to the hay bale to be retrieved1 0

FERaised when the Bale Retriever is close

to the field edge for dropping of the bale1 0

BWRaised when the bale is loaded on the

Bale Retriever1 0

MRSCA (Continued)Modest Cooperation (Baling – Bale Retrieving)

23

(a) (b) (c)

Active states of the Baler (Leader)

Active tasks a) Go To; b) Load; and c) Unload of the Bale Retriever

during the execution of the bale retrieving task

MRSCA (Continued)

External Wireless Input

(Follower 1)

External Wireless Input

(Follower 2)

StatMsgF2.mat

To File2

StatMsgF1.mat

To File1

StatMsgL.mat

To File

ScopeLeader

ScopeFollower2

ScopeFollower1

m

r

c

od

PS

FE

InputsF2 (IRCA )

m

r

c

od

PS

FE

InputsF1 (IRCA)

m

r

c

PS

Inputs (IRCA)

m

r

c

PS

SYNCL2

SYNCL1

SA

L1

L2

F1

F2

F3

Global Information Module

Leader

m

r

c

od

PS

FE

SA

L1

L2

F1

F2

F3

SYNCF2

Global Information Module

Follower2

m

r

c

od

PS

FE

SA

L1

L2

F1

F2

F3

SYNCF1

Global Information Module

Follower1

SA

L1

L2

F1

F2

F3

SAf1

L1f1

L2f1

F1f1

F2f1

F3f1

Sf1

Sf1

SAf2

L1f2

L2f2

F1f2

F2f2

F3f2

Sf2

Sf2

Absolute Cooperation

(Harvesting)

24

Flags Definition High Low

PSRaised when the Grain Carts are full with grain or

when the grain is available in the Combine for unloading1 0

ODRaised when the Grain Cart is at a desired bearing

(heading and location) relative to the Combine1 0

SYNCF1/S

YNCL1

Raised when Grain Cart I wants to synchronize with

the Combine for the transfer of grain1 0

SYNCF2/S

YNCL2

Raised when Grain Cart II wants to synchronize with

the Combine for the transfer of grain1 0

MRSCA (Continued)

25

MRSCA (Continued)

26

(a) (b)

MRSCA (Continued)

27

Autonomous Vehicle Platform (AVP)

Development

28

AVP framework: (a) solid model of basic frame, (b) fabricated AVP frame with

mechanical components

Drive

MotorDifferentialFlexible

Coupling

(a) (b) (c)(a) Drive motor mounting and roller chain drive to differential at rear axle (b) 24 VDC

steering actuator, (c) steering actuator and linkage mounted on the front axle of the AVP

Ground speed sensor: (a) schematic and wiring diagram and (b) actual

mounting location and configuration.

Infra-Red

Sensor

Orthogonal Steel Plate

Oscillating DC Motor

(a) (b) (c) (d)

Infra-red sensor array construction: a) solid model, b) SHARP GP2Y0A700K NIR sensor, c)

OEM 212 series oscillating motor drive, d) assembled sensor array mounted on the AVP

AVP Development (continued)

MC012-010 Plus+1 microcontroller

(Sauer Danfoss, MN)

Speed

Controller

Steering

Controller

System

Controller

IRF

Controller

IRR

Controller

CAN Bus

Te

rmin

ato

r

Te

rmin

ato

r

IXF Module IXR Module

AVP distributed controller network topology

Leaf Light HS (Kvaser, Sweden)

CAN to RS232 gateway

9XTend PKG (B&B Electronics

Manufacturing Co., Ottawa, IL) radio

frequency modem

Trimble AgGPS 132 (Trimble

Navigation Ltd., Sunnyvale, CA)

GPS engine and antenna

Task Computer for the AVP (Eee PC 1000, ASUSTeK

Computer, Inc., Peitou, Taipei, Taiwan, R.O.C)

29

AVP Development (continued)5

0 A

Rela

y

Manual

24 V

Key Switch

Steering Driver

M+

M-

V+

V-S + -

Speed Driver

M+

M-

V+

V-S + -

Auto

Fuse Block

10 A

Signal Manual

Signal Auto

30 A

5 A

5 A

CAN PWR

Wireless module

FWD

Bkwd

LeftRight

Stop

Stop

System

Controller

Inputs

4.4 KΩ 4.4 KΩ

Speed

Controller

Steering

Controller

(a)

(b)

AVP PDP (a) schematic of PDP, and (b)

completed PDP installed on AVP

30

RF Modem

GPS

RS 232 RS 232

CAN to Serial Converter

RS

23

2

Te

rmin

ato

r

Task Computer

Drive Motor Steering Actuator IR Sensor

FrontIR Sensor

Rear

Speed

Controller

Steering

Controller

System

Controller

IRF

Controller

IRL

Controller

CAN Bus

Te

rmin

ato

r

Component Map of the AVP

VB.Net User Interface

GUIDE software used for programming the Micro

Controllers

AVP Development (continued)

31

Multi-Robot System (MRS) created by replicating the AVP

Completely electric - 24 VDC

Operates in Manual and Automatic modes

On-board rechargers for Lead-Acid Batteries

CAN based distributed controller network

Ability to override the automatic mode via manual commands in the

event of emergency

Three way AVP safety – 1) IR sensory arrays, 2) onboard emergency

stops, and 3) wireless remote stop.

Task computer, GPS and wireless communications for automatic

operation.

Validation of IRCA

Deliberative and Reactive Behavior

32

20 21 22 23 24 25 26 27 28 29 305

10

15

20

25

30

35

40

Easting (m)

No

rth

ing

(m

)

AB Line Robot Path Obstacle

A

B

AVP tracking of AB line with obstacle in its path

Speed States of the AVP

Validation of MRSCA

Standalone Behavior

33

Automatic tracking of AB lines by three

AVPs simulating planters.

Validation of MRSCA (continued)

Modest Cooperation

34

20 22 24 26 28 30 325

10

15

20

25

30

35

40

45

Easting (m)N

ort

hin

g (

m)

Baler Path Bale Retriever Path

A

C

B

D

F

Bale drop off location

Field Edge

Motion paths of Baler and Bale Retriever AVPs

Validation of MRSCA (continued)

Absolute Cooperation

35Paths of Harvester, GC1 and GC2 AVPs during TS3.

Post-Doctoral Research

36

Next Gen Autonomous Plat Form (AVP – II)

Post-Doctoral Research

37

Controller Area Network (CAN) Data Acquisition from Field Machinery

ISO 11783 (Source: www.vector.com)

Anhydrous Applicator 16 row planter 12 row planter Sprayer

Data Acquisition from the ISO Diagnostic Port

(Tractor: MX340)

Post-Doctoral Research

38

CAN Data Acquisition

Screenshot of Vector CANalyzer Interface (Data collection from

CASE IH MX340 Tractor)

Decoded GPS CAN Data

GPS CAN Message

Time ID Data length D0 D1 D2 D3 D4 D5 D6 D7

0.044144 1 18FEF31Cx Rx d 8 E2 C8 3 95 F0 ED AE 4B

Post-Doctoral Research

39

CAN Data Acquisition

Work

PeriodTurning Dwell

Period

Future Work at UNL

Machine Automation and Agricultural Robotics for Row Crop and Bio Energy

Production

40

Moving Biomass Bales

from the field to On-Farm

Bio Mass Processing

Facility

Spreading the by-product

(fertilizer) in the field

Soil Sampling, selective spraying,

Weed Mapping, Crop Health

MRS Planting or Spraying

UAV (Source: www.Precision Hawk.com)

AR Drone 2.0 (www.ardrone.parrot.com/)

Thank You!!

Questions?

42