CHAPTER VI

GRouP - THEORETIC PROBI~I~S

One of the major motivating factors in the investigation of graph isomorphism is

the hope that a successful determination of its complexity status will provide new

insights into P vs. NP problem. In the preceding chapters, we have developed a

number of results which (partially) settle the complexity of testing isomorphism in

certain subclasses of graphs. We have concentrated on the group-theoretic approach

because it adds a wealth of algebraic properties which may be exploited when deter-

mining the automorphism group of a graph. In this development, group-theoretic

problems have assumed an auxiliary, although crucial role, and have served to reveal

algebraic properties of the problem which were exploited in the design of the algo-

rithms. In this chapter, we reverse this situation and discuss the complexity of

group-theoretic problems in the abstract. Although we take graph isomorphism and

other combinatorial problems as point of departure, these group-theoretic problems

are of interest in their own right, not only as mathematically interesting questions,

but also as algorithmic problems with considerable significance for Computer Sci-

ence. In particular, there is a rich set of such problems which appear to be natural

candidates for problems in NP that are neither in P nor are NP-eomplete~

I. S o m e Combinator ia l P r o b l e m s as Group-Theoretic P r o b l e m s

In this section, we translate a number of combinatorial problems, graph isomor-

phism for one, into purely group-theoretic questions. The purpose of this translation

is to expose a (sometimes hidden) algebraic nature of such problems, and to establish

a continuity of the present subject with the preceding ones.

In Chapter If, we interpreted the set of isomorphisms between two graphs as a

coset of the automorphism group. Our first task is to translate the problem of graph

isomorphism into a membership question in do~bL~ eos~ts of certain permutation

groups, thereby entirely voiding the problem of its topological and combinatorial

aspects.

232

Let A, B < S n be p e r m u t a t i o n g roups of d e g r e e n.

ing an e l e m e n t ~r o~ S n is t h e se t

The double ~oset of A and B con ta in -

ATrB= IaTrfi t a EA,~ ~ B t

of p e r m u t a t i o n s in S n.

In p a r t i c u l a r , t he se t AB = f aft ! a E A, fl ~ B ~ is t he double c o s e t of A and B con-

ta in ing the i d e n t i t y p e r m u t a t i o n . Since r i g h t eose t s of A a re e i t h e r d i s jo in t or equal,

the double c o s e t A~B is the union of some r igh t c o s e t s of A. S imi lar ly , i t is t he union

of c e r t a i n !eft eo se t s of B. Therefore , b o t h t he o r d e r of A and t h e o r d e r of B divide

the e a r d i n a l i t y of A~rB. Specif ical ly , one knows

The cardinality of the double coset A~rB is I A~B I = I AI' { B { IA"C~S I "

The l e m m a follows f rom L e m m a 20 of ChapLer III by pu t t i ng the e l e m e n t ~Trfl in

A~B into 1-1 c o r r e s p o n d e n c e with the e l e m e n t ~-la~rfl of A~B.

Double c o s e t s a r e e i t h e r d i s jo in t or equal. However, t he doub le e o s e t ATrB n e e d

no t be of c a r d i n a ! i t y equal to t he c a r d i n a t i t y of t he double c o s e t ACD. Thus, double

cose t s p a r t i t i o n the e l e m e n t s of Sn into c lasses of n o n u n i f o r m size. We f i rs t show t h a t

the m e m b e r s h i p p r o b l e m for c e r t a i n double cose t s is j u s t g r a p h i somorph i sm .

Let X = (V,E) be a g r a p h wi th n v e r t i c e s , V = ~ 1 . . . . . nt . We c o n s i d e r X o b t a i n e d by

s u p e r i m p o s i n g two s t r u c t u r e s , LX and Cx. Specif ical ly , the s t r u c t u r e Cx is the com-

p le te g r a p h K n = (~1 ..... hi , t(i,j) I 1 <- i < j -< n]), and the s t r u c t u r e L x is the pa i r (E, ~,),

where E is the c o m p l e m e n t of E, i.e., E = t(i,j) I 1 -- i < j -< n, (i,j) ~t El. Now s u p e r i m -

posing L x and C x m a y be t hough t of as label l ing the edges of K n with labe ls of two

kinds, "edge" and "not an edge". More a b s t r a c t l y , we c o n s i d e r s u p e r i m p o s i n g Lx onto

C x as a p e r m u t a t i o n in Sym(ELjE), i.e., a 1-1 m a p f rom the se t of all u n o r d e r e d pa i r s

(i,j) in EL)E onto all u n o r d e r e d pa i r s (i,j) in the edge se t of K n.

We f i r s t i nves t iga t e when two p e r m u t a t i o n s ~ and ~ in S y m ( E u E ) spec i fy t he

g r a p h X iden t ica l ly . S ince we c o n s i d e r the labels in the se t s E and E ind i s t ingu i shab le ,

i t is c l e a r t h a t exac t l y the p e r m u t a t i o n s in the r i g h t c o s e t Sym(E)xSym(E)Tr spec i fy

i d e n t i c a l g r aphs . We s e t A = Sym(E)xSym(E) and no te t h a t A is t he a u t o m o r p h i s m

group of the ~abelting s t r u c t u r e Lx.

233

Next, we ask when two p e r m u t a t i o n s n, ~ c S y m ( E u E ) spec i fy isomorphic graphs .

Let X, be t he g r a p h spec i f ied by n, X~ the g r a p h spec i f ied by ~ . Then X# m u s t be

ob ta inab le f rom X~ by a p e r m u t a t i o n in S n, So, l e t B be the p e r m u t a t i o n group

induced by S n on the s e t of al l edges of K n. Then X~ and X~ a re i somorph i c iff the pe r -

m u t a t i o n ~ m a y be ob t a ined as t he p r o d u c t an# , where a c A and fl E B, i.e., iff

E AnB. Note t h a t B is t he s y m m e t r y group of t h e labelled s t r u c t u r e C x. We t h e r e -

fore o b t a i n

PI~POSlTION 1 (Hoffmann)

Let X = (V,E) and X' = (V,E') be two g r a p h s with n ve r t i ce s . Let E and E' be t he c o m -

p l e m e n t edge se t s of X and X', r e spec t i ve ly . Then X and X' a re i somorph i c iff e a c h p e r -

m u t a t i o n ~ ~ S y m ( E u E ) which m a p s E onto E' and E onto E' is in the double c o s e t AB,

where A = Sym(E)×Sym(E) and B is the g roup of ac t ions of Sym(V) on the s e t EUE.

The proof is s t r a igh t fo rward . Note t h a t if one such p e r m u t a t i o n ~ is in AB, t hen

the s e t of all such p e r m u t a t i o n s is p r ec i s e ly one of the r igh t cose t s of A c o n t a i n e d in

AB.

EXAMPLE 1

Consider the g raph X of F igure 1 below. Its edge se t E is la = (1,2), b = (2,3), c = (3,4),

1

2 / \ 5 \ /

3 4

F i g u r e I

d = (4,5), e = (1,5)~. The c o m p l e m e n t s e t is E = I f = (I ,3) , g = (2,4), h = (3,5),

i = (1,4), k = (2,5)~. X is spec i f ied by every p e r m u t a t i o n in

A = Sym(Ia ,b ,c ,d , e l )xSym(~f ,g ,h , i ,k l ) . The g r a p h X' of F igure 2 below

2 5

F igure 2

234

has t h e edge s e t E' = ia ,e ,h , i ,k l with t he c o m p l e m e n t se t E' = ib,d,e , f ,gl . This g r a p h

m a y be spec i f ied by q /= (b,c ,h,e ,k,g,d, i , f ) , which is a p e r m u t a t i o n m a p p i n g E onto E'

and E onto E'. One verif ies t h a t q /= cxfl, where a = (a ,e) (b ,c) (h ,k , i ) is m A and

fl = (a ,k ,e)(b ,h , f ) (d , i ,g) . Let B be the p e r m u t a t i o n group i n d u c e d by S 5 on the se t

E u E , Then f l e 13 s ince i t is i n d u c e d by the p e r m u t a t i o n ~ = (1,2,G), t hus ~ e A13, i.e.,

the two graphs are isomorphic. Note that the factorization of ~ is not unique. A

different faetorization is, for example, a'fl', where a' = (b,d,e)(f,g,h) and

f l ' = (b,k,g) (e,h,d) (e,i,f).

More genera l ly , l e t L x be a c o m b i n a t o r i a l s t r u c t u r e with the a u t o m o r p h i s m group

A, C x a combinatorial structure with the automorphism group B, where A and B are

permutation groups with the same permutation domain V. Let Tr E Sym(V) be a per-

mutation specifying how to superimpose L x onto Cx, thereby obtaining the "labelled"

structure X~. Then a permutation ~ ~ Sym(V) specifies a structure isomorphic to X~

iff !/~ e ArrB.

Next, the automorphism group of the structure specified by ~ is clearly A~v~B:

Let ~ c Sym(V) be an automorphism of Xn. Then ~ mush also be an automorphism of

Cx, hence is in B. Furthermore, since the structure L X has been mapped into (Lx) ~, ~k

must be in A n. Conversely, it is clear that any permutation in A=f~B is an automor-

phism of X~. Theorem 7 of Chapter II is a special case of this observation.

Having interpreted double eosets as isomorphism classes, we may consider the

number of double cosets into which Sym(V) is partitioned by the groups A and B as

the number of ~o~%~orrLorIJh~c ways in which the structures L X and C X may he super-

imposed (see also Section 6 below). We therefore know

COROLLARY 1

The n u m b e r of n o n i s o m o r p h i c g r a p h s with n v e r t i c e s and p edges is equal to t he

n u m b e r of doub le cose t s in to which Sp+q is p a r t i t i o n e d by the subg roups A and B,

where A = SpxSq, q = (~)-p, and B is the group induced by S n on the set of all unor-

dered pairs (i,j), i ~ i,j-< n, assuming these pairs have been enumerated in some

order .

Other c o m b i n a t o r i a l app l i c a t i ons inc lude the n u m b e r of non i somorph i e g r a p h

v e r t e x label l ings , e.g., the n u m b e r of d i s t i n c t n e c k l a c e s wi th n beads and a p r e s c r i b e d

n u m b e r of b e a d s of equa l colors , and edge labe l l ings of g r a p h s o t h e r t h a n the com-

p le te g r a p h K n,

235

E~tmLg 2

We d e t e r m i n e the n u m b e r of non i somorph i c g r a p h s wi th 4 ve r t i c e s and 3 edges .

Assuming a lex icographJc e n u m e r a t i o n of the u n o r d e r e d pa i r s (i,j), I -< i < j -< 4, t h e

group B is g e n e r a t e d by the two p e r m u t a t i o n s (2,4)(3,5) and (1,4,6,3)(2,5). We have

p = 3 and q = 3, so t h a t A is S3xS 3. By exhaus t ive e n u m e r a t i o n one can prove t h a t

/0 , (3,4), (3,5,4)~ is a c o m p l e t e s e t of r e p r e s e n t a t i v e s for double cose t s of A and B,

thus t h e r e a re exac t ly 3 non i somorph ic g raphs with 3 edges and 4 ve r t i ces . The dou-

ble c o s e t of A and B conta in ing 0 cons i s t s of 4 r i gh t cose t s of A, and so does t he dou-

ble cose t of A and B conta in ing (3,4), b u t the double cose t con ta in ing (3,5,4) cons i s t s

of 12 r igh t cose t s of A. 71

We have t r a n s l a t e d g r a p h i s o m o r p h i s m into a m e m b e r s h i p p r o b l e m of c e r t a i n

double eosets , and g r a p h a u t o m o r p h i s m into an i n t e r s e c t i o n p r o b l e m of c e r t a i n pe r -

m u t a t i o n groups . We t h e r e f o r e cal l the p r o b l e m of t e s t ing m e m b e r s h i p in a double

cose t ge~zercdized isomorplzisrr~, and the p r o b l e m of d e t e r m i n i n g g e n e r a t o r s of the

i n t e r s e c t i o n of two p e r m u t a t i o n groups genercLl'Lzed uutorr~o~ohisnz. We should

inquire whe the r admi t t i ng a r b i t r a r y p e r m u t a t i o n g roups in th is r e f o r m u l a t i o n a l t e r s

the diff icul ty of t he se p rob l ems , and we exp lore this ques t ion next .

2. Group-Theoretic Problems of Intermediate Difficulty

We examine some g r o u p - t h e o r e t i c p r o b l e m s which a r e a t l e a s t as h a r d as g r a p h

i somorph i sm, i.e., g r a p h i s o m o r p h i s m can be po lynomia l t ime r e d u c e d to eve ry one of

them. All p r o b l e m s d i s cus sed in th is sec t ion a re c l ea r ly in NP. F u r t h e r m o r e , the

ex i s t ence p r o b l e m s a re po lynomia l t ime equiva len t to t he a s s o c i a t e d count ing p rob -

lems. For NP-comple te ex i s t ence p rob l ems , the a s s o c i a t e d count ing p r o b l e m s a re

be l ieved to be m o r e difficult . Therefore , we do no t be l ieve t h a t t h e s e p r o b l e m s a re

NP-comple te .

We g roup the p r o b l e m s loosely by t h e i r a p p a r e n t r e l a t e d n e s s . All p r o b l e m s a re

shown to be po lynomia l ly t r a n s f o r m a b l e into each o t h e r and a re t h e r e f o r e of equal

difficulty. F u r t h e r m o r e , t h e r e a re easy r e d u c t i o n s es tab l i sh ing t h a t these p r o b l e m s

are at l e a s t as h a r d as g r a p h i somorph i sm.

Throughout this sect ion, we a s s u m e t h a t all g roups a re speci f ied by gene ra t i ng

se t s of size po lynomia l ih t he i r degree .

236

2. I. Double Coset Problems

Recall t h a t double cosets may be unde r s tood as the equivalence classes of an

equivalence re l a t ion induced on S n by the subgroups A and B of Sn. That is, ~, ~ E S n

are equiva lent iff t he re exists a E A and fl c B such t h a t 9 = a~fl. We inves t iga te first

the complexi ty of tes t ing this equivalence given gene ra t ing sets for A and for B. We

call this t e s t the Double Coset Membership Problem. As poin ted out before, the prob-

lem is a genera l ized i somorph i sm quest ion. Note t h a t Double Coset Membership is

c lear ly a p rob lem in NP.

As a subprob lem, we consider tes t ing whether ~ is equiva len t to the iden t i ty per -

m u t a t i o n (), i.e., whether ~ = c~fl for some ~ E A, fl E B. This 6~oup Factoriz~tion

Problem is of difficulty equal to Double Coset Membership for reasons h in ted at in the

r e m a r k following Lemma i above.

The count ing p rob l em associa ted with Croup Fac tor iza t ion is de t e r mi n i ng the

n u m b e r of d i s t inc t fac tor iza t ions ai# i of ~ over A and B. We show t h a t this p r ob l e m is

no ha rde r t han Group Factor iza t ion. There are no known NP-complete exis tence

p rob lems whose associa ted count ing p rob lems are known to be in NP. We therefore

view the polynomial t ime equivalence of Group Fac to r i za t ion and i ts associa ted coun t -

ing p rob l em as evidence agains t the possibi l i ty t h a t the Double Coset Membership

P rob l em is NP-complete .

P I ~ B ~ I (Double Coset Membership)

Given the groups A, B < S n by genera t ing sets and the p e r m u t a t i o n s ~, ~/E S n, t e s t

whether ~ c A~B.

PROBLEM 2 (Group Factorization)

Given the groups A, B < S n by genera t ing sets and the p e r m u t a t i o n ~ c S n, t e s t

whe ther the re are a < A, fl c B, such t h a t x = aft. Equivalently, t e s t whether ~ e AB.

PROBLF~ 3 ( N u m b e r of Faetorizations)

Given the groups A, B < Sn by gene ra t ing sets and the p e r m u t a t i o n ~ E Sn, d e t e r m i n e

the n u m b e r k -> 0 of d i s t inc t fac tor iza t ions ~ = a# of ~, where a cA, # E B.

Clearly P r o b l e m i is in NP, We show tha t P r o b l e m s 1 th rough 3 are of equal

difficulty, i.e. po lynomial t ime equivalent .

237

1 (Hoffmann)

P r o b l e m s i a n d 2 a r e p o l y n o m i a l t i m e equ iva l en t .

P roo f S ince P r o b l e m 2 is a spec i a l case of P r o b l e m l , we on ly n e e d to r e d u c e

P r o b l e m I to P r o b l e m 8. For th i s r e d u c t i o n , we r e c a l l t h a t t he e l e m e n t s a ~ of A~B

m a y be p u t in to l - i c o r r e s p o n d e n c e wi th the e l e m e n t s 7T-la~fl of A~B. T h u s , ~ c A~B

iff ~ - I ~ E A~B. S ince <K> ~ = <K~>, th i s e s t a b l i s h e s a p o l y n o m i a l t i m e r e d u c t i o n by

the r e s u l t s of C h a p t e r If. -

In o r d e r to show the p o l y n o m i a l t i m e e q u i v a l e n c e of P r o b l e m s 2 a n d 3, we n e e d

t he following

!.m~MA 2

If ~ = alf l 1 = ~ # 2 = " ' " = ~kflk, ai ~ A, #i c B, a re the d i s t i n c t f a c t o r i z a t i o n s of ~ over

A a n d B, t h e n k = IA(~BI.

P roof Let C = ANB. S ince a i # i = ajflj, we have a j - l a i = fljfl~-l, a n d so fli a n d flj a re

in t he s a m e r i gh t c o s e t of C. F u r t h e r m o r e , if 7r = a f t a n d 7 E C, t h e n also

= (ay-1)(yfl) = a' f l ' , t h u s the fli f o r m a r i gh t c o s e t of C. Final ly , obse rve t h a t a i is

u n i q u e l y d e t e r m i n e d by ~ a n d #i, t h u s k = I C]. "

T t t g o ~ 2 ( H o f f m a n n )

P r o b l e m s 2 a n d 3 a r e p o l y n o m i a l t i m e equ iva len t .

P roo f I t is c l ea r t h a t we c a n r e d u c e P r o b l e m 2 to P r o b l e m 3 in p o l y n o m i a l t i m e .

We e s t a b l i s h t he c o n v e r s e r e d u c t i o n as follows: F i r s t , t e s t w h e t h e r ~ ~ AB us ing t h e

a l g o r i t h m for P r o b l e m 2. This d e t e r m i n e s w h e t h e r to o u t p u t zero or a pos i t ive

n u m b e r . Second , if ~ E AB, t h e n d e t e r m i n e JC[ wi th the a l g o r i t h m be low which m a k e s

r e p e a t e d cal ls on the a l g o r i t h m for P r o b l e m 2.

Let C = Ac~B, a n d le t A (i), B (i), a n d C (i) be t he po in tw i se s t a b i l i z e r s of t l . . . . . i - l t in

A, B, a nd C, r e s p e c t i v e l y . Here A (1) = A, ]3 (1) = B, C (1) = C, a n d

A (n+l) = B (a+l) = C (n+l) = I. By C h a p t e r II, we m a y a s s u m e t h a t we have c o n s t r u c t e d

c o m p l e t e r i gh t t r a n s v e r s a l s Ui a n d Vi for A (i+1) in A (~) and for B (i÷1) in BCi), r e s p e c t i v e l y ,

in p o l y n o m i a l t ime . We will d e t e r m i n e t h e o rb i t Ai of i in C(0. Having done so, we c a n

11

d e t e r m i n e ]CI f r o m the f o r m u l a [CI = ~ I~1, in p o l y n o m i a l t ime . Note t h a t i = l

I~1 -< n - i + 1 .

Let ~i c U~ be a r i gh t co se t r e p r e s e n t a t i v e for A (i+1) in A (i) m a p p i n g i i n to j, i.e.,

wi th i ni = j. We do the following:

238

(a) Find ~i ~ Vi such that i ~i = j. If there is no such ~i, then j cannot be in A i.

(b) Let X = ~i~i -I, where #i was found in Step (a). Then j -c A i iff X e A(i+I)B (i+l).

The correctness of Step (a) follows trivially from C (O = A(i)c~B (i) and Theorem ~ of

Chapter If. For Step (b), observe that j 6 Ai iff there are representatives ~k e U k and

~k 6 Vk, i --< k -< n, such that

Thus, j ~ ~t iff

X .= ~i~F I = Tri~ • - " #qllb~!b._~ " • - !bi+1 = #'!b' -c A(i+i)B 0+I)

Observe tha~ we test at most n-i+l products ?ri~ -I for membership in A(i+I)B (i+i).

Therefore, the reduction is polynomial time.

2.2. Intersect ion Problems

Recal l L e m m a 20 of C h a p t e r III: There is a t - i c o r r e s p o n d e n c e b e t w e e n the r i gh t

cose t s of A c o n t a i n e d in t he double c o s e t AB and the r igh t eose t s of A•B in B. This

r e l a t i onsh ip is also r e f l e c t e d in t he i n t e r p r e t a t i o n of double eose t s and of g roup i n t e r -

s ec t i on as g e n e r a l i s o m o r p h i s m and g e n e r a l a u t o m o r p h i s m , r e spec t i ve ly .

We have exp lo i t ed the r e l a t i onsh ip be tween AB and Af~B in the r e d u c t i o n of

Theorem 2 when d e t e r m i n i n g I ANB ] . We now show t h a t Double Coset Membersh ip is

po lynomia l t ime equiva len t to Group In tersee t ion~ Thus the i n t e r p r e t a t i o n of t he two

problems as generalized isomorphism and automorphism, respectively, is accurate in

the sense that Theorem 6 of Chapter II remains valid in this generalization,

PROBTXU 4 (Coset Intersection Emptiness)

Given the groups AN B < Sn by generating sets and given a permutation ~ c S n, test

whether A~f~B is empty,

P R O ~ 5 (Group ~ntersec~ion)

Given the g roups A, B < S n by g e n e r a t i n g se ts , d e t e r m i n e a g e n e r a t i n ~ s e t for

C = AQB.

239

PRO~.~S~ 8 (Setwise Stab i l i z e r )

Given the group A < S n by a gene ra t i ng set , and given a s u b s e t X of [1 . . . . . nl , d e t e r -

mine a g e n e r a t i n g se t for the setwise s t a b i l z e r Ax of X in A.

T , ~ 3

P r o b l e m s 4 and 2 a r e po lynomia l t ime equivalent .

P roof A s imple ca l cu la t ion shows t h a t 7r c AB iff ATr(hB ~ ¢. -

We will show t h a t P r o b l e m s 4 t h r o u g h 6 are po lynomia l t ime equ iva len t wi th P rob -

l em 2.

THEOREM 3 (Hoffmann)

P r o b l e m 5 can be r e d u c e d to P r o b l e m 2 in po lynomia l t ime.

Proof Recal l the proof of Theorem 2. We will ex tend the a l g o r i t h m given t h e r e

and d e t e r m i n e g e n e r a t o r s for C = A(~B. As before , l e t A (i), B (i), and C (i) deno te t h e

pointwise s t ab i l i ze r s of f l . . . . . i - l J in A, B, and C, r e spec t ive ly , and r eca l l t h a t A (I) = A,

B (1 )=B, C 0 ) = C , and A (n+ l )=B (n÷l)= C (n+O=I , Let U i and V i be c o m p l e t e r i gh t

t r a n s v e r s a l s for A (i+O in A (i) and for B0+;) in B (i). We will d e t e r m i n e c o m p l e t e r igh t

t r a n s v e r s a l s W i for C (i+1) in C (i), i -< i -< n.

Fo r each ~T i ~ Ui, we f i rs t d e t e r m i n e w h e t h e r j= i ni is a po in t in A i, t he o r b i t of i in

C (i), Next, we out l ine how to find a c o s e t r e p r e s e n t a t i v e ~i E Wi mapp ing i in to j, p ro -

v ided S tep (b) of the p r o c e d u r e in the proof of Theorem 8 d e t e r m i n e s t h a t j is in ~i.

Let ni and ~i be t he r e p r e s e n t a t i v e s found in S tep (b), and r eca l l t h a t

iff

X = 7Till -i C A(I+I)B(~+O

where Iri+ k E Ui+ k and ~'/i÷k E Vi÷ k. Therefore,

Xi = ~i~/i -1 E ACi+t)D (i+l)

iff

We p r o c e e d as follows:

X i ÷ k = 7 r i + k ? T i ÷ k - I . . . . " ' ?Ti~i -~I q/Yi+k-l~i÷k-1 - I E A(i+k+1)B (i+k+O

Let Xi = ~i~/~ - I c A(i+OB(i+l) be the pa i r of c o s e t r e p r e s e n t a t i v e s

240

determined by Step (b), and set k to i.

(e) Find a pair ~k+1 e Uk+1, #k+1 E Vk+1, such that Wk+iXkg~+11 ~ A(k+2)B (k+s).

Note that such a pair must always exist.

-I Repeat Step (c) letting Xk+1 = Trk+IXk~Pk+l for the found pair, and increment k until

we have determined Xn = 0. The desired coset representative is now 7rnTIn_ 1 • • " 7r i,

,*'here the n k have been found in Step (c) above.

This procedure makes at most 0(n s) calls on the algorithm for Problem 2. There-

fore, we can determine the sets W i forming a generating set for C by a polynomial

time reduction to Problem 2 and therefore, by Lemma 3 above, to Problem 4. •

THV.ORI~M 4 (Luks)

Problems 5 and 6 are polynomial time equivalent.

Proof Let A < S n and X a subset of I I ..... n~. Let X be the complement of X, i.e.

= ~i ..... n~-X. Since A X = A (~ Sym(X)xSym(X), it is clear that Problem 6 can be

reduced to Problem 5, in polynomial time.

Conversely, let A, B < Sr~ be the groups we wish to intersect. We construct the

group D isomorphic to A×B acting on a set of size n 2 consisting of all pairs (i,j),

l-<i<-n. The elements of D are the pairs (a,/~), where ~ cA, ~ eB. The action of

(cx,/~) is defined by

(i,j)~,~ = (i~,j~).

If A and 13 are generated by the sets K A and K B of cardinality m I and ms, respectively,

then we may construe% a generating set for D of size m1+m 2 in polynomial time.

Let Z= I (i,i) i l~i<-n~. Then it is clear that the setwise stabilizerD z of ZinD

is isomorphic to the intersection of A and ]3. Furthermore, using projection, it is easy

to construct generators for A~Bfrom generators for D z. "

THEOR~ 5 (Lipton, Kannan)

Problem 4 can be reduced to Problem 6 in polynomial time.

Proof We wish to test whether A~AB is empty given an algorithm for Problem 6.

We assume thatA,}3 < Sym(X), and consider the group D = ~ (cx,~) I c~ eA,~ e ]3 ~ act-

ing on X×X by the rule (x,y) (a'~) = (xa,y~). Consider the sets Z'= I (x,x) I x e X I and

Z' = ~ (x=-',x) I x [ X ~. We test in the manner described below whether D contains an

element 5 such that Z ~ = Z' using an algorithm for Problem 6. Clearly this is the ease

if[ A~r(~B is not empty.

241

So, c o n s i d e r t h e p r o b l e m of f inding an e l e m e n t 6 in t he g roup D < Sym(Y) which

m a p s t h e s u b s e t Z of Y onto t h e s u b s e t Z' of Y. Here we c o n s t r u c t t h e g roup

G = D%Cz ac t ing on the d is jo in t union Y1uY~ of two copies of t he se t Y. Using an algo-

r i t h m for P r o b l e m 6, we d e t e r m i n e a g e n e r a t i n g se t for t he subg roup G' of G which

s tab i l i zes se twise Z~uZ ' 2, where Z 1 is the s u b s e t Z in the copy Y1, Z'a the s u b s e t Z' in

the copy Y2 of Y. Let H = G'(~DxD be the setwise s t ab i l i z e r of Y1 in G'. Since

(G:DxD) = 2, t he index of H in G' is a t m o s t 2. Therefore , using the t echn iques of

C h a p t e r II, we can d e t e r m i n e H f rom a gene ra t i ng s e t for G'. We conc lude the proof of

the t h e o r e m by showing t h a t (G':H) = 2 iff D con ta ins an e l e m e n t 6 which m a p s Z onto

Z',

Assume t h e r e ex is t s an e l e m e n t 6 e D such t h a t Z 6 = Z'. Then IP = (6,6-I;(1,2))

m u s t exchange ZI wi th Z'~, hence ~ e G'. Since ~ does no t s tab i l ize Y1, t h e index of H

in G' m u s t be 2. Conversely, l e t (G':H) = ~ and cons ide r an e l e m e n t ~ e G' which is no t

in H. Then ~# m u s t be of the fo rm (6,7;(1,2)). Since ZIcY ~ and Z'ecY z, Z~ m u s t be Z' 1,

i.e., 6 is t he d e s i r e d e l e m e n t of D. -

We have now e s t a b l i s h e d the po lynomia l t ime equiva lence of P r o b l e m s t t h r o u g h

8.

2.3. IwIiscellaneous P r o b l e m s

We now c o n s i d e r t h e following two p r o b l e m s which a r e po lynon~a l t ime equ iva len t

to Double Coset Membersh ip :

PROeI~m 7 ( C e n t r a l i z e r in A n o t h e r Group}

Given the g roups A, D < S n by g e n e r a t i n g se ts , d e t e r m i n e a g e n e r a t i n g s e t for the cen -

t r a l i z e r CA(B ) of B in A.

PROSLEM 8 (Restricted Graph Automorphism)

Given a g r a p h X = (V,E) and a p e r m u t a t i o n g roup A < Sym(V), d e t e r m i n e g e n e r a t o r s

for al l a u t o m o r p h i s m s of X which are also in A, i.e., find g e n e r a t o r s for ANAut(X).

F i rs t , we e s t ab l i sh the po lynomia l t ime equiva lence of P r o b l e m 7 with P r o b l e m s 4

t h r o u g h 6. We beg in with a po lynomia l t ime r e d u c t i o n of P r o b l e m 7 to P r o b l e m 5,

Group In t e r sec t ion . For this , we need some e l e m e n t a r y r e s u l t s a b o u t the s t r u c t u r e of

the c e n t r a l i z e r (in Sn) of a cycl ic g roup Z g e n e r a t e d by a p e r m u t a t i o n 7r c Sn.

242

Let ,w, 7~ ~ S n be p e r m u t a t i o n s , w h e r e 7r, in cyc l e n o t a t i o n , is

7r = (i 1 . . . . . is)(is÷ I . . . . . it) • • - (i r . . . . . iq). Then t h e c o n j u g a t e of 7r u n d e r 7 / i s t h e p e r m u -

t a t i o n ~r~ = (ii~ . . . . . %~)(%÷i ? . . . . . it~ ) . • . (ir~ . . . . . iq~). We i l l u s t r a t e th i s e l e m e n t a r y

o b s e r v a t i o n in t h e fo l lowing

~ L E 3

Let ~ r = ( i , 2 ) ( 3 , 4 ) ( 6 , 7 , 9 ) and ~ = (1,2,3,4)(5,6) be p e r m u t a t i o n s in S 9. Then

~ = (1~,2~)(3~,4~)(6#,7~,9 ~) = (2,3)(4,1)(5,7,9) , w h i c h is r e a d i l y ver i f ied . Z]

LF~MA 4

Le t A < S n be g e n e r a t e d by I~ l, a z . . . . . apt , and l e t Ci be t h e s e t of all p e r m u t a t i o n s in

P S n wh ich c o m m u t e wi th a i, i .e., Ci = / ~ ¢ S n t 7r- lai ~r = ai ~. T h e n Cs,(A) = N Ci.

i=1

P P r o o f I t is e a s y to s e e t h a t t h e s e t s C i a r e g r o u p s . T h e r e f o r e , D -- Q C i is a lso a

i = l

group . Let 7 ~ D, a ~ A. S i n c e a is a f in i te p r o d u c t of t h e ai, c l e a r l y 7 - ] a 7 = a , t h u s

7 ~ Cs,(A). Conve r se ly , l e t 7 c Cs~(A). S ince a i c A , 7 - ] a i 7 = a i, i - < i - p , t h u s 7 c D .

T h e r e f o r e , Cs~(A ) = D. ~

As a c o n s e q u e n c e of t h e l e m m a , C i = Cs~(ai) = Cs~(Zi), w h e r e Z i is t h e cyc l i c g r o u p

g e n e r a t e d by ai.

L ~ 5

Le t 7r ~ S n be a p e r m u t a t i o n , Z = <TT> t h e cyc l i c g r o u p g e n e r a t e d by ~. T h e n g e n e r a -

t o r s fo r t h e c e n t r a l i z e r Cs~(Z) c a n be d e t e r m i n e d in p o l y n o m i a l t i m e .

P r o o f Le t ~ be w r i t t e n in c y c l e n o t a t i o n wi th t h e d i s t i n c t cyc l e l e n g t h s

11, lz . . . . . lr, and wi th m~ c y c l e s of l e n g t h 1 i. H e r e we also c o n s i d e r c y c l e s of l e n g t h 1.

7 / e Sn c o m m u t e s wi th 7r iff ~ = ~. By L e m m a 4, we t h e r e f o r e look fo r all t h o s e p e r -

m u t a t i o n s ~ s u c h t h a t c o n j u g a t i o n u n d e r ~ " r o t a t e s " t h e c y c l e s of ~ a n d / o r

e x c h a n g e s c y c l e s of e q u a l l eng th . S i n c e t h e cyc l e n o t a t i o n is u n i q u e up to t h e s e two

r e w r i t i n g r u l e s , Cs~(Z) m u s t c o n s i s t p r e c i s e l y of t h o s e p e r m u t a t i o n s .

Fo r e a c h i-< r, we e x a m i n e t h e c y c l e s of l e n g t h 1 i. By i n s p e c t i o n , we will p r o d u c e

a s e t Ki of p e r m u t a t i o n s g e n e r a t i n g all t h o s e p e r m u t a t i o n s 7/ w h i c h c o m m u t e wi th

and a r e s u c h t h a t all p o i n t s wh ich lie in c y c l e s of ~ of l e n g t h o t h e r t h a n 4 a r e f ixed by

Le t J1 . . . . . Jr~ be al l t h e c y c l e s in ~ of l e n g t h l i. The s e t Ki will c o n s i s t of p e r m u t a -

t i ons ¢~,~ ..... ~i,m i and two permutations ai and ~i. K~ contains no other permutation.

243

We let (~,~ = 4 , t -¢ s s; m i. Fu r the rmore , if 4 = (J9,1, is2 . . . . . J~,ti), t h e n

ai = (Jl,l,J2,1) " ' ' (Jl,ll,J2,1~), and ~i = (Jl,l,J2,1 .. . . . Jn~,l)(Jl,2 . . . . . Jmi,2) " " ' (Jl,1 i . . . . . Jr.,h)"

Recall t ha t Sn is gene ra t ed by a = (1,8) and fl = (1,2 .. . . . n). Thus, it is c lear t ha t

a~ and fli gene ra t e pe rmu ta t i ons ~ which conjugate ~r by pe rmut ing the o rde r of the

cycles of length 1 i in all possible ways, Since the ¢i,k gene ra t e all ro ta t ions of these

cycles, K i gene ra te s all pe rmuta t ions ~ ~ S n such tha t conjugat ion under ~ fixes all

cycles in ~r of length o ther t h a n li and r ea r r anges a n d / o r ro t a t e s the cycles of length li

r

in all posssible ways. By Lemma 4, therefore , < L) Ks> = Cs~(Z). 5--1

Observe tha t ~, mi'li = n, thus [KI is 0(n), and so K can be cons t ruc t e d in polyno- 5=1

mial t ime. •

In Sect ion 4 we will show t h a t Csn(Z ) arises as the a u tomorph i sm group of a ve ry

simple g raph which can be cons t ruc t ed f rom w in 0(n) steps.

~ I ~ 4

Let ~r = (1,2)(3,5,6)(4,7)(8,12)(9,11,13)(10)(14) be a pe rmu ta t i on in $14 genera t ing the

group Z. ~r has two cycles of length l, th ree cycles of length 2, and two cycles of

length S,

We cons ider first the cycles of length 3, which are (3,5,6) and (9,1t,13). We obtain

~1,1 = (3,5,6), ¢1,~ = (9,11,13), a 1 = (3,9)(5,11)(6,13), and ~i = al . Thus,

K1 = [¢14, ¢1,a, all . Next, we consider the th ree cycles of length 2, which are (1,2),

(4,7), and (B,18). Here we obtain the set Kg consist ing of ¢~,1 = (1,~), ~2,2 = (4,7),

¢2,3 = (8,12), a a = (1,4)(2,7), and fla = (1,4,8)(2,7,12). Finally, Ior the cycles of length

1, the pe rmuta t ions ¢ are each the ident i ty permuta t ion , and as =/~s = (10,14).

Together, these pe rmuta t ions genera te Cs.(Z ). []

TREOn~ 6

Prob lem ? can be polynomial t ime r educed to Prob lem 5.

Proof Let A, ]3 < Sn, with known genera t ing sets. We can find gene ra to r s for

CA(D ) i n t w o steps:

(a) Determine gene ra to r s for Csn(B ).

(b) I n t e r s ec t Cs~(B ) with A.

In Sect ion 3, we will show how to do Step (a) in polynomial t ime. For the present , we

do Step (a) by cons t ruc t ing Cs~(<~> ) for each of the gene ra to r s fl of ]3 using Lemma 5,

244

and then intersect these groups, thereby obtaining Cs~(B) by Lemma 4. -

THEOR~ 7 (Lugs)

Problem 6 can be reduced to Problem 7 in polynomial time.

Proof Let A <Sm X a subset of ~I ..... nl. We will determine generators for Ax,

the setwise stabilizer of X in A, using an algorithm for Problem 7.

Let A' be the group isomorphic to A acting on

Y= I (i,j) I i<_i<_n,j = 1,2~

constructed by associating with ~ £ A the permutation ~' of Y where (i,j) a' = (i=,j).

Let ~ be the permutation of Y where, for i c X, (i,j) == (i,j), and, for i ~X,

(i,l) = = (i,2) and (i,2) = = (i,i). Let Z be the group generated by n. Then it is easy to

see that CA,(Z), the centralizer in A' of Z, is isomorphic to A x, Generators for A x are

easily obtained from generators for CA,(Z ). -

Note that the proof of Theorem 7 shows that Problem 7 remains hard even when

restricting the group B to a cyclic group. Finally, we establish the equivalence of

Problem 8 with the Problems !-7.

THEOREM 8 (Luks)

Problem 8 is polynomial time equivalent with Problems 5 and 6.

Proof Let A < Sn be a group in which we wish to stabilize the subset Y of 11 ..... n~.

Without loss of generality we may assume that IY[ > i. Now Ay = ANSym(¥)×Sym(~r).

We observe that Sym(Y)×Sym(~/) is the automorphism group of the graph

X = (1i ..... nl, ~(i,j) I i, j e Y~), hence finding generators for Ay is an instance of Prob-

lem 8. Conversely, by Theorem 7 of Chapter H, given a graph X, Ac~Aut(X) is the inter-

section of three permutation groups with known generators. -

We have now established the polynomial time equivalence of Problems i through

8. A number of special cases of these problems are in P, and we discuss them in Sec-

tion 4. In Section 3 we discuss a special ease of Group Intersection that is in

NP N coNP.

245

2.4. I s o m o r p h i s m C o m p l e t e P r o b l e m s

Recal l t h a t p r o b l e m s which a re po lynomia l t ime equ iva len t to g r a p h i s o m o r p h i s m

have b e e n ca l led i s o m o r p h i s m c o m p l e t e . There a re m a n y known i s o m o r p h i s m com-

p le te p r o b l e m s . Because of Sec t ion 1, we use t he following two p r o b l e m s as r e p r e s e n -

ta t ive s:

PROBLI~ 9 (Ex i s t ence of Graph I s o m o r p h i s m )

Given nonnega t ive n u m b e r s p, q, and n, such t h a t p + q = (~) and a p e r m u t a t i o n

~T e Sp÷q. De t e rmine w h e t h e r ~r e AB, where A is i somorph ic to SpXSq and B is the p e r -

m u t a t i o n group i nduced by Sn on the s e t of all u n o r d e r e d pa i r s (i,j), I -< i < j - n.

PROBL~ 10 (Graph A u t o m o r p h i s m )

Given nonnega t ive n u m b e r s p, q, and n, such t h a t p + q = (~) , d e t e r m i n e a g e n e r a t i n g

se t for A~B, where A is i somorph ic to SpXSq and B is t he p e r m u t a t i o n group induced

by S n on the s e t of all u n o r d e r e d pa i r s (i,j).

The i n t e r p r e t a t i o n of t h e s e p r o b l e m s as g r a p h i s o m o r p h i s m and g r a p h automor-~

phism, r e spec t i ve ly , follows f rom Propos i t i on i and the s u b s e q u e n t d iscuss ion . So,

the po lynomia l t ime equiva lence o f .P rob lems 9 and I0 follows f rom the r e s u l t s of Sec-

t ion 2 of Chap te r II. I t is c l ea r f rom the p r o b l e m fo rmu la t i on t h a t P r o b l e m 9 can be

r e d u c e d in po lynomia l t ime to P r o b l e m I. We have no t found a r e d u c t i o n in the con-

ve r se d i r ec t ion , and we c o n j e c t u r e t h a t none exis ts .

2.5. Remarks

We have given a co l l ec t ion of g r o u p - t h e o r e t i c p r o b l e m s which a re a t l e a s t as h a r d

as g r a p h i somorph i sm . The p r o b l e m s a re n a t u r a l l y r e l a t e d t o g r a p h i somorph i sm.

For one, g r a p h i s o m o r p h i s m c a n be r e d u c e d to each of t h e s e p r o b l e m s in po lynomia l

t ime by easy and s t r a i g h t f o r w a r d r educ t i ons . These a r e the f i rs t e x a m p l e s of such

r e d u c t i o n s among n a t u r a l p r o b l e m s in NP of unknown c o m p l e x i t y s t a tus . F u r t h e r -

more , as po in t ed ou t a l ready , Double Coset Membersh ip is i t se l f an a b s t r a c t i somor -

p h i s m ques t ion involving s t r u c t u r e s which a r e a p p a r e n t l y m o r e g e n e r a l t h a n g raphs .

246

Of course, our conjecture that these problems lie properly between P and NP-

complete is more speculative than the implied conjecture that P ~ NP.

It is not out of question that these problems ultimately belong to P. However, the

length of time that graph isomorphism, the problem on the lowest level of difficulty,

has resisted a polynomial time solution does not suggest that this possibility is very

likely. It is also possible that the problems are polynomial time equivalent with graph

isomorphism. If they are not, then it must be that permutation groups of degree n

are hltrinsieally more complicated objects than the automorphism groups of graphs

of size polynomial in n. !n particular, an open question closely related to this is the

following:

PKOBL~g 11

Given a permutation group G of degree n, is there a graph X with m vertices such that

G is isomorphic to Aut(X), the automorphism group of X, and m is polynomial in n?

It is known that every permutation group G is isomorphic to the automorphism

group of some graph X. However, the size of the graph X is polynomial in the order of

G, not the degree of G. Should the above question have an affirmative answer, then it

is not difficult to show that Problems i through 8 are isomorphism complete~

3. A P r o b l e m w i t h a S h o r t V e r i f i a b l e S o l u t i o n

In this section, we show that Problem 5, Group Intersection, has a special case

which is in NPC]coNP. That is, for certain permutation groups we may guess a gen-

erating set for their intersection and can then verify with certainty in polynomial

time that we have guessed correctly. Specifically, we consider

PROBI.~ 12 (Intersee~don of Commuting Groups)

Given generating sets for the commuting permutation groups A and B, fund a generat ~

ing set for AC]B.

Recall that %we subgroups of S n co~n~e iff AH = HA. Note that the commuta-

tivity of groups does not necessarily imply that the two groups commute element by

247

e l emen t , o r t h a t one is a subg roup of the n o r m a l i z e r of t he o t h e r (cf. C h a p t e r V,

Definition 7). For example, one verifies easily that with A = <(1,2), (1,3)(2,4)> and

B = <( t ,2 ,3 )> we have AB = BA = S 4. Yet n e i t h e r B n o r m a l i z e s A no r A n o r m a l i z e s B.

The k e y p r o p e r t y of c o m m u t i n g g roups to be exp lo i t ed is s t a t e d in the following

l e m m a which is e a sy to prove

L ~ 6

Let A and B be subg roups of a finite group G. Then the complex AB is a g roup iff AB =

BA.

We explo i t th is I e m m a b y observ ing t h a t s ince t he c o m p l e x AB is t h e g roup <A,B>

for c o m m u t i n g groups , we m a y d e t e r m i n e the o r d e r of AB using the t e c h n i q u e s of

Chap te r II. For n o n c o m m u t i n g g roups t h e r e s e e m s to be no easy way for d e t e r m i n i n g

the c a r d i n a l i t y of t he complex AB. Applying L e m m a 20 of Chap t e r III, we t h e n com-

pu te the o r d e r of the i n t e r s e c t i o n ANB f rom the o r d e r s of A, B, and AB.

Having d e t e r m i n e d the o r d e r of AV~B in po lynomia l t ime , we m a y now ver i fy t h a t

a (guessed) se t K of p e r m u t a t i o n s g e n e r a t e s AC~B: F i r s t , t e s t for e a c h lr < K w h e t h e r

c AraB, i.e., w h e t h e r n E A and ~ c B. This e s t ab l i shes t h a t K g e n e r a t e s a subgroup

of A~B. Then, c o m p u t e the o r d e r of the g roup g e n e r a t e d by K. If i t is equal to the

o r d e r of ANB, t h e n <K> = AraB. Otherwise, <K> is a p r o p e r s u b g r o u p of t he i n t e r s e c -

t ion. By the r e s u l t s of C h a p t e r II, all s t e p s r equ i r e po lynomia l t i m e only. In s u m -

mary , we have j u s t shown

THEOREM 9 (Hoffmann)

P r o b l e m IS is in NP C~ coNP.

We know of no polynomial time test of whether two permutation groups commute.

In some eases, this may be verified using Lemma 7 below.

L ~ 7

If the indices (<A,B>:A) and (<A,B>:B) are eoprh~e, then AB = BA = <A,B>.

Proof Since (AB:B) = (A:Af~B) we have (<A,B>:A) -~ (A:Af~B). So,

(<A,B>:AC~B)-<(<A,B>:A).(<A,B>:B). Since (<A,B>:A) and (<A,B>:B) b o t h divide

(<A,B>:A(~B), by copr ima t i ty , (<A,B>:A).(<A,B>:B) m u s t divide (<A,B>:Af~B). Thus,

(<A,B>:AC~B) = (<A,B>:A).(<A,B>:B), h e n c e <A,B> = AB. -

For example , for A = <(1,2), (1,3)(2,4)> and B = <(1,2,3)> we have (<A,B>:A) = 3

and (<A,B>:B) = 8, h e n c e A and B c o m m u t e .

248

4. Subproblems in P

We now discuss a number of special cases of the problems of Section 2 with known

polynomial time solutions. We group these problems by the general problem of which

they are special instances. Of particular prominence are intersection algorithms,

since they form the basis of many of the algorithms in the preceding chapters.

4.1. Group I n t e r s e c t i o n P r o b l e m s

We cons ide r the following set of problems:

PROBLEIg 13 (Intersection with a Normalizing Group)

Given permutation groups A, B < S n by generating sets, determine generators for

A~B, provided that B normalizes A.

PROBTXU 14 ( I n t e r s e c t i o n wi th a p-Group)

Given permutation groups A, B < S n by generating sets, determine generators for

A(~B, provided that A is a p-group.

PROB~.~ 15

Given p e r m u t a t i o n groups A, B < S n by gene ra t ing sets, d e t e r m i n e gene ra to r s for

AC~B, provided that A is in the class F b and b is a fixed constant.

PROBL~ 16

Given p e r m u t a t i o n groups A, B < S n with a po lynomia l t ime m e m b e r s h i p test , de te r -

mine gene ra to r s for A(~B, provided tha t bo th A and B are (k,c)-accessible f rom a

group G < Sn with known generators.

PROBnm~ 17

Given permutation groups A, ]3 < S n with a polynomial time membership test, deter-

mine generators for Ac~B, provided that A is (k,e)-aecessible from a group G < S n with

known generators, B is a subgroup of G, and generators for B are known.

Except for P rob l em 13, we can ob ta in po lynomia l t ime a lgor i thms f rom ear l ier

a lgor i thms af ter su i table modif icat ions. We begin with P r ob l e m 13.

The first c ruc ia l observa t ion to be made is t ha t AB = BA provided tha t one of the

groups is a subgroup of the no rma l i ze r of the other. For if B normal izes A, say, t h e n

249

A~ = ~A for all e l e m e n t s ~ E B. So, we have AB = <A,B> and can therefore t e s t

m e m b e r s h i p in AB.

The second c ruc ia l observa t ion is t ha t the normal iz ing p r o p e r t y is " inher i ted" by

the pointwise stabi l izers:

L~m~iA 8

Let B normal ize A, where A, B < S n. Then for any po in t i E fi ..... hi , Bi normal izes A i.

Proof Let ~ E Ai, ~ E B v Since B normal izes A, we have ~ - I ~ E A, But i ~ - ' ~ = i,

hence ~ - 1 ~ ~ Ai" •

As a consequence of Lemma B, we also have A(1)B Ci) = <ACi),B(i)>, where A Ci) and B 0)

are the pointwise s tabi l izers of ~f ..... i - i ~ in A and B, respect ively . Thus, we have a

polynomial t ime m e m b e r s h i p t e s t in these double eosets as well. That is, we may con-

ve r t the proof of Theorem 3 into a de t e rmin i s t i c po lynomia l t ime algori thm.

Before giving the algori thm, we make two observa t ions which inc rease the

efficiency of the method.

First , cons ider the t r ia l and e r ro r m e m b e r s h i p tes t s of Step (c) in the proof of

Theorem 3: The purpose of these tes ts is to factor a p e r m u t a t i o n 7r ~ AB as ~r = a~,

where ~ E A, ~ c B. We observe t ha t this fae tor iza t ion can be achieved direct ly,

provided every e n t r y ~ in the r e p r e s e n t a t i o n ma t r ix for AB is a l ready fac tored in this

way. For, let 7t = ~ n ' ' ' ~ 1 , where ~k = akl?k- Then, s ince B normal izes A, we have

~r = (a 'n" ' " ~dl)(~n ' ' " ~I) = ~'~'. Here we know ~ and ~', hence we can also compu te

To see how the en t r i e s of a r e p r e s e n t a t i o n ma t r ix can be ob ta ined in the su i table

form, cons ider the ma t r ix c o n s t r u c t i o n by the t echn iques of C h a p t e r lh We sift the

gene ra to r s for both A and IS followed by sifting pair products , thus, in pr inciple , we

know each en t ry as a word of p e r m u t a t i o n s in A and B. These words are now brough t

into the r equ i red form a~ using the above considera t ions . It is ne c e s sa r y to do this

for each new coset r ep re sen t a t i ve when en te r ing it, for otherwise the new words

r ~ h t grow to exponent ia l length (cf. the d iscuss ion at the end of Subsec t ion 3.1 of

Chapter II).

Now observe t h a t for A, B < S n the group <A1,BI> is a subgroup of <A,B> 1. There-

fore, if a is a coset r ep re sen t a t i ve for (A(i)B(i))m in A0)B (i), t h e n it m u s t also be a eoset

r ep resen ta t ive for (A0)B0))m in A0)B 0), where j < i. So, the r e p r e s e n t a t i o n m a t r i c e s for

A(i)B (i), 1-< i-< n + l , may be combined into a single n by n ma t r ix with n o n e m p t y

en t r i es of the fo rm (a, 8, k), where a a is a n o n e m p t y e n t r y in the r e p r e s e n t a t i o n

250

m a t r i c e s for A(k)B Ck), ACk-OB (k-I) . . . . . A(1)B (1).

An i m p l e m e n t a t i o n of t h e s e ideas is qu i te s t r a i g h t f o r w a r d , b u t i t r e q u i r e s c o m -

p u t i n g t he m a t r i x for A(i)B (i) c o m p l e t e l y be fo re c o m p u t i n g t he m a t r i x A(i-1)B (i-1). i t is

n o t h a r d to see t h a t t h e c o m p u t a t i o n r e q u i r e s a t m o s t O(n 8) s t e p s a s s u m i n g we have

a l r e a d y c o m p u t e d r e p r e s e n t a t i o n m a t r i c e s for A a n d B f r o m which to o b t a i n g e n e r a t -

ing sets for the groups A 0) arid B (i) of size O(n z) at most.

Given below is the algorithm for intersecting normalizing groups based on these

ideas.

ALGORITHM t ( I n t e r s e c t i o n w i t h a N o r m a l i z i n g G r o u p )

I n p u t

Output

Comment

M e t h o d

I. b e g i n

2.

G e n e r a t i n g se t s K A and K B for the p e r m u t a t i o n g r o u p s A, B < S n, whe re B

n o r m a l i z e s A,

A representation matrix for C = A(]B.

See Proposition 3 below for a method of testing whether B normalizes A.

Note that Step 3 below computes representation matrices for the groups

A(i)B 0) in accordance with the ideas outlined above.

Compute representation matrices for A and B using Algorithms 2 and 3 of

Chapter iI;

comment Let A (i), B (i), and C (i) be as in the proof of Theorem 3;

3. Compute representation matrices for the groups A(i)B (i), n -> i >- I;

4. Initialize a representation matrix for C = AC~B to represent the trivial group I;

5. f o r i := n d o w n t o i do

6. fo r e a c h p a i r 7T, ~ of e n t r i e s in row i of t he r e p r e s e n t a t i o n m a t r i c e s of A a n d

B, r e s p e c t i v e l y , s u c h thai, i 'T = i ~ = j do

7. if ,V~ -i E A(i+i)B (i+l) t h e n b e g i n

8, l e t 7T~/-i = ~n " ' " ~i+1, w h e r e ~k = ak/~k, ak C A (i+l), flk E B (i+1), n -> k -> i+ t;

9. enter flnfln-~. ' " " /~i+I~ as coset representative into r-ow i, column j of the

representation matrix for C;

lO. e n d ;

1 i. output the representation matrix for C;

12. e n d .

251

The correctness of the algorithm is obvious from Theorem 3 and above discussion.

Furthermore, an elementary analysis shows

PROPOSITION 2 (Hoffmann)

Problem 13 may be solved in O((IKA]+ [KBI).nZ+n 6) steps, where K A and K~ are gen-

erating sets for the two groups.

Given generating sets for both groups, it is easy to test whether a group B nor-

realizes a group A:

PROPOSITION 3 (Hoffmann)

Given generating sets KA and KB for the groups A, B < Sn, one can test whether B nor-

malizes A in 0( IKAI. IKBI-n2+n 8) steps.

Proof In 0(IKAl-n2+n 6) steps we can construct a representation matrix for A

from KA, Having done so, we then have an 0(n 2) membership test in A. Clearly B nor-

malizes A iff, for every pair of generators ~r ~ KA and ~/c KB, the conjugate ~-irr~ is in

A. "

We next discuss Problem 14. Recall Section 4 of Chapter IV. Based on Lemmata 8

and 9 of that chapter we may design a reeursive procedure for intersecting the p-

group G with any other group H. The computation of this recursive procedure can be

reorganized, just as in the case of intersecting two p-groups, resulting in reduced

overhead. In this manner, we obtain an intersection algorithm much like Algorithm 3

of Chapter IV. Essentially, the new algorithm differs only in how to determine point

stabilizers when computing the coset J(AW,[xl), where projl(A ) fixes x (Line 31 of the

algorithm). Since projz(A ) is not a p-group, Algorithms Z and 3 of Chapter If must be

used. Consequently, the algorithm has to spend 0(n s) steps when processing this base

case of the recursion. It is not difficult to develop the details, and we can prove in

summary

PROPOSITION 4 (Hoffmann)

Let A, B < S n, where A is a p-group, and assume given generating sets K A and K B for A

and B, respectively. Then a generating set for A(~B can be determined in

0((IKAI+IKBI)-n2+ n 7) steps. Moreover, if B is also a p-group and if composition

sequences and complete imprimitivity structures for both A and B are given, then a

composition series for A~B may be determined in 0(n 3) steps.

The second part of the proposition restates Theorem 5 of Chapter IV. The term

( ! KAI + I KB I)'nz in the general bound originates from deriving strong generating sets

252

for A and B f rom the given sets K A and K B.

Now cons ider P r o b l e m 15. Regarding the re la t ionsh ip of L e m m a t a 8 and 9 of

Chapter W with the ru les for comput ing the set func t ion Sy(GTr,Z) of Subsec t ion 8. i of

the same chap te r , it is c lear t h a t we can modify Algori thm 3 of Chapter V so t ha t i t

compu te s the i n t e r s e c t i o n of a p e r m u t a t i o n group in Fb with an a r b i t r a r y p e r m u t a t i o n

group. The detai ls are s t ra igh t forward and i t is c lear t ha t the resu l t ing a lgor i thm

requ i res t ime polynomial in the degree of the groups, provided b is a cons tan t .

Finally, we observe tha t P rob lems t6 and 17 are in P by Theorems 13 and !4 of

Chapter tI.

4.2. C e n t r a l i z e r a n d C e n t e r

We now show that Prob l em 7, Cent ra l i ze r in Another Group, has two special cases

which are in P. ] 'hat is, we give a po lynomia l t ime a lgor i thm for finding the cen t ra l -

izer (in Sn) of the p e r m u t a t i o n group G < S n with a given gene ra t ing se t K.

F u r t h e r m o r e , observing t h a t the cen t r a l i ze r of G normal izes G, we apply Algori thm i

and ob ta in a po lynomia l t i m e a lgor i thm for finding the c e n t e r of G. In pa r t i cu la r , we

cons ider

PROBLE]~ 18 (Cent ra l i ze r )

Given a gene ra t ing se t K for the group G < S a, find a gene ra t ing se t for the cen t r a l i ze r

Cs~(G ) of G.

PROBLER 19 (Center)

Given a gene ra t ing set K for the group G < S m find a gene ra t ing se t for the c e n t e r

Co(G) of G.

We consider Problem 18 first, and begin by reexamining the centralizer Cs=(Z) of

the cyclic group Z g e n e r a t e d by a p e r m u t a t i o n ~. Recall the proof of Lemma 5. We

argued tha t ~/is in Cs=(<~> ) iff con juga t ion u n d e r lh r e a r r a n g e s the order ing of cycles

of equal l eng th in ~ a n d / o r ro t a t e s the individual cycles. We now show tha t the re is a

d i rec ted graph Y~ whose a u t o m o r p h i s m group is p rec i se ly Csn(<~>) = Cs~(~r).

253

A cyel~ graph is a d i r e c t e d g r a p h X = (V,E) such tha t , for eve ry v E V, t h e i nde -

g ree and the o u t d e g r e e of v a re e i t h e r b o t h 0 or 1. In tui t ively , a cycle g r a p h cons i s t s

of d is jo in t d i r e c t e d cyc les a n d / o r i so l a t ed points .

DEFINITION 2

Let ~ E Sn be a p e r m u t a t i o n . The cycle graph X~ of ~ is t he cyc le g r a p h (V,E~) where

V = f l . . . . . n l , and t h e r e is an edge f rom i to j in E~ if ff = j.

EXAmPLZ 5

For the p e r m u t a t i o n ~ = (1,2)(3,5,6)(4,7)(8,12)(9,11,13)(10)(14) in $14 we ob t a in the

cycle g r a p h X~ of F igure 3 below. []

2 3--~5 4 8 9 ---~ 11

Figure 3

10 14

Note t h a t X= is unique, and t h a t i t c an be c o n s t r u c t e d f rom ~T in 0(n) s t eps . F r o m

the p roof of L e m m a 5, t he following is now obvious:

Ia:u Iuu~ 9

If 7r c Sn is a p e r m u t a t i o n , X~ the cyc le g r a p h of ~, t h e n Aut(X~) = Cs~(lr).

In Sec t ion 2, we c o n s t r u c t e d Cs~(<K>) by f i rs t c o n s t r u c t i n g CS,(~) for all Ir ~ K,

and t hen i n t e r s e c t i n g these g roups using L e m m a 4. We obse rve now t h a t i n s t e a d of

i n t e r s e c t i n g t h e g roups Cs~(Tr), ~ E K, we can superimpose the g r a p h s X~ r e su l t i ng in a

new g r a p h XK whose a u t o m o r p h i s m group will be Cs~(<K> ). Here we need to l abe l the

edges of e ach g r a p h X~ un i fo rmly to p r e v e n t an a u t o m o r p h i s m of XK f rom map ing an

edge belonging to X~ to an edge belonging to X~, where ~ ¢ ~.

A labelled cycle graph X of d e g r e e k is a d i r e c t e d m u l t i g r a p h (V,E) and a m a p p i n g

f rom E in to the s e t f i . . . . . k l such t h a t t he s u b g r a p h o b t a i n e d by de le t ing all edges in

E which a re no t m a p p e d to i is a cyc le g r a p h (1-< i-< k). Intui t ively, a l abe l l ed cycle

g r a p h is o b t a i n e d by s u p e r i m p o s i n g k cycle g raphs , where each edge of t he i th cycle

g r a p h is l abe l l ed i. The resu l t ing g r a p h is a mu l t i g r aph , s ince t h e r e m a y be m o r e t h a n

one edge f r o m a v e r t e x v to a v e r t e x w. Note, however , t h a t two d i f fe ren t edges f rom

v to w m u s t be l abe l led d i f ferent ly . The s u b g r a p h s cons i s t ing of al l the v e r t i c e s and

those edges which have the s ame label a r e ca l led the constituents of X.

254

DEFINITION 3

Let K be a set of p e r m u t a t i o n s in S n. The lu, beUed cycle graph X g o f tf is the label led

cycle g raph whose/KI c o n s t i t u e n t s are the cycle graphs X=, ~ e K.

Thus, Xt~ is of degree 1K[. Observe t h a t we can c o n s t r u c t X K f rom K in O(n-]Kl)

steps,

~ L E O

Let G = <K>, where K = f(1,~,3,4), (2,4)t, Here G = D 4, the d ihedra l group of degree 4.

The label led cycle graph XK of K is shown in Figure 4 below. []

[ <----------~4

// //

$/,1

Figure 4

Because of the labelling of X K and by Lemma 4, the following is obvious.

L~MA 10

Let G = <K>, K a subset of S n. Then Csa(G ) = Aut(XK), where X K is the labelled cycle

graph of K.

We will show how to test isomorphism and determine t.he automorphism group of

labelled cycle graphs.

DEFINITION 4

Let X be a connected labelled cycle graph, x and y distinct vertices of X. The cycle

d/stance d(x,y) of x and y is defined by

(i) d(x,y) = 0 iff x and y lie on a common cycle in X whose edges are uniformly

labelled.

(2) d(x,y) = k iff z~ ..... z k is the smallest number of vertices in X such that

d(x, zl) = O, d(zk,y) = O, and d(zi,zi+~) = O, i -~ i < k.

~Ju~ 7

Let X be the connected labelled cycle graph shown in Figure 5 below.

Figure 5

255

Then d(Z ,2 ) = O, d (1 ,3 ) = l , d(Z,4) : 2.

The isomorphism tesL of labelled cycle graphs to be described next rests on the

following key observation.

THEO~ tO

Let X = (Vx, Ex) and Y = (Vy, Ey) be isomorphic connected labelled cycle graphs. Then

every isomorphism L from X to Y is fully determined by the image x ~ o2 an arbitrary

v e r t e x x ~ Vx.

Proof Without loss of generality we assume that both X and Y have more than

one vertex. So, .let ~ be an isomorphism from X to Y, x any vertex in X, and x ~ the

image of x under L in Y. Since cycles of uniformly labelled edges are mapped again

into such cycles whose edges have the same label in Y, and since cycles in every con-

stituent are disjoint, it follows that the image of every vertex u in X of cycle distance

0 from x is determined by xt Since vertices z in X of cycle distance i from x have a

cycle distance 0 from some other vertex u in X with d(x,u) = 0, the same argument

shows that the image of every such z is fully determined by u ~ and therefore by xt

Proceeding by induction on the cycle distance from x, since both X and Y are con-

nected, it follows that the image of every vertex of X under ~ is fully determined by xt

From Theorem iO, we obtain the following corollaries:

COROLLARY 2 (Fontet, Hoffmann)

If X and Y are connected labelled cycle graphs of degree k with n vertices each, then

isomorphism of X and Y can be tested in O(n2.k) steps.

Proof Let x be a fixed arbitrary vertex of X. In O(n-k) steps we can test, for each

vertex y in Y, whether mapping x into y determines an isomorphism. •

Note XK is a connected multigraph iff the group <K> is transitive. Consequently,

Theorem i0 is a restatement of Theorem 5 of Chapter V. It is a stronger form of the

theorem, since the fact that AuL(X) is semiregular for some graph X does not neces-

sarily imply that there is a simple procedure for deciding whether mapping a vertex x

to a vertex y may be extended to an automorphsm of X]

COROILA~ 3

If X is a connected labelled cycle graph of degree k with n vertices, then generators

for Aut(X) can be determined in 0(n2.k) steps~

Proof Immediate from Corollary 2 and the observation that IAut(X) f ~ n. -

256

We now consider how to test isomorphism of labelled cycle graphs which are not

necessarily connected. ~ntuitively, we will split the graphs X and Y into components,

and test isomorphism of each component. Having classified the components into iso-

morphism c lasses , i t is c l ea r t h a t X and Y a re i s o m o r p h i c iff exac t l y half the com-

ponen t s in e a c h i s o m o r p h i s m c lass be long to X.

THZORE~ 11 (Fontet, Hoffmann)

Let X and Y be l abe l l ed cyc le g r a p h s of deg ree k wi th n v e r t i c e s each . Then we can

t e s t i s o m o r p h i s m of X and Y in 0(n&k) s t eps .

Proof In 0(n.k) s t e p s we can find the c o m p o n e n t s of X and Y. If X and Y do no t

have an equal n u m b e r of c o m p o n e n t s of equal size, t hen t h e y c a n n o t be i somorph ic .

So, l e t X and Y have Pi c o m p o n e n t s of size mi, i <- i-< r. We t e s t a t m o s t pi g com-

ponen t s of size m i for i s o m o r p h i s m . By Corol la ry 2, th is r e q u i r e s a t o t a l of

r 0(pig.mi&k) s t eps . Since ~ p rmi = n, i s o m o r p h i s m of X and Y can be t e s t e d in O(nS-k)

i = t

s teps . I t is c l e a r t h a t we can c o n s t r u c t an i s o m o r p h i s m in the s a m e t ime bound, i

We t h e r e f o r e ob t a in t he following

CORO~ 4

if X is a l abe l l ed cyc le g r a p h of d e g r e e k wi th n ve r t i c e s , t h e n g e n e r a t o r s for Aut(X)

can be found in 0(nS.k) s t eps .

Proof Aut(X) is g e n e r a t e d by a s e t K1uK~, where K1 g e n e r a t e s all a u t o m o r -

p h i s m s which fix se twise the v e r t i c e s of e ach c o m p o n e n t of X, and K s g e n e r a t e s all

poss ib le p e r m u t a t i o n s of i s o m o r p h i c c o m p o n e n t s of X.

Let X have Pi c o m p o n e n t s of s ize mi, 1 - i - r. We can find t h e s e c o m p o n e n t s in

r g

0(n-k) s t eps . By Corol la ry 3, we find t h e se t K 1 in 0 ( ( ~ p r m i ).k) s t eps , which is dom- i = I

i n a t e d by 0(rig-k).

By T h e o r e m 1 i, in 0(nZ-k) s teps , we can c lass i fy t he c o m p o n e n t s of X into i somor -

p h i s m c lasses and can find, in eve ry i s o m o r p h i s m class , an i s o m o r p h i s m f rom an a rb i -

t r a r y r e p r e s e n t a t i v e c o m p o n e n t to each c o m p o n e n t in t he class. I t is t r iv ia l to p ro -

duce the s e t Ks f rom t h e s e maps . Note t h a t bo th IKll and ]Ksl a re 0(n). =

ENASI'I~ 8

Let K = t(1,2,3,4)(5,6,7,8), (2,4)(5,7)t. The l abe l l ed cyc le g r a p h XI{ is of d e g r e e 2 and is

shown in F igure 5 below. X K has two i s o m o r p h i c c o m p o n e n t s wi th

= ( i ,6)(2,7)(3,8)(4,5) e s t ab l i sh ing t h e i s o m o r p h i s m .

257

/ / / /

/ , / / / / /

4 e 3

6 < - - ~ 5

/x~// / / $ , / /

7 ~ )8

Figure 6

Note t h a t ~ is comple te ly d e t e r m i n e d by, for ins tance , (i ,6). F u r t h e r m o r e , the auto-

morph i sms setwise stabil izing the c o m p o n e n t ver t ices are g e n e r a t e d by a = (1,3)(2,4)

and ~ = (6,8)(5,7). Thus, Aut(XK) = <q a, fl>. E]

Corollary 4 has es tab l i shed tha t P rob lem i8 is in P and so we could design a cen-

t ra l izer a lgor i thm based on the proof of the corollary. However, we can do b e t t e r and

will show how to c o n s t r u c t a r e p r e s e n t a t i o n ma t r ix for the cen t r a l i ze r in the same

t ime bound. Here it will be useful to rewri te the cen t ra l i ze r as d i rec t p r oduc t of

wrea th p roduc t s of semi regu la r groups.

Let X K be the cycle g raph of the gene ra t ing se t K of the group G < S n. We spli t X K

into its c o n n e c t e d componen t s , which we then classify into the i somorph i sm classes

J1 . . . . . Js. Since c o m p o n e n t s in d i s t inc t classes are no t isomorphic , it follows t h a t

Aut(XK) and therefore Cs~(<K> ) is the (external) d i r ec t p r o d u c t

Wl×W~x • • • xW~

of ce r t a in groups W i which act, respect ively, on the ver t i ces of the c o m p o n e n t s in the

class Ji.

Next, le t Ji cons is t of m i componen ts . Since these c o m p o n e n t s are i somorphic

graphs, they can be p e r m u t e d in all mi! ways. F u r t h e r m o r e , s ince the a u t o m o r p h i s m

groups of i somorphic graphs are isomorphic, it follows tha t the group W i is i somorphic

to the wrea th p roduc t of a group C i by the s y m m e t r i c group Smi. Here the group C i i s

the a u t o m o r p h i s m group of a r ep re sen t a t i ve c o m p o n e n t in the class Ji.

The groups C i, in tu rn , are s emi regu la r (Theorem 5 of Chapter V, or Theorem 10

above). Therefore, a r e p r e s e n t a t i o n ma t r ix for C i m u s t con ta in only one row of

n o n e m p t y en t r i es off the diagonal, i.e., i t is s imply a l is t ing of i ts e lements .

With these s t r u c t u r a l p roper t i es of Cs.(G ) in mind , we now discuss how to con-

s t ruc t a r e p r e s e n t a t i o n ma t r ix M w for a d i rec t factor W of Cs.(G ) f rom the cycle graph

X K of G = <K>. Let J = IZ 1 . . . . . Zmt be a class of i somorphic connec t ed c o m p o n e n t s of

258

XK, where Zi = (Vi,Ei) is a connected component with t vertices and G is the (semiregu-

far) automorphism group of ZI. We assume given a representation matrix M I for G

and m isomorphisms ~i from Z 1 to Z i. Here we set nl = 0.

We describe how to construct a representation matrix M w for W = GrbSm . For this

construction (but not for its timing) the semiregularity of G is immaterial. M w is first

subdivided into the txt submatrices Mid, 1 -< i,j --- m. Clearly, all entries of Mi4 should

be empty whenever i > j. The diagonal submatrices Mi, i should be representation

matrices for the automorphism groups of the components Z i. Let vl, ..., v t be the

enumeration of the rows and columns of M 1, We enumerate the rows and columns of

M w in the sequence

V I, - . , V t , Vl ~z, . . . , Vt ~r2, , , , , Vl ~rrn, ...~ Vt ~rm

and le t M~, i (1-< i-< m) be the m a t r i x o b t a i n e d f r o m M 1 by r ep l ac ing the n o n e m p t y

e n t r y q with ~ ' i By Corol lary 2 of C h a p t e r II, Mi, i is a r e p r e s e n t a t i o n m a t r i x for

Aut(Zi).

Observe t h a t ~r/-l~j def ines an i s o m o r p h i s m which m a p s t h e c o m p o n e n t Z i to Zj.

This i s o m o r p h i s m is eas i ly a l t e r e d to an i s o m o r p h i s m ~A,j which is a p e r m u t a t i o n

exchanging Z~ with Zj and fixing the r e m a i n i n g c o m p o n e n t s pointwise. Now the m a t r i x

Mi, j, j > i, is o b t a i n e d f rom the m a t r i x Mi, i by r ep l ac ing the n o n e m p t y e n t r y ~ in Mi, i

with ~ i , j - The c o r r e c t n e s s of th is s t ep follows f rom T h e o r e m 4 of C h a p t e r II. We

observe t h a t the c o n s t r u c t i o n of M w given M I and t h e m a p s ~i r e q u i r e s 0( t3 .m 2) s teps .

However, if G is s e m i r e g u l a r , t h e n M1 con ta in s a t m o s t 0( t ) n o n e m p t y en t r i e s . In th is

case, the t i m e r e q u i r e d to c o n s t r u c t M w r e d u c e s to O(t&m ~) s t eps .

Final ly, c o n s i d e r c o n s t r u c t i n g a r e p r e s e n t a t i o n m a t r i x for G = Wlx • • - xW r given

r e p r e s e n t a t i o n m a t r i c e s M i for the g roups W i, where the Wi a re g roups with d is jo in t

p e r m u t a t i o n domains . Here i t is obvious t h a t with the p r o p e r e n u m e r a t i o n of the

po in t s in t he p e r m u t a t i o n d o m a i n of G, t h e m a t r i x M w is in b lock d iagona l form, where

the b locks a re j u s t t he m a t r i c e s M i. The r e su l t i ng a l g o r i t h m is given beiow

259

~LGO~Hi 2 (Centralizer)

Input

Output

Method

A gene ra t i ng se t K of t he p e r m u t a t i o n g roup G < S n.

A r e p r e s e n t a t i o n m a t r i x M for t he c e n t r a l i z e r Cs~(G ) of G.

I.. beg in

2. Construct the labelled cycle graph X K from the generating set K;

3. Partition X K into its connected components;

4. Classify the c o n n e c t e d c o m p o n e n t s into the i s o m o r p h i s m c lasses J l . . . . . gs

using Theo rem 10;

c o m m e n t Step 4 chooses in each i s o m o r p h i s m class an a r b i t r a r y c o m p o n e n t Z I

as c lass r e p r e s e n t a t i v e and c o n s t r u c t s an i s o m o r p h i s m ~T~ f rom ZI to e a c h of

t he r e m a i n i n g c o m p o n e n t s Z i in the class;

5, fo r e a c h i s o m o r p h i s m class J = tZ 1 . . . . . Zm~ do b e g i n

6. find all a u t o m o r p h i s m s of Z 1 using Corol la ry 4 and c o n s t r u c t f r om t h e s e a

r e p r e s e n t a t i o n m a t r i x M1 for t he group Aut(Z~);

7. f rom M1 and the i s o m o r p h i s m s ~i, c o n s t r u c t a r e p r e s e n t a t i o n m a t r i x M] for

the g roup W of al l a u t o m o r p h i s m s of t he s u b g r a p h of XK cons i s t ing of t he

c o m p o n e n t s in J ;

8. end;

9, comb ine the r e p r e s e n t a t i o n m a t r i c e s Mj into a r e p r e s e n t a t i o n m a t r i x M of

Cso(C); 10. output(M);

11. end.

The c o r r e c t n e s s of t he a l g o r i t h m is c l ea r f r o m Corol la ry 4 and the d i s cus s ion

above. Also f rom the p r e c e d i n g d i scuss ion follows

PROPOSITION 5 (Hoffmann)

Algor i thm 2 r e q u i r e s a t m o s t O(n 2. IKI) s t eps , where n is t he d e g r e e of G = <K>.

P ropos i t i on 5 should be c o m p a r e d with P ropos i t i on 2 of Chap t e r II: Due to t he

s t r u c t u r a l p r o p e r t i e s of the cen t r a l i ze r , i t is of ten c o m p u t a t i o n a l l y e~s~T to d e t e r -

mine the c e n t r a l i z e r of a g roup G f rom i ts g e n e r a t i n g se t K than , for example , t he

o r d e r of G.

260

We now turn to Problem 19 and give an efficient algorithm for determining the

center of a permutation group. It is clear that Cs~(G ) normalizes G, i.e., for all

~ Cs=(G)~ ~G = G~. We m a y therefore use Algorithm I to intersect Csn(G ) with G

from which we obtain the center of G. By Propositions 2 and 5 we obtain immediately

PROPOSITION 6 (Hoffmann)

Let G < S n be a p e r m u t a t i o n group of degree n g e n e r a t e d by a set K of pe r mu t a t i ons .

Then we can d e t e r m i n e a gene ra t ing set for CG(G), the cen t e r of G, in 0(n ~. IK] +n c)

steps.

4.3. An Automorphism Restriction Problem

We finally cons ider the following special case of P rob lem 8, Rest r ic ted Graph

Automorphism:

Given a graph X = (V,E) and a gene ra t ing set for a Sylow p-subgroup G Of Sym(V),

where p is a fixed pr ime, d e t e r m i n e the group H of all au tomorph i sms of X which are

also in G,

A polynomial time solution for this problem becomes evident once we recall that

G may be represented as the automorphism group of a regular (directed) cone graph

X G of degree p (cf. Chapter Ill, Subsection 3.2). We propose to superimpose X and X O

obtaining a new graph Y. The pointwise stabilizer in AuL(Y) of the roots of X G will be

the group H which now can be determined using the techniques of Chapter V. The

fact that X 0 is, in general, a directed cone graph presents no problems and is handled

by routine modifications of the algorithms.

As i l lus t ra t ion of the approach, cons ider the following

~b~PLE 9

We are given the Sylow 2-subgroup G = <(1,3), (5,6), (1,2)(3,4)> of S 8 and the graph

K3,~ = X = (1t ..... 6~, I(1,4), ( ! ,5), (1,6), (2,4), (2,5), (2,6), (3,4), (3,5), (3,6)]), and wish to

find H = Aut(X) C~ G.

The graph X O is shown in Figure 7 below. We superimpose X and Xo in the natural

way, and obtain the graph Y of Figure 8. Note that the stabilizer of a and b in Aut(Y)

is nowAut(X)C~G...~

261

1 3 2 4 5 6

R e p r e s e n t a t i o n of G

Figure 7

v, vv a

The Graph Y

Figure 8

Therefore , we ob t a in

PROPOSITION 7 (Hoffmann)

For a fixed p r i m e n u m b e r P r o b l e m 20 is in P.

A c o n s i d e r a b l y h a r d e r p r o b l e m would be finding a Sylow p - s u b g r o u p of the au to -

morphism group ol the graph X. While it is true that every Sylow p-subgroup of Aut(X)

is the intersection of Aut(X) with some Sylow p-subgroup of Sym(V), it is not clear how

to even recognize that a given Sylow p-subgroup of Sym(V) is suited for this purpose.

5. Normal Closure, Commutator Subgroups, Solvability, and Nilpotence

There is a large va r i e t y of g r o u p - t h e o r e t i c p r o b l e m s which a r i se as m a t h e m a t i -

cany i n t e r e s t i n g ques t ions and have no d i r e c t r e l a t ionsh ip with g r a p h i somorph i sm.

In this sec t ion , we s amp le a few of these p rob l ems . We have s e l e c t e d p r o b l e m s which

are in P and whose solut ions, f u r t h e r m o r e , involve easy va r i an t s of the m e t h o d s of

Chap te r II.

5. t. Normal Closure

262

DEFINITION 5

Let H be a subgroup of G. The norrncd closure of H in G is the sma l l e s t n o r m a l sub-

group N of G which con ta ins H.

Since the n o r m a l c losure is the i n t e r s e c t i o n of ai1 n o r m a l subgroups of G which

con ta in H, i t is c l ea r t ha t H has a unique n o r m a l c losure in G.

~ l0

Consider G = S 4 = <(1,2), (1,2,3,4)> and the subgroup H = Ca = < ( t & a ) > . Since

(t,2,3) O,&&4) = (2,3,4) ¢ H, H is no t its own n o r m a l c losure and m u s t con ta in the sub-

group A 4 = <(1,2,3), (2,3,4)>. Since A 4 is n o r m a l in S 4, i t is the n o r m a l c losure of H in

G. []

We will develop an a lgo r i t hm for finding the n o r m a l c losure of a subgroup with

known gene ra t ing set. The a lgo r i t hm witl be appl ied l a te r to c o n s t r u c t c o m m u t a t o r

groups and to Lest solvabi!i ty and n i lpo tence of p e r m u t a t i o n groups. Specifically, we

cons ide r

PROBL~I 2t (Normal Closure)

Given a gene ra t i ng se t K~ for the p e r m u t a t i o n group G < S n and a gene ra t i ng se t K H

for a subgroup H of G, d e t e r m i n e g e n e r a t o r s for the n o r m a l c losure of H in G,

The a lgo r i t hm to be given for P r o b l e m 21 will be an ex tens ion of Algor i thm 3 of

Chapte r It. Conceptual ly , we begin with the c o n s t r u c t i o n of a r e p r e s e n t a t i o n m a t r i x

Mo for H. We t h e n c o n s t r u c t a se t K 1 consis t ing of the en t r i e s in M 0 con juga t ed u n d e r

every g e n e r a t o r in K G and c o n s t r u c t f r o m M 0 and K 1 a r e p r e s e n t a t i o n m a t r i x M 1 for

the group H1 = <HLjKI>. We r e p e a t this p rocess for H t provided H is a p r o p e r sub-

group of H~. Continuing in this fashion, the ascend ing cha in of subgroups

H < H I < H~ < - • " < Hr = Hr+t

of G eventual Iy b e c o m e s s t a t iona ry with a group H r which is the no rma l c losure of H in

G. The following t h e o r e m s t a t e s this a p p r o a c h prec ise ly :

T H E O ~ i2

Let H = <KH> be a subgroup of G = <KG>, a p e r m u t a t i o n group of deg ree n, and le t Ko

be the se t of n o n e m p t y en t r i e s in a r e p r e s e n t a t i o n m a t r i x ?go of H 0 = H. For i-> l,

263

define reeurs ive ly

Hi = <Ki-i DKi>,

Ki = the set of all nonempty entries in M i, a representation matrix for H i.

Then there exists an integer 0 - r --- logs( 1G I) such that N = H r = Hr+ I is the normal

closure of H in G.

Proof Clearly H i is a subgroup of Hi÷ 1 and of G for all i-> 0. If H i is a p roper sub-

group of Hi+ l, t hen the order of Hi+ 1 is a t leas t twice the order of H i. On the o ther

hand, if Hi is equal to Hi+ l, t hen also Hi = Hi+j, j > 0, and the subgroup chain is s ta t ion-

ary pas t i. Since H i < G, the chain m u s t become s ta t ionary pas t r = log2( [ G[).

Now let N be the n o r m a l closure of H in G. Clearly H i < N for all i-~ 0. Fu r the r -

more, if Hr = Hr÷l, t hen Hr is n o r m a l in G. Hence Hr = N.

A good i m p l e m e n t a t i o n of the n o r m a l etosure a lgor i thm should no t imi ta te

Theorem 12 by cons t ruc t ing the groups H i in succession, since this leads to sifting

many r e d u n d a n t pe rmu ta t i ons . Ratherl as each new e n t r y into the r e p r e s e n t a t i o n

ma t r ix is made, we add to the queue of p e r m u t a t i o n s to be sif ted subsequen t ly no t

only pai r p roduc t s bu t also all conjugates of the new en t ry u n d e r each ge ne r a t o r of G.

The resu l t ing a lgor i thm is now as follows:

264

kLeml~nlM 3 ( N o r m a l Closure)

Input

Output

Method

~_. begin

2.

3.

4.

Generating sets K H and K G of the permutation groups H and G, respec-

tively, where H < G < S n.

Representation matrix M of the normal closure N of H in G.

in i t ia l ize a r e p r e s e n t a t i o n m a t r i x M to r e p r e s e n t t he t r iv ia l g roup I;

Queue := KH;

while Queue is not empty do begin

5. r e m o v e f f r om Queue;

8. if sif t ing ~ causes the new e n t r y ~0 in M t h e n beg in

7. add to Queue all pa i r p r o d u c t s f o r m e d with ? and with n o n e m p t y en t r i e s in

M;

8. add to Queue h0 ~ [ ~ e KHt;

9. end;

tO. end;

11. output(M);

12. end.

Let I%{ be the final matrix output by Algorithm 3. Since M is closed under pair product

sifting, it is the representation matrix of a group N. Clearly, H < N < G. The group N

is normal in G because the conjugation of each entry of M under every generator of G

is again in N. Moreover, each entry in M arises as a product of generators of H, gen-

erators of H conjugated under generators of G, and inverses of such permutations.

Therefore N is the normal closure of H in G, i.e., Algorithm 3 is correct. The time

required by the algorithm is analyzed in the following

PROPOSITION 8 (Furst, Hopcroft, Luks, Sims)

Algorithm 8 requires at most 0(IKHI -nZ+ IKGI -n4+n 6) steps.

Proof Queue contains initially IKH] permutations. For each of the up to 0(n 2)

entries made in M, we add up to 0(n z) pair products and IKGI conjugates. Hence a

total of at most 0(!KHi + n4+ iKGl.n ~) permutations are sifted, from which the

stated bound follows. "

265

5.~. C o m m u t a t o r s a n d C o m m u t a t o r Groups

A n u m b e r of i m p o r t a n t group proper t ies can be fo rmula ted in t e r m s of c o m m u t a -

tors and c o m m u t a t o r derived subgroup towers. We now develop the necessa ry con-

cepts to express solvability and n i lpo tence of groups in this f ramework.

DEFINITION 6

For the e l emen t s 7r and ~ of a group C, we call [~T,~] = 7r-I~-lTr~p the c o m m u t a t o r of ~r

by]b~

The t e r m commutaLor derives from the iden t i ty ~ = ~ [ ~ , ~ ] . The following

l e m m a s ta tes this and o ther e l e m e n t a r y ident i t ies for c o m m u t a t o r s and is easy to

prove.

L ~ i i

(a) ~ = ~[~,~].

(b) [Tr,lp] = [~p,n'] -1.

(c) [ ~ , ~ ] = [~ ,~] [~ ,~ /F.

(d) [ ~ , ~ = [~ ,~ /F[~ ,~] .

Extending Definition 6 for the complexes A, B c G, we se t

[A,~] = <[~ ,~ ] I a c A, ~ c B >

Note t ha t [[a~,lfl~] = <[a,fl]> is no t equal to [a,fl]. F u r t h e r m o r e , the group [A,B] con-

ta ins in genera l e l emen t s which are no t c o m m u t a t o r s of e l emen t s in A by e l emen t s in

B. This r ema ins t rue even when A and B are groups.

1 , r :Mm 12

For the subgroups A and B of G the following is t rue:

(a) [A,B] = [B,A].

(b) [A,B] <~ <A,B>.

(c) [A,B] < A iff B normal izes A.

Proof Pa r t (a) follows from Lem_ma i l b . For Pa r t (b), le t w, ~ < A, ~, X E B. By

Lemma 1 lc, [Tr,~p] x = [~r,X]-I[~T3pX] c [A,B], and by Lemma l i d ,

[n,@]~ = [n~op][~,~] -] ~ [A,B]. Thus [A,B] is no rma l in <A,B>. P a r t (c) is trivial. -

DEFnV~ON 7

The subgroup C' = <[~r,~] I ~T, ~/ ~ G> - [G,G] of G is called the cornrn~ta2o~ group or

the der ived group of G.

266

We define h igher c o m m u t a t o r g roups r ecu r s ive ly by

(t) G~ °) = G.

(2) G (i+I) = (G(i)) ' = [G(9,G(i)], 0 -< i,

and prove a result about the subgroup tower

G>G'>G (2)>G (s)> o.- >G (r)=G (r+1)

which clearly consists o~ characteristic subgroups of G. This subgroup tower is also

cal led the d e ~ v e d s e ~ e s of G.

THEOREM 13

Let N be a subgroup of G. Then N is normal in G with the abelian factor group G/N iff

G ' < N ,

P roof If N is n o r m a l in G and G/N is abel ian, then, for n, 9 E G,

u~N = (~rN)(~N) = ('pN)(vrN) = "~TN,

hence [~,~] c N by L e m m a 1 la. By the defini t ion of G', t he re fo re , G' < N.

Now a s s u m e t h a t G' < N. For ~r ~ N, Ip ¢ G we have

n ~ = w[wJp] ~ NG' = N,

hence N is n o r m a l in G. Now for ~, ~ ~ G we have

n~N = "~n[~,%']N = "¢,rrN,

so G/N is abelian.

By Theo rem !3, we may i n t e r p r e t G' as the m i n i m a l n o r m a l subgroup of G such

that G/G' is abel ian. F r o m the un iqueness of G' follows that G' is c h a r a c t e r i s t i c in G.

If G is c h a r a c t e r i s t i c a l l y s imple, t he re fo re , e i t he r G' = G, and then G is nonabet ian, or

G' = I, and t h e n G is abelian. Note t h a t G' = I iff G is abelian.

DEFINITION 8

The finite group G is so/vabt~ if G has a composition series in which every composition

fac to r is abelian,

We can c h a r a c t e r i z e solvabil i ty using c o m m u t a t o r groups . The following r e su l t

follows readi ly f r o m Theorems 13 above and f r o m T h e o r e m s 2 and 3 of Chap te r ¥:

PROPOSITION 9

The finite group G is solvable Lff G (r) = I for all r ~ !ogz( [ G I )-

267

Next, we d i scuss n i lpo tence of g roups . Recal l t he def in i t ion of a c e n t r a l s e r i e s for

t he g roup G (Chap te r III, Definit ion i0).

DEFINITION 9

A group G is n / / p o t e n t if G has a c e n t r a l ser ies .

F r o m Chap t e r III we know t h a t every (finite) p -group is n i lpo ten t . However, t he

c lass of n i l p o t e n t g roups is l a r g e r t han the c lass of p -groups . I t inc ludes , for example ,

the abe l i an groups . We define r e c u r s i v e l y the following subg roups of the f ini te g roup

G:

Ks(G ) = G, and

Ki÷I(G ) = [Ki(G),G ].

We will exp re s s n i l po t ence in t e r m s of the subg roup tower

G : KI(G) >K2(G) > ' ' - >Ks(G ) = KS÷I(G)

which c l ea r ly cons i s t s of c h a r a c t e r i s t i c subg roups of G.

I ~ M ~ 13

The subgroup tower

G = N (1 )>N (~)> - . . > N ( t ) = I

is a c e n t r a l se r i e s for G iff [N0),G] < N 0+0.

Proof Let [N(0,G] < N 0+0 < N 0). By L e m m a 12c, N O) is n o r m a l in G. F u r t h e r -

more , for 7r c N (0 and ~ ~ G, we have lr~bN (i+1) = ~T[~r,~]N 0+1) = ~pnN (i+l), h e n c e 7rN 0+0

c o m m u t e s with ~kN C+O, i.e., N(0/N C+0 is a subg roup of the c e n t e r of G / N (i÷1).

Conversely , a s s u m e t h a t t he g roups N (0 fo rm a c e n t r a l ser ies . Then, for 7r c N (i)

and ~p c G we have n~N 0+0 = lpTrN (i+~), t hus [~r,~] c N 0+0, i.e., [I@),G] < N (I+1), -

Since Ki+I(G ) = [K~(G),G] and by L e m m a 13 above, the subg roup tower

G = KI(G) >K~(G) > . . . >Ks(G ) =KS+I(G)

is a c e n t r a l s e r i e s for G p rov ided t h a t Ks(G ) = t. In th is case we ca l l t he tower t h e des-

cending central series of G.

THEOR~I 14

G is n i l p o t e n t iff Ks(G ) = I for all s -> log~( I G ] ).

P roof By L e m m a 13, G has a c e n t r a l s e r i e s if Ks(G) = I, and so G is n i lpo ten t .

Conversely, a s s u m e t h a t G has the c e n t r a l se r ies

268

G=N O)>N (~)> .~. >N (t)=I.

We vdll show that Ki(G ) < N (i). Clearly KI(G) = G = N (I). Assume inductively that

Ki(G ) < N (9. Then, by Lem_ma 13, Ki+I(G ) = [Ki(G),G ] < [N(i),G] < N (i+0.

The bound for s follows from the fact that the Ki(G ) tower becomes stationary

after the smallest value of i such that K~(G) = Ki+I(G ). "

It is clear that nilpotence implies solvability. For if G has the central series

G = N ( ' ) > N (z)> - - > N ( r ) = I ,

t hen the f ac to r g roups N(9/N (i+1) a r e c e r t a i n l y abel ian , hence G is solvable. However,

n i l po t ence is the s t r o n g e r p r o p e r t y :

~Puz t !

Cons ider G = S4 = <( t ,2) , (1,2,3,4)>. Then G 0) = A 4 = <(1,2,3), (2,3,4)>,

G (z) = V 4 = <(1,2)(3,4), ( t ,3)(2 ,4)>, and G (s) = I. Consequent ly , S 4 is solvable. However,

Kz(G ) = Ks(G ) = A4, and therefore S 4 is not nilpotent. [3

5.3. Testing So!vabi/ity and Nflpotence

By the resuks of the preceding section, testing solvability and nilpotenee of per-

mutation groups may be based on an efficient algorithm for constructing the commu-

tator group [A,B] of the groups A and B in S n. We therefore consider the following

three problems:

PROBLEM 22 (Commutator Group)

Given generating sets K A and KB for the groups A and B, respectively, where A, B < S n,

find a generating set for the commutator group [A,B].

PROBLEM 2:3 (Solvability)

Given a generating set K for G < S n, test whether G is solvable.

PROBLEM 24 (Nilpotence)

Given a generating set K for G < S n, test whether G is nilpotent.

A seeming diff icul ty in des igning an ef f ic ient a l g o r i t h m for P r o b l e m 22 is f inding a

suitable generating set for [A,B] given generating sets for A and for B. Since the

groups A and ~ may be large, we should not attempt to construct the set

269

K = f [a,~] I a E A, fl E B I which is c l ea r ly a gene ra t i ng se t for [A,B]. The following

t h e o r e m reso lves th is diff icul ty by reduc ing P r o b l e m 22 to P r o b l e m 21, finding the

n o r m a l c losure :

THEOEEM 15

Let A : <KA> and B : <KB> be two p e r m u t a t i o n g roups of deg ree n. Then [A,B] is the

normal closure in <A,B> of the group generated by the set

KI= I [~.~] I ~ ~ KA,~ cKs I.

Proof By L e m m a l~b, [A,B] is n o r m a l in <A,B>. Hence i t suffices to show t h a t

eve ry c o m m u t a t o r [a,fl], a c A, fl c B, is in the n o r m a l c losure N of H = <K~>. Since

K A g e n e r a t e s A and K s g e n e r a t e s B,

[~,~] = [ a l a 2 ' ' ' ~ , t ~ 1 ~ 2 ' ' " t~.],

where a i o r ai -1 is in KA and flj or Bj -1 is in Ks, 1 -< i -< r, 1 -< j -< s. By r e p e a t e d app l i ca -

t ion of L e m m a 11, t he re fo re , [aft] can be e x p r e s s e d as finite p r o d u c t of e l e m e n t s of

the form [Bj,ai]? or [ai,flj] ~, w h e r e a~ ~ K A, f l j< K s, and y E <A,B>. Therefore ,

[~,~] c N, a ~ d s o [A ,S~ = N. -

F r o m T h e o r e m 15 we i m m e d i a t e l y ob ta in po lynomia l Lime a lgo r i t hms for P rob -

l ems 22 t h r o u g h 24:

COROLLARY 5 (Furst, Hopcroft, Luks, Sims)

Given gene ra t i ng se ts K A and K s for the g roups A and B, r e spec t i ve ly , where A, B < S n,

a r e p r e s e n t a t i o n m a t r i x for [A,B] can be d e t e r m i n e d in 0((IKAI + I Ks l ) .ne+n 6) s t eps .

Proof In 0 ( ( IKAI+IKs i ) -n z + n ~) s t eps we c o m p u t e r e p r e s e n t a t i o n m a t r i c e s for

A, B, and <A,B>, f rom which we ob ta in gene ra t i ng se ts for e ach group of size O(nZ).

F rom these , we can c o m p u t e an 0(n a) se t Ka of Theo rem 15. The co ro l l a ry now follows

f rom Theorem 15 and P ropos i t i on 8, •

COROLLARY 6 (Furs t , Hopcrof t , Luks , S ims)

Given a gene ra t i ng se t K for the group G < S n, we can t e s t in 0( I K] 'n2+n°"log2( I G [))

s t eps w h e t h e r G is n i l p o t e n t or solvable.

Proof I m m e d i a t e f rom Propos i t ion 9, Theorem 14, and Corol lary 5. •

270

6. Open Problems

Despi te much r e sea rch , an a c c u r a t e complexity classif icat ion of Prob lems 1

through I0 remains open. In Section 2 of this chapter, we have given technical evi-

dence suggesting that none of these problems is NP-complete. However, we conjec-

ture thai at least Problems I through 8 do not belong to P.

We now discuss additional open problems. Problem 35 below is a question related

to Problem i, Double Coset Membership. Problem 36 generalizes the difficulty

encountered in Chapter Ill in the design of a subexponential algorithm for (regular)

cone graphs. It also generalizes the Labelled Graph Autornorphism Problem (Problem

i of Chapte r III) in the g r o u p - t h e o r e t i c sense.

PI~OBLES 25 (Double Coset Partition)

Given gene ra t i ng sets ior the p e r m u t a t i o n groups A, B < S n, d e t e r m i n e r, the n u m b e r

of d i s t inc t double cosets of A and B in S n.

PI~BLEM 26 (Intransitive Subgroup Problem)

Given g e n e r a t o r s for the group G = C~xGzx . - . xGm < S~. where !Gil ~ k for some

constant k, and given a polynomial time membership test for a subgroup H of G as

well as for the subgroups H i = Hf~Gi, determine generators for H.

P r o b l e m 25 has the following specia l case: If the groups A and B are r e s t r i c t e d to

the groups occur r ing in P rob l ems 9 and 10, t h e n P r o b l e m 25 asks for the n u m b e r of

non i somorph ic graphs with n v e r t i c e s and a p r e s c r i b e d n u m b e r of edges. It is not evi-

den t t ha t even the r e s t r i c t e d vers ion of P r o b l e m 25 is in NP. There is, however , a

subexponen t i a i (in n) a lgo r i t hm for this r e s t r i c t e d version.

We po in ted out ea r l i e r t h a t an eff ic ient a lgor i thm for P r o b l e m 26 entai ls a subex-

ponential algorithm for (regular) cone graphs along the ideas of Section 2 of

Chapter Ill. Such an algorithm would also entail an efficient isomorphism test for

graphs whose adjacency matrices have eigenva!ues with multiplicities bounded by a

fixed constant, but this class of graphs can be handled using a slightly different

approach. A good starting point for Problem 36 might be the following restriction:

271

PROBLEM 27 (Subspace Problem)

Given a polynomial t ime m e m b e r s h i p t e s t for a l inear subspace S < 7_e m of the m-

d imens iona l vec tor space over the in tegers modulo 2, and given the vec tors

v 1 . . . . . v k ~ Z~ m such tha t dim(<S, v 1 . . . . . Vk> ) = dim(S)+k, de t e r mi ne the d imens ional -

ity of S.

Typically, the subspace S of P rob lem 27 (or the subgroup H of P rob lem 26) is

specified as the a u t o m o r p h i s m group of a combina tor ia l s t ruc tu re , e.g. a g raph or a

matr ix . So far, every successful a t t ack on such ins t ances of P rob lem 27 has exploited

special p roper t ies of the combina to r i a l object and therefore canno t be general ized.

Finally, we m e n t i o n a class of open prob lems more d i rec t ly re la ted to graph iso-

m o r p h i s m abou t which li t t le seems to be known: Given a graph X whose au tomor -

ph ism group is to be de t e rmined . Which addi t ional a p r i o r i i n fo rmat ion about Aut(X)

simplifies this p rob lem? For example, such addi t ional in fo rmat ion could be t h a t

Aut(X) is e l e m e n t a r y abel ian, or t h a t the s tabi l izer of a ve r tex v, Autv(X ), is a known

p e r m u t a t i o n group G. The Labelled Graph Automorph i sm P r ob l e m is a solved

ins tance of this quest ion.

Related to the spir i t of this p rob lem area is the open p rob lem of tes t ing isomor-

ph ism of rigid graphs, i.e., of graphs tha t have only tr ivial au tomorphisms . It is

known tha t this p rob lem is equivalent to recognizing rigidity, b u t it is no t known to be

i somorph i sm complete . Moreover, an i somorph i sm be tween two rigid graphs is

unique.

7. Notes and References

Brown, Hjelmeland, and Masinter [ 1974] consider double coset partitions of S n as

nonisomorphic vertex labellings of graphs, and so interpreting double cosets as iso-

morphism classes is implicit in their work. Problem 25 is from that paper, and the

authors give an exponential algorithm for it.

The g roup- theore t i c genera l iza t ion of g raph i somorph i sm is f rom Hoffmann

[1981a]. P rob lems 7 and 8 were proposed in pr iva te co r r e spondence by, Luks who also

proved first the polynomial t ime equivalence of P rob lems 5 th rough 8. The re la t ion-

272

ship between testing membership in double cosets and intersecting permutation

groups is f rom Hoffmann [1980b], The idea of the proofs of Theorems ~ and 3 goes

back to an exponential algorithm for intersecting permutation groups of Sims

[1971b]. Problem 4 was considered by Kannan and Lipton (unpublished) who proved

the equivalence of Problems 4 and 5, The proof given here (Theorem 5) is due to Lul<s

(private communication).

The structure of problems in NP which are neither complete nor in P has been

studied by Ladner [1975]. Valiant [1979] discusses the complexity of enumeration

problems. A brief survey of the material appears in Garey and Johnson [ 1979].

Problem 15 and Theorem 9 are from Hoffmann [1980b], as are Problem 13 and

Algorithm I. Proposition 4 is noted in Luks [1980].

Section 4.2 follows Hoffmann [ 1981]. According to C.C. Sims, the centralizer algo-

rithm dates back to at least 1970. A published version can be found in Fontet [1977].

The center algorithm, in contrast, appears to be new.

The presentation of the group-theoretic material in Section 5 follows Huppert

[1967]. Fursh, Hoperoft and Luks [i980b] give a polynomial time algorithm for com-

puting the normal closure. Their version is based on the proof of Theorem 12. Algo-

rithm 3 is superior and was suggested by C.C. Sims. Furst, Hopcroft and Luks [i980b]

apply the normal closure algorithm to test solvability, but do not mention the nilpo-

tence test.