doi.org/10.26434/chemrxiv.10031888.v1

Visualizing 3D Molecular Structures Using an Augmented Reality AppKristina Eriksen, Bjarne Nielsen, Michael Pittelkow

Submitted date: 23/10/2019 • Posted date: 25/10/2019Licence: CC BY-NC-ND 4.0Citation information: Eriksen, Kristina; Nielsen, Bjarne; Pittelkow, Michael (2019): Visualizing 3D MolecularStructures Using an Augmented Reality App. ChemRxiv. Preprint.

We present a simple procedure to make an augmented reality app to visualize any 3D chemical model. Themolecular structure may be based on data from crystallographic data or from computer modelling. This guideis made in such a way, that no programming skills are needed and the procedure uses free software and is away to visualize 3D structures that are normally difficult to comprehend in the 2D space of paper. The processcan be applied to make 3D representation of any 2D object, and we envisage the app to be useful whenvisualizing simple stereochemical problems, when presenting a complex 3D structure on a poster presentationor even in audio-visual presentations. The method works for all molecules including small molecules,supramolecular structures, MOFs and biomacromolecules.

File list (1)

download fileview on ChemRxivAR-pittelkow.docx (7.95 MiB)

Visualizing 3D molecular structures using an augmented reality app

Kristina Eriksen, Bjarne E. Nielsen, Michael Pittelkow

Department of Chemistry, University of Copenhagen, Universitetsparken 5, DK-2100Copenhagen, Denmark.

E-mail: pittel@chem.ku.dk

ABSTRACT

We present a simple procedure to make an augmented reality app to visualize any 3D

chemical model. The molecular structure may be based on data from crystallographic data or

from computer modelling. This guide is made in such a way, that no programming skills are

needed and the procedure uses free software and is a way to visualize 3D structures that are

normally difficult to comprehend in the 2D space of paper. The process can be applied to

make 3D representation of any 2D object, and we envisage the app to be useful when

visualizing simple stereochemical problems, when presenting a complex 3D structure on a

poster presentation or even in audio-visual presentations. The method works for all

molecules including small molecules, supramolecular structures, MOFs and

biomacromolecules.

Journal 10/23/19 Page 1 of 18

5

10

15

20

GRAPHICAL ABSTRACT

KEYWORDS

Augmented reality, Unity, Vuforia, Application, 3D models.

INTRODUCTION

Conveying information about three-dimensional (3D) structures in two-dimensional (2D) space, such

as on paper or a screen can be difficult. Augmented reality (AR) provides an opportunity to visualize

2D structures in 3D. Software to make simple AR apps is becoming common and ranges of free

software now exist to make customized apps. AR has transformed visualization in computer games and

films, but the technique is distinctly under-used in (chemical) science.1

In chemical science the challenge of visualizing in 3D exists at several levels ranging from teaching of

stereo chemistry problems at freshman university level to visualizing complex molecular structures at

the forefront of chemical research. Visualization can be especially challenging since molecules are

getting larger and more complex and span 3D. An elegant way to visualize molecules in 3D is to 3D

print the desired structure, and protocols of how to do this starting from molecular structures have

recently been described.2 To describe the geometry and symmetry of complex molecules chemists are

often forced to draw molecules in simplified or schematic ways and thus neglect information. One

example of a highly complex molecule that is difficult to display in 2D is the Molecular Borromean

Journal 10/23/19 Page 2 of 18

25

30

35

40

Rings prepared by Stoddart and co-workers (Figure 1a). In their structural representation some atoms

and labelling of atoms are omitted to simplifying the structure.3 In the simplified 2D image bonds and

atoms are overlapping and thus still make it difficult to visualize the geometry and symmetry. Many

chemists even draw the same molecule twice in different formats in the same paper to better explain

the connections of the different elements and its geometry. This is illustrated with the porphyrin nano-

ball by Anderson (Figure 1b) and the supramolecular complex between biotin[6]uril and the iodide

anion (Figure 1d).4-5 It can be challenging to come up with a new synthetic route for complicated

molecules such as Paclitaxel (Figure 1c) because it is hard to visualize how sterically congested regions

effects each other.6 For these types of problems described above, a simple way to visualize molecules in

3D would be beneficial.

Journal 10/23/19 Page 3 of 18

5

45

50

Figure 1. Examples of complex molecules that are difficult to comprehend in 2D. Download the AR app on your device (phone or tablet)to view these structures in AR. The app can be found via the QR code or the link in Figure 2. a) The Borromean rings are shown in twodifferent ways to emphasize the geometry and symmetry of it. b) The porphyrin nanoball is presented in two different ways to highlight thegeometry and the connection of the different elements in the nanoball. c) Paclitaxel is complex molecule with many stereo centers. A 3D viewof it helps to appreciate the complexity and may aid retrosynthetic analysis. d) A topview and sideview of the crystal structure of thebiotin[6]uril. Download the app by following the link below (Figure 2) and see how the AR works.

In this contribution we describe how to make a simple AR app for a mobile device (e.g. phones and

tablets) to visualize molecules in 3D. It is important to emphasize that this guide is made in such a

way, that no programming skills are needed, and that only free software is used. When the app is

made it is free to transfer the app via an USB cable from the computer to the mobile devices. However,

if you want to publish the app in Google Play it requires a one-time payment to Google Play of 25 $.

Journal 10/23/19 Page 4 of 18

55

60

65

This guide must be followed tightly, because a series of programs are needed. When the AR app is

made and transferred to a mobile device, a camera opens and it recognises an image. The image may

be on a poster, in a book or on a screen. The recognition leads to a 3D model (of one or more molecules

of your own choice) is opening as a part of the real world through your mobile device. When all the

software is installed correctly, the app is simple to make and can be used at a poster session or in

classroom. Download an example of the AR app (follow the link or QR code below) on your android

mobile device and see how it works and what it looks like (in the real augmented world). Once the app

is downloaded and opened, then point the camera at Figure 1 and 2 and a 3D model of the molecules

will appear in AR.

Link to the AR app: https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR

PFigure 2. A 2D image of Biotin[6]uril. Download the app from the link below or the QR code and point the mobile device to see the structure in3D through the camera. https://play.google.com/store/apps/details?id=com.UniCPH.Android.MoleculAR

Journal 10/23/19 Page 5 of 18

70

75

80

10

THE HOW TO GUIDE:

STEP 1: MAKING A 3D MODEL OF THE MOLECULE

If you don’t have a specific molecule or just want to make

the app to visualize a common molecule, then skip this

step and go to “Step 2: Working with Jmol”.

There are many programs that allow you to draw molecules in 3D. The most realistic 3D

models for the given molecule are either from a crystal structure or a high level optimization

calculation by using software such as “Gaussian”. It is also possible to draw the molecule in

chemical drawing software, such as “Chem3D” in the “ChemDraw” software package.

When the geometry of your molecule is satisfying, then save it at the desktop as a “mol”-file.

Only the geometry of the molecule is needed to be correct at this state. The format of your

molecule is added in the next step.

STEP 2: WORKING WITH JMOL

The format of your molecule is changed by using the

software “Jmol”. If you don’t have Jmol then download it

here: http://jmol.sourceforge.net/download/ and

Journal 10/23/19 Page 6 of 18

85

90

95

100

105

unfold the “Jmol-14.29.46-binary.zip” in a known folder. Click at “jmol.bat” and Jmol will

open. If you have any problems by running Jmol, then make sure that you are running the

latest Java version from Sun. You find and install it here: www.java.com.

If you do not have a mol-file from Step 1 then click File→Create MOL and write e.g.

“porphyrin”. Then a porphyrin molecule will appear on your screen in Jmol.

If you saved a mol-file on the desktop from Step 1 then open it in Jmol. When it is opened

click at File→Console. The console is a terminal where you write commands and thereby

change the design of your molecule. Here is a list of useful commands to change your design

of your molecule:

http://cbm.msoe.edu/teachingResources/images/jmolTraining/changingColorsAndDisplayF

ormats.pdf

If you want the spacefill format of your 3D molecule, then write “spacefill on” in the console

window and press Enter. The “Enter” activates any code in the console window. If you want

the ball-and-stick format then write “spacefill 20%,wireframe 0.15”and press Enter.

If you want all the nitrogen atoms to be blue, then write “color nitrogen blue” or if you want

the bonds to be black then write “color bonds black” etc.

When you are happy with the design of your molecule then write “write molecule.obj” in the

console, press Enter and wait ca. 60 seconds until the console reports that the data is saved

properly. Check that your molecule is saved as an MTL- and an OBJ-file in the unfolded Jmol

Journal 10/23/19 Page 7 of 18

110

115

120

125

folder. As a check you can open the “molecule.obj”-file and determine whether the format of

your molecule looks satisfying.

STEP 3: WORKING WITH UNITY

The software “Unity” is made by Unity Technologies

and is a commercial multi-platform 3D game

development tool and can be used at all levels of

computer skills. It has a free version for people who

are not using Unity at a professional level. Unity runs

on Windows and macOS but the app is only made for

“Android” devices in this paper.

Download “Unity Hub”, create an account and

activate the license. Install the program “Unity 2019.2.4f1” and in “Add modules to your

installation” select “Android Build Support” and the two subprograms “Android SDK” and

“Open JDK”! The subprograms becomes visible by clicking at the grey arrow next to “Android

Build Support”.

When Unity is downloaded then create a new 3D project under “Projects” in Unity Hub. A

new project is now open. Click (bottom left) at Project→Create→Folder and name the new

folder “molecule”.

Now, drag the molecule.MTL file into the new folder “molecule” in Unity and then the

molecule.OBJ file. The order is important! The MTL file stores the design and the OBJ file

Journal 10/23/19 Page 8 of 18

15

130

135

140

145

150

stores the geometry. A new small box showing your molecule should appear in the folder

“molecule” in Unity. Drag the box onto the “Scene window” (in the middle of the screen).

STEP 4: WORKING WITH VUFORIA

“Vuforia” is an augmented reality

software development kit (AR-SDK) for

mobile devices. It recognizes planar

images through a camera of the

mobile device and the recognition

leads to the chosen 3D model is

displayed as a part of the real-world

scene.

There are different ways to implement

Vuforia in Unity. In the newest version

of Unity (from 2019.2) is automatically installing Vuforia but it needs to be activated. Vuforia

is activated in Unity by clicking: Edit→Project settings→Player→XR Settings and check the”

Vuforia AR supported” box.

For implementing an image into your app go to the website: https://developer.vuforia.com.

Click on “Register” and go through the Creating an Account process. At the same website

click on “Develop” in the Navigation Bar and click “Get Development Key”. Type any name you

want in the box “License name”, confirm the agreement and click at “confirm” in the end. In

the new opened window click on the name you entered before and copy the license key in the

Journal 10/23/19 Page 9 of 18

155

160

165

170

grey box. Now click at “Target Manager” in Navigation Bar (next to “License Manager” in the

Navigation Bar) and press “Add Database”. Give the database a name, select “Device” and

click “Create”. A new Database is now created. Now find the image (choose an image with

high quality) you want your app to recognize and save it on your desktop. The image should

be the one you want on your poster and ideally just a picture of your chosen molecule.

Return to Vuforia and click at the new Database you just made and press “Add target”. Select

the image you just saved on the desktop and upload it. Add a name for the Database and

click “Add”. Finally click at “Download Database (All)”, select “Unity Editor” as a development

platform and save the unity package onto the Desktop and open it directly after downloading

it. When it is opened, a new window in Unity is opening too. In the new opned window in

Unity click “Import”.

STEP 5: USING VUFORIA IN UNITY

Step 5 guides you through how to build the

AR app in Unity by using Vuforia.

In the Hierarchy panel (top left) right click at

“Main Camera” and press Delete. We'll only

use the AR Camera. Now click at

GameObject→ Vuforia Engine→AR camera

and an AR camera shows up in the Hierarchy panel.

Click at “AR camera” in the Hierarchy panel and a new small window named “Inspector” will

appear. In Inspector click at “Vuforia Behaviour (Script)” and then “Open Vuforia Engine

Journal 10/23/19 Page 10 of 18

175

180

185

190

195

20

configuration”. In the new window, paste the License Key in the App License Key Box. You

copied the license key in Step 4.

Now, click at GameObject→Vuforia Engine→Image and your Image target shows up in the

Scene window. Back in the Hierarchy panel, click at ImageTarget. In the new Inspector Tab,

under the “Image Target Behaviour (Script)”, click at the Database dropdown and select

“[name] Database” instead of empty. Click at “molecule” and in the Hierarchy panel and find

the Transform section where you can modify the position, rotation, and scale of your 3D

model relative to the Image target. Change the size and position of the molecule so it is

proportional to the image. Do the same for the AR camera, so the small box showing the AR

camera’s view is acutally pointing at the molecule and image in the Scene window,

Finally in the Hierarchy panel, click and drag the “molecule” object exactly onto the

ImageTarget so it becomes a child of it. The AR app is now one step from being done. You can

choose between building the AR app on your mobile device (go to Step 6A) or the more

complicated way where it is shown how to publish the app in Goggle Play (go to Step 6B). As

mentioened the last step (Step 6B) requries an development account in Google Play and costs

25 $ (2019).

Journal 10/23/19 Page 11 of 18

200

205

210

THE STEP 6A: TRANSFERRING THE DATA TO THE MOBILE DEVICES

Step 6 explains how you install your

app on your mobile device. First you

have to allow installing the app on

your mobile device via a USB cable

connecting your device and the

computer. At your mobile device go

to “Settings” and find “Build

numbers”. Tap several times on

“build numbers” until a new

window appears saying “you are now a developer!”. Go back to Settings and find “Developer

option” on your mobile device. Scroll down and find and activate “USB debugging”.

Connect your mobile device via an USB cable to your computer. A new window pops up on

your device: check the box to allow debugging again. In Unity at your computer click:

File→BuildSettings→Android→Switch Platform and then “Build and Run”. Save the project

as an apk-file on your computer. The app is now built on your mobile device. If you have any

problems with Unity recognizing the device then it is must likely because your computer

needs a USB-adb-driver. This is not an issue for windows 10, but later versions should install

an universal USB-adb-driver. One example on a free universal USB-adb-driver can be found

here:

Journal 10/23/19 Page 12 of 18

215

220

225

230

https://software.intel.com/en-us/xdk/docs/installing-android-debug-bridge-adb-usb-driver-

on-windows

When it is completed you can remove the connection to the computer and open your new

app on your mobile device. Point the camera on your image target and enjoy the AR view.

STEP 6B: PUBLISHING THE APP IN GOOGLE PLAY

To publish an app to Google play, the

app must comply with the design

rules set by Google play. One of these

rules is that the app must be provided

in a package containing boths

package for the 32 bit and 64 bit

architecture. To include 64 bit

package navigate in Unity to the

Player settings by clicking

File→BuildSettings →Android→Player

Settings. In the Player setting expand

the “Other Options” section and under

configuration locate the scripting backed and change this from “mono” to “IL2CPP”. Now

activate the “Arm64” check box. When publishing an app to Google play, the app cannot be

Journal 10/23/19 Page 13 of 18

25

235

240

245

250

255

signed with an developer key. To change the signing of the package expand the “Publishing

Settings”, and click the “Keystore manager” button. If this is the first app you are creating

then press “Create new keystore” and fill in the required information. Close the Player

Settings and return to Build Settings and check the box “Build app bundle”. This generates

the aab-file which is needed to upload to Google play.

To publish an app to Google play a developer account is needed, this requires a one-time

payment of 25 $ (2019). Find the website by searching “Google play developer console”, pay

for the account and log in. A new app is created by clicking the “Create application” button

and filling in the name and default language, finish by clicking “Create” and the “Store

listing” menu opens. In this menu fill in the following information: 1. Short description, 2.

Full description, 3. Upload an image to create the icon for the app, this image must have a

resolution of 512x512. 4. Upload at least two screen shots. 5. Upload a “feature graphic” this

image must have a resolution of 1024x500. 6. Chose the application type from the dropdown

menu. 7. Choose the category from the dropdown menu. 8. Fill in the contact email for the

app author. 9. As this app have access to the camera on the device you are also required to

provide a link to a privacy statement describing how you handle personal informaiton. Finish

the Store listing section by clicking Save.

Open the “App releases” dialog via the left hand navigation menu, create a new app release by

clicking “manage” in the “Production track” section and press “Create release”. In this

windows carry out the following steps. 1. Choose to let Google manage and protect your app

signing key. Please note that this means you need the same keystore as used when signing

creating the intial .AAB to sign any updates. 2. Browse the .aab file you created in unity in

Journal 10/23/19 Page 14 of 18

260

265

270

275

“Android App Bundles and APKs to add” section. 3. Update the release notes for the app in

the “What's new in this release?” section. Complete this step by clicking Save.

The left hand navigation menu contains the two mandatory steps “Content rating” and “App

content” both of these are questionaires pertaining to the nature of the content, complete

these by filling in the required information.

Now navigate to the “Pricing & distribution” and perform the following steps. 1. Choose

“FREE” app. 2. Select the countries that you want the app to be avaliable in. 3. Verify that

you app complies with content guidelines and US export laws by checking the boxes in the

“Consent” section. Finalise by pressing “save draft”.

To publish the app navigate back to the App eleases section, press the “Edit release” button

in the “Production track” section. Finalise by clicking the review button. Google play will now

review that all the necesary information have been provide and the app complies with

registration. This can take up to seven days.

NOTE : PERFORMANCE IMPROVEMENTS

If very large structures or slower android devices are used performance of the app can be a

little sluggish. It is not withing the scope of this paper to describe the full set of rendering

optimizations one can do in Unity but the following two hints should get you a good portion of

the way.

Remove excess strutures:

When Jmol is used to generate the obj-file, an addition sphere is created for each atom, even

though these spheres is located at the same spot as the original atom sphere unity takes up

Journal 10/23/19 Page 15 of 18

280

285

290

295

300

30

processing power calculating positions of these. To remove these extra sphere open the unity

project again. In the Hierarchy panel expand the hierachy until the structure is expanded,

right click at the structure and select “Unpack prefab”. Now scroll down to the children

named “SprereXXX” where XXX is a integer, select them all and press “delete” this should not

affect the 3D structure of the molecule but signigicantly increase the performance.

CONCLUSION

We have described how to make a simple augmented reality app to visualize any 3D chemical

model using free softwares. The method works for all molecules including small molecules,

supramolecular structures, MOFs and biomacromolecules.

AUTHOR INFORMATION

Corresponding Author*E-mail: pittel@chem.ku.dk

ACKNOWLEDGMENTS

We acknowledge support from the Danish Council for Independent Research (DFF 4181-

00206) and from the University of Copenhagen.

Journal 10/23/19 Page 16 of 18

305

310

315

320

REFERENCES

1. Maier, P.; Klinker, G. Augmented Chemical Reactions: 3D Interaction Methods for

Chemistry. iJOE. 2013, 9 (S8), 80−82. b) Yang, S.; Mei, B.; Yue, X. Mobile Augmented

Reality Assisted Chemical Education: Insights from Elements 4D. J. Chem. Educ.

2018, 1060-1062. c) Wolle, P.; Müller, M. P.; Rauh, D. Augmented reality in Scientific

Publications – Taking the Visulization of 3D Structures to the Next Level. ACS Chem.

Biol., 2018, 13, 496-499.

2. Chen, T.-H.; Lee, S.; Flood, A. H.; Miljanic, O. S., How to print a crystal structure

model in 3D, CrystEngComm 2014, 16, 5488–5493. B) Scalfani, V. F.; Vaid, T. P. 3D

Printed Molecules and Extended Solid Models for Teaching Symmetry and Point

Groups. J. Chem. Educ., 2014, 91, 1174-1180.

3. Chichak, K. S.; Cantrill, S. J.; Pease, A. R.; Chiu, S-H.; Cave, G. W. V.; Atwood, J. L.;

Stoddart, J. F. Molecular Borromean Rings. Science. 2004, 304, 1308-1312.

4. Cremers, J.; Haver, R.; Rickhaus, M.; Gong, J. Q.; Favereau, L.; Peeks, M. D.; Claridge,

T. D. W.; Herz, L. M.; Anderson, H. L. Template-Directed Synthesis of a Conjugated

Zinc Porphyrin Nanoball. J. Am. Chem. Soc. 2018, 140, 5352-5355.

Journal 10/23/19 Page 17 of 18

325

330

335

340

5. Lisbjerg, M.; Jessen, B. M.; Rasmussen, B.; Nielsen, B. E.; Madsen, A. Oe.; Pittelkow,

M. Discovery of a cyclic 6+6 hexamer of D-Biotin and formaldehyde. Chem. Sci. 2014,

5, 2647-2650.

6. Holton, R. A.; Somoza, C.; Kim, H.-B.; Liang, F.; Biediger, R. J.; Boatman, P. D.;

Shindo, M.; Smith, C. C.; Kim, S.; Nadizadeh, H.; Suzuki, Y.; Tao, C.; Vu, P.; Tang, S.;

Zhang, P.; Murthi, K. K.; Gentile, L. N.; Liu, J. H. First Total Synthesis of Taxol. 1.

Functionalization of the B Ring. J.Am. Chem. Soc., 1994, 116, 1597-1598.

Journal 10/23/19 Page 18 of 18

35

345

download fileview on ChemRxivAR-pittelkow.docx (7.95 MiB)