2nd Round Tech Sharing For IWSA Barry Yuan Aug 18 th, 2006

2nd Round Tech Sharing For IWSA

Barry Yuan

Aug 18th, 2006

Copyright 2005 - Trend Micro Inc.

Agenda

• IWSA technical issues summary;• Suggestions on the issues and workarounds/tools

sharing;• Basic troubleshooting skills sharing;• Issues will be addressed by SP1:

– Fully transparency;

– Vlan Tagging;

– Link loss, carry forward

• SP1 beta;• SWAT best practice.


IWSA technical issues summary

• IWSA Issues Summary

• 1. Insufficient hard disk space.• 2. Memory Low shown in LCD Module.• 3. CPU utilization not balanced, one CPU 100% while others are 0%• 4. No Concurrent Connections numbers are shown in the verbose logs.• 5. Does not support OpenLDAP well.• 6. LDAP connection issue.• 7. LDAP configuration not easy.• 8. Customer need to input domain before user name• 9. Very slow report generation.• 10. Problems working with NetCache.• 11. Problems working with BlueCoat.• 12. Fully transparency in bridge mode.• 13. Vlan Tagging Support• 14. External database & its HA solution.• 15. ICAP, REQMOD and RESPMOD, what is the difference, how to pick the best

mod for each user environment.• 16. Not able to determine the disk usage of each table in the database.


硬盘问题—扫描方式、硬盘接线– In some customer environment the Hard disk space is not enough to accommodate

data for IWSA for more than one month, some even shorter.– Service will crash when disk full. There is no alert before disk full– Even after rest to factory default, IWSA can only run a period of time like 2-3 weeks.

• Suggestions:• As a workaround we can:

• In Web GUI: Change “Number of days to store logs in database” from 30 to a smaller number like 7. (By doing this we can only generate weekly reports)

• Uncheck “Log HTTP/FTP access events” if it is checked.• Disable all verbose logging by changing “verbose” to 0 in /etc/iscan/intscan.ini.

• Suggestions to PM and PDG: • I believe it would be better if we can compress logs in database to save space to enable

Monthly report generated in customer environment. Maybe even can generate report for last month or even last a few months.

• It would be even better if we can delete the logs every week, only use these data to generate daily/weekly report (maybe store this report in a table in database), and generate monthly report use the data from daily/weekly reports (instead of from the logs). This will enable us to store much more data (which is much smaller than logs) for reports (maybe even can do yearly reports) while saving a large part of disk space.

– Alternative besides workaround:• In cases listed below:

• Some big customers may need the database to be stable and HA;• Or when IWSAs are deployed in Server Farm;

• We can use external and centralized database for IWSA, this database has to be postgreSQL database, and we can use third party software to achieve HA for the database.


Memory Low shown in LCD Module.

– When you use “free” command in Linux, you will get the free memory size and free SWAP space. As what I have observed that the Free MEM displayed on the LCD is the size of free memory we get in the “free” command.

– Linux is not like Windows, it seldom cleans Memory even when it find memory is not in use. It only cleans memory when it needs memory allocated for a new process. So it is normal in Linux when you see over 90% of memory is in use.

– The more accurate sign of resource low is when both Memory and SWAP space are low. We can use the command “free” to see the current free memory and free SWAP space.

– We have created a script to query disk usage, free memory size and free SWAP space and put into a file for analysis.

• File is located in: FTP://test:[email protected]/test.sh• Please use binary mode to download this file, and run: chmod 744 test.sh

on this file. Use command: ./test.sh >test.txt to export the logs to test.txt and send the file to us. You can copy the file to /etc/iscan/UserDumps/ and get the file in Web GUI (Administration->Support).


Script output sampleDate/Time/Uptime----------------Wed Aug 16 09:15:05 WIT 2006 09:15:05 up 5 days, 23:24, load average: 0.02, 0.01, 0.00

IWSA Process Count------------------ 78

Files in Temp Directory------------------- 1

CLOSE_WAITs----------- 1

Disk Space----------Filesystem 1k-blocks Used Available Use% Mounted on/dev/hda2 94695 17892 71914 20% //dev/hda4 53544 4176 46604 8% /etc/conf/dev/md0 988088 214536 723360 23% /usr/iwss/dev/md3 31546652 4597304 25346872 15% /var/dev/md2 5913608 24 5613184 0% /var/tmp

Free MEM------ total used free shared buffers Mem: 2076708 1939916 136792 0 48504 Swap: 4016232 1248 4014984Total: 6092940 1941164 4151776


Memory Low shown in LCD Module (cont)

The threasholds are defined in /var/iwss/hardwarechecklist.ini:

/var/iwss # more hardwarechecklist.ini [checklist]# The threshold settings of components # Unit: percentage# when cpu/disk space/memory usage reach the settings, the signal will be sent# to LED and the short message will be sent to LCD disk_usage=95cpu_usage=95mem_usage=95swap_usage=95

[log]# more information is logged to /etc/iscan/log/syschecking.XXXX if debug is set

as "on",# value: on/offdebug=off


Memory Low shown in LCD Module (cont)

• To get the extended information about the Hardware Monitor measurements and decisions, enable debugging by setting the [log]/debug configuration parameter to “on” and re-starting the daemon.

Check the log file for the extended log entries:2006/01/30 17:02:54 GMT+01:00 3--------------------------------2006/01/30 17:02:54 GMT+01:00 Check CPU....2006/01/30 17:02:54 GMT+01:00 cpu[1]: 0.56 %,...2006/01/30 17:03:54 GMT+01:00 Check Memory.....2006/01/30 17:03:54 GMT+01:00 Memory usage: 57.312006/01/30 17:03:54 GMT+01:00 Swap usage: 0.002006/01/30 17:03:54 GMT+01:00 Signal LED and LCD .....2006/01/30 17:03:54 GMT+01:00 Memory is low.2006/01/30 17:05:14 GMT+01:00 Sleep 2 minutes


CPU utilization not balanced

• CPU utilization not balanced, one CPU 100% while others are 0%– IWSA has Duel CPUs with Hyper-threading enabled. Load

balance between CPUs should be hardware based.

– It could be hardware issue if only one CPU is working while others are idle.

– Just for reference, the CPU usage data are from /proc/stat file.


Concurrent Connections numbers

• No Concurrent Connections numbers are shown in the verbose logs. – In IWSS 2.5 a concurrent connection number is indicated in

the verbose log when there is a new connection coming or closed. This is very helpful since concurrent connection number is an key indicator of the work load of IWSA and to for us to evaluate the performance.

– Suggestion: We can add concurrent connection number into either verbose log or into performance log.


Does not support OpenLDAP well.

• We support DC attribute only, if customer defines their own attribute, IWSA is not able to support it.– We found this problem when IWSA is working as ICAP

server, PDG has released hotfix build 1147 to work around this issue, by using NetCache to get LDAP info.

– But the problem is still there with IWSA. I hope this problem could be completely solved when there is sufficient time.


LDAP connection issue

– It is reported that sometimes LDAP connection is not so stable. Sometimes it may cause service crash.

– The Error "Out of LDAP connection in pool, waiting for connection to be returned" happens when the LDAP server is not able to handle all requests from IWSA.

– If it is because LDAP connection not enough, Here's my suggestion on how to try address this issue:1. Check and change settings in intscan.ini also: a. Find in section [http] the value of num_threads; b. Find in section [LDAP-Setting] and check the value of "ConnectionPoolNumber", it is recommended that we change this value to be at least bigger than num_threads; c. Restart HTTP main service.

If this does not work, please do item 2 under customer's permission.

2. If our customer is using Windows AD, I would like to advise our customer tweak their LDAP Server, please see instructions from Microsoft below:http://support.microsoft.com/default.aspx?scid=kb;en-us;315071


LDAP configuration not easy

• It is not easy to configure LDAP settings, especially when using OpenLDAP or Sun Directory, it is troublesome to input user names, and there is no instruction available in help.


Customer need to input domain before user name

• (like: Trend\Barry_Yuan), when they are prompt to input user name.

• In some companies users are not easy to get used to this. They would like to input username only without domain info.

• We should be able to support both with and without Domain.


Very slow report generation

• Generating reports may need lots of database query and log analysis,

• If IWSA is preoccupied with massive traffic load, greater delay of Real-Time report will take place.

• If this is the case we would like to suggest using scheduled report instead, and change the report generation time to a low traffic time. (default 1am)


Problems working with NetCache

• “Page not displayed” error due to Timeout issue:• NetCache device is opening the ICAP connection to IWSA

many minutes before it actually sends the request. IWSA times out this connection, but NetCache is ignoring the FIN packet from IWSA.

• We can avoid this by increasing the timeout duration IWSA uses from 30 seconds to a larger value (such as 600 seconds) by modifying intscan.ini->[main]/timeout, and restarting S99ISproxy.

• I will advise this setting to be set in SP1


Problems working with BlueCoat

• Large file handling is not supported. None of "Scan before delivering", "Deferred scan", "Scan behind" will work with bluecoat.

• That is because all large-file handling options require the same ICAP trickling header from the ICAP client, but BlueCoat doesn't support such header; instead they come with their own trickle feature to handle large files, it's called Patience Page.

• Workaround: Do not use ICAP, deploy IWSA as upstream proxy to BlueCoat. And the large file handling will still work.


External database & its HA solution

• PostgreSQL database is needed for external database; currently we do not have HA solution for database.

• I was working with Jack Kuo (TMM) on the HA solution but later on since there was not update from the deal, we did not run test on the solution we found.


ICAP, REQMOD and RESPMOD

• ICAP, REQMOD and RESPMOD, what is the difference, how to pick the best mod for each user environment.– REQMOD: In the Request Modification Mode, the ICAP client

sends the request information to IWSA before retrieving the content. This allows IWSA to perform the following types of content management:

URL blockingURL FilteringContent Scanning for uploaded files

– RESPMOD: In the Response Modification Mode the ICAP client sends the retrieved content to IWSA before returning it to the Web Client. This mode allows management of the retrieved content.

Scanning of downloaded files / contentURL-Blocking & Filtering – for already retrieved content!


Disk Usage

• Not able to determine the disk usage of each table in the database.– Use admin_checksize.sh (/usr/iwss/bin)to check the size of

database and each table, in MB.

– Use df –k to check the disk space available.


IWSA FAQ

• These issues, once have been answered or addressed, will be put into IWSA FAQ.

• Please refer to the IWSA FAQ for other frequently asked questions.

• The file is located in:

ftp://iwsa:[email protected]/IWSA/IWSA FAQ.doc

Basic Troubleshooting


Access IWSA using SSH

• Access IWSA Using SSH:– Login to IWSA console, – default username: root– default password: iwsa– Press Enter to start;– Press 1, to enter System Configuration;– Press 8, to Set “remoteadmin” Password;– Enter the new password, note the new password should at least contain

5 characters.– Check the IP setting of IWSA.– Find a machine, make sure it can access IWSA using its IP, and has

SSH client software installed, like SecureCRT, or PuTTY (http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html).

– Now you can use the client to access IWSA port 22 which is SSH, the account name is “remoteadmin” and the password is the one you set earlier.


Transfer files to/ from IWSA

• Customer can use FTP to transfer files to/from IWSA. IWSA is using NcFTP and the command can be used in the Shell interface:– The parameters:

• -u XX Use username XX instead of anonymous.• -p XX Use password XX with the username.• -P XX Use port number XX instead of the default FTP

service port (21).• -j XX Use account XX with the username (rarely needed).• -F Dump a sample $HOME/.ncftp/firewall prefs file to

stdout and exit.– ftp www.corelab.cn –u iwsa –p password


Procedures to enable verbose log

• Procedures to enable verbose log and the easiest way to get the log from IWSA:• Login to IWSA console,

– default username: root– default password: iwsa

• Press Enter to start;• Press 2, to enter Utilities• Press 1, to start Shell Interface• Enter y for yes• Now we are in Linux Shell, please follow the steps below to enable verbose:

• Input “cd /etc/iscan”• Input “vi intscan.ini”• Input “/verbose” and press enter, and then press “N” for next;• You will find:

• [http]• #Switch for debug log.• # 1 -> turn on • # 0 -> turn off • verbose=0

• Move cursor to the “0” after verbose, press “R” and input “1”.


Procedures to enable verbose log (cont)

Now the configuration was changed to:• [http]• #Switch for debug log.• # 1 -> turn on • # 0 -> turn off • verbose=1

• Press “Esc” and then input “wq”, press enter.• The configuration file is saved.

• Input: “./S99ISproxy reload” to restart IWSA HTTP service. Output should be:-----------------------------------------------------------------------------------------------------------Please wait while the IWSS daemon is being checked...okRestarting the InterScan HTTP daemon...Please wait while the IWSS daemon is being checked...ok-----------------------------------------------------------------------------------------------------------

• Go to the log folder by input: “cd log”, and use “ls” to display all file.• Replicate the issue.• When the issue is replicated, get the file named “http.log.yyyymmdd.000x” for instance

if today is April 12, 2006, the file name should be “http.log.20060412.000x” and the “x” means any, usually it is “1”, there might be multiple files.

• Copy the said files to UserDumps folder: “cp http.log.yyyymmdd.000x ../UserDumps”• Open Web GUI of IWSA by entering “http://IP_OF_IWSA:1812” in a client machine,

login (default password “adminIWSS85”), go to “Administration->Support” find the above file, click it, and press “Download to your computer”. Compress the files and send them to us.


Get TCP Dump

• Login to IWSA console, – default username: root– default password: iwsa

• Press Enter to start;• Press 2, to enter Utilities• Press 1, to start Shell Interface• Enter y for yes• Now we are in Linux Shell, please follow the steps below to do packet

capture:– Enter folder: /etc/iscan/UserDumps– Input “tcpdump –s0 -w tcpdump.cap”

• Replicate the issue;• Press “Ctrl+c” to finalize the packet capture;• Now the file is generated, you can easily get it from the IWSA Web GUI:

– Go to the web GUI, login and find Administration->Support;– Find the file tcpdump.cap in Step1, click on it;

• Press “Download to your computer” and download the file and submit to us.


Customer can use the script below:

• All configuration files and logs will be backup into a archive file and will be automatically put into \etc\iscan\UserDump\ folder, customer can easily download this archive from Web GUI.

• (Administration->Support->ini_logs_package_yyyymmdd.gz, click “Download to your computer”).


Testing IWSA Software

At the command prompt, type “/etc/iwsaSmokeTest.sh” and press <Enter>.This will launch a script which will do a high level test of the applications on the Hard Drive image.



The test will run for 1-3 minutes.

If successful, you will see a screen like the one below.



Failure of the smoke test indicates a software error.Possible causes include:

Incorrect configurationSoftware defectsFile corruptionComplete hard disk failure

Before continuing with software troubleshooting, verify that the hard

disks are mounted and functioning without errors.

Hard Disk troubleshooting tools that can be used are:

mount, mdadm, fsck, fdisk, setup_sata.sh, ls


Verifying Hard Disk operation

First check if the partitions are mounted using the mount command as shown below.

/ # mount/dev/hda2 on / type ext2 (ro)none on /proc type proc (rw,nodiratime)none on /sys type sysfs (rw)devpts on /dev/pts type devpts (rw)/dev/hda4 on /etc/conf type ext3 (rw)/dev/md0 on /usr/iwss type ext2 (rw)/dev/md3 on /var type ext3 (rw)/dev/md2 on /var/tmp type ext2 (rw)

If the above mount points are present, use the ls command to verify that files are present in the partitions. A complete file list can be found at the end of this presentation.# ls /var# ls /usr/iwss



If all mount points are correct and files are present in each partition,

the next step is to check the /var/iwss/log/syschecking.log for errors.

Display the log page by page using the following command.

#more /var/iwss/log/syschecking.log

Search for keywords “error” or “fatal” using the following command

#more /var/iwss/log/syschecking.log |grep error



If errors are found in the syschecking.log, the hard disk may have some problems. Use the mdadm tool to check the integrity of the RAID.

/ # mdadm --detail /dev/md0/dev/md0:Version : 00.90.02Creation Time : Tue Nov 15 16:05:01 2005Raid Level : raid1Array Size : 1003904 (980.54 MB 1028.00 MB)Device Size : 1003904 (980.54 MB 1028.00 MB)Raid Devices : 2Total Devices : 2Preferred Minor : 0Persistence : Superblock is persistentUpdate Time : Mon Jan 9 10:34:46 2006State : cleanActive Devices : 2Working Devices : 2Failed Devices : 0Spare Devices : 0


Repairing failed RAID devices

If the RAID device shows failed devices, it may be possible to repair the mirror unless the hard disk has lost power or completely failed.

Break RAID sets:/ # mdadm –break md0 md2 md3

Check physical disks for errors:/ # fsck –r /dev/sda /dev/sdb

Recreate RAID sets:/ # mdadm –-create md0 /dev/sda1 /dev/sdb1 / # mdadm –-create md1 /dev/sda2 /dev/sdb1/ # mdadm –-create md2 /dev/sda5 /dev/sdb5/ # mdadm –-create md3 /dev/sda6 /dev/sdb6

Remount partitions:/ # mount -a


Initializing Hard disks and recreating the RAID

If the disks have I/O errors which cannot be repaired using fsck, it may be possible to repair the disks by re-initializing them.

To initialize the disks use the following procedure:/ # /etc/setup_sata.sh to reinitialize disks and recreate the Mirror set.

The script will reformat the disks and recreate the RAID devices. Once completed, you will need to reinstall the IWSA software


Reinstalling IWSA software

Copy the latest build of the IWSS software to the device using FTP:/ # mkdir /var/installers/ # cd /var/installers/ # ftp –u <user> -p <passwd> <ftpserver>/ # bin/ # hash/ # mget <iwsa installer files>/ # bye

Reinstall the IWSA Software./ # tar –xvf <iwsa files> (extract files from tar file)

/ # mount –o remount /dev/hda2 / (remount the root file system with RW permissions)

/ # chmod 744 /var/installers/install_appliance.sh (assign execute permissions to the install script)

/ # ./install_appliance.sh (Execute the install script)


Recovering System Image boot Failure

Reset the device and enter Rescue mode when prompted.Use the IWSARESCUE.EXE or tftp to upload a known good System Image (.R file) to the device’s DOM.

The IWSA solutions CD includes a copy of the .R file, or you can download a newer version from the Trend Micro update center.


IWSA Software Rollback

IWSA keeps a copy of its original installation in a backup partition which can be restored at any time from the Pre-configuration menu.

Login to the pre-configuration menu using the console cable:Enter <3> (Rescue System) Enter <2> (Restore IWSA to factory default settings)Enter “y” when prompted to rescue the device. *device will reboot automatically

After the device reboots, login into the pre-configuration menuAccess the system shell by entering <2> (Utilities) and <1> (Start Shell Interface) enter “y” when prompted.

Verify that IWSS processes are running: ps –ef | grep –I iwss


IWSA Checklist

• Hardware vs Software• GUI• Performance• Crash & Exception• Virus & Policy Detection• Update• Database• Reporting• General Issues

• ftp://iwsa:[email protected]/iwsa/IWSA Checklist.xls


Troubleshooting IWSA Hardware

• Troubleshooting IWSA Hardware• IWSA Hardware Checklist.pdf

• ftp://iwsa:[email protected]/iwsa/IWSA Hardware Checklist.pdf


Level 3 Document

• ftp://iwsa:[email protected]/iwsa/IWSA_2500_L3SH_02Mar2006_A4.pdf


Process, SWAT Website:

• http://swat.trendmicro.com• http://nj-core-web:8080/SWAT/

http://swat.trendmicro.com/

http://nj-core-web:8080/SWAT/


New Deal Registration

• Here’s an procedure on how a deal opportunity is registered and processed in our SWAT website: (http://swat.trendmicro.com )

• If it’s an New Product Launch deal, which means a pre-sale consultant task from APAC Sales or TAMs on IWSA, IMSA, NVWE, IGSA or other newly launched products, it’s going to be registered to our Opportunity Registration Bank – New Product Launch.

• If it’s an SyMac Attack Project, our company already has a system to track those deals (http://macattack.us.trendnet.org/MacAttack/Opportunity/opportunityList.aspx)

• There is another entry for VLE escalations, if the customer is bigger than 2,500 seats they will be able to grant SWAT assistance with technical issues.

http://swat.trendmicro.com/

http://swat.trendmicro.com/Lists/Opportunity%20Registration%20Bank%20%20New%20Product%20Launch/AllItems.aspx

http://macattack.us.trendnet.org/MacAttack/Opportunity/opportunityList.aspx

http://macattack.us.trendnet.org/MacAttack/Opportunity/opportunityList.aspx


New Deal Registration

• When a deal is registered, SWAT Project Manager will evaluate the deal and then contact SE or TAMs;

• The task will be assigned to an engineer.• We might need to allocate proper time and resources to

analyze/replicate/troubleshoot this issue/query.• WebEx/Con-call/On-site will probably be needed to try

solve this issue.• When the issue is addressed, and there’s no more

issues/queries from the SEs and TAMs, the opportunity entry can then to closed.


Coordination between SWAT, TSS and other groups

• TSS– Pre-sale issues– Performance test, sizing guidelines

• AV– Detection rate & clean rate issues

• APS– Third party products, such as Cisco, CrossBeam…

• Product Manager, JM and PDG– Request for HotFix– Official statements


Suggesitons/Concerns/Queries

• Thank you for your support!

Documents

2nd Round Tech Sharing For IWSA Barry Yuan Aug 18 th, 2006