Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA

Copyright 2009 Trend Micro Inc.Classification 04/21/23 1

Troubleshooting TMSP

Marks Shen • Senior Engineer – QAEvan Wang • Engineer - QA


Agenda |

Frequent Case

Debug log and information

Troubleshooting

Q&A

Copyright 2009 Trend Micro Inc.

No report was generated

• Logs need to collect on Daemon Server– /opt/TrendMicro/tdss/tdes/log/iae_log.txt– /var/log/cron

• Information – Customer expiration date– Device register to TMSP


No report was generated Cont.

• Normal debug log for log correlation and report generation

• Crontab task

cron_iae.sh will be executed at 2:15 am every day


No report was generated Cont.

Report will not be generated if

• Customer service get expired

• Customer without device registered


No incident in report• Logs need to collect on Daemon Server

– /opt/TrendMicro/tdss/tdes/log/iae_log.txt

• Information– Check if TDA log has uploaded

Latest log time


Cannot access Admin console (err 404)

• Logs need to collect on Daemon Server– /var/log/httpd/access_log– /var/log/httpd/error_log

• Information – ps –ef | grep httpd– Netstat –anp | grep httpd


No Rsync log uploaded

• Logs need to collect on Access Server– Log receiver

• /home/tdalog/log/pre-post-exec.log /home/tdalog/log/db_import_tda.log

• /home/tdalog/log/db_import_tdm.log

– Authentication (describe in next sides)

• Information – Ps –ef | grep tmsshd– Netstat –anp | grep tmsshd

• Listen on port 22


No Rsync log uploaded Cont.

• Normal procedure debug log of TDA log processing– /home/tdalog/log/pre-post-exec.log


CAS server caseProblems caused by CAS failure:

– Device register to TMSP fail– Customer portal login fail– Log uploading fail through RSYNC

Logs need to collect on Access Server:– /var/log/messages– /var/log/cas_8000.log– /var/log/cas_8001.log– /var/log/cas_8002.log

Information:• Ps –ef | grep pound• Ps –ef | grep rubcasd

Normal log of CAS authentication– /var/log/cas_8000.log


CAS server case – Service down

• TDA register fail

• Check /var/log/messages

• Recover– If pound or cas service is down

• /etc/init.d/pound start• /etc/init.d/rubcasd start


Data Gateway Case

Problems caused by Data Gateway failure:– OCS Heartbeat / OCS log real-time cannot upload– T2 / T3 mitigation request cannot deliver to TMTM– SIC sample cannot upload from TMTM

Logs need to collect on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB-

INF/logs/dg.log– $APACHE_HOME/logs/ssl_request_log_dg– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out

Information:• Ps –ef | grep httpd• Ps –ef | grep tomcat• Netstat –anp | grep 443• Netstat –anp | grep 8009


Data Gateway Case – DB disconnect

• TDA register fail

• Check apache error log: /usr/apache/logs/error_log


Customer portal cannot login

• Logs need to collect on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/

tms2/WEB-INF/logs/tms.log– $APACHE_HOME/logs/ssl_request_log_portal– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/

catalina.out

• Information• Ps –ef | grep httpd• Ps –ef | grep tomcat• Netstat –anp | grep 443• Netstat –anp | grep 8009


Cannot get eMail notification

• Exclude Mail server problem, collect debug logs:– /root/infomation.log

– /var/log/cron

No Subscription

DB connection fail



FAQ

Q: How to re-generate report manually?Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/";

– 1. Daily Report#php gendailydata.php user_id YEAR MONTH DAYexample: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“

– 2. Executive Report (Weekly / Monthly)#php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weeklyexample: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01

– 3. Upsell Report# php genupselldata.php user_id START_DATE END_DATE noexample: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31Note: Upsell report will not be imported into DB and cannot download from admin console


FAQ

Q: What the difference between two service type “TDS” and “TLMS”?

A: “TDS” service is set to the customer, who has TDA deployed. “TLMS” service is set to the customer, who has both TDA and TMTM deployed.

TMSP provide different report for these 2 service type, and for TLMS, TMSP provide additional UI for abnormal endpoints handling process.


FAQ

Q: Why AE status show N/A on Admin console?

A: AE (Abnormal Endpoint) status is designed for the customers, who has TMTM deployed, to show the mitigation failed client on TMSP UI.

That means, only the customer with service type “TLMS” will show the AE details in TMSP admin UI as link.


FAQ

Q: Whether the IAE will be updated?

A: Yes, TMSP IAE rules will be updated timely to detect new threats. Currently, the rule is updated monthly. In TMSP side, we get new threat rule from threat team and deploy to each site after verification.


FAQ

Q: Can TMSP generate DOC format report?

A: No, currently, TMSP can only generate report in PDF format.


FAQ

Q: How to backup Database?

A: Here are two type of DB backup, Full Backup and Lightweight Backup.

• Full backup:Backup full database of TMSP

#mysqldump -uroot -p"$password" tdors_new > tdors_new_backup.sql


FAQ

• Lightweight DB backup: Backup customer info, configuration and some correlation result.

#mysqldump -h $host -u$account -p"$password" tdors_new access_code admin_log admins contacts customer customer_expiration_conf customer_expiration_notif notification ocs_ast_event ocs_malware scsd_case scsd_case_followup scsd_case_status scsd_pattern scsd_pattern_status scsd_sample_info scsd_sample_summary t2t3_dce_result tb_devprofile> SJDC2.5.sql

After backup entire database, it is recommended to store the backup file into another physical storage.


Q&A


THANK YOU!

Documents

Copyright 2009 Trend Micro Inc. Classification 9/19/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA