Copyright 2009 Trend Micro Inc. Classification 9/23/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA

Copyright 2009 Trend Micro Inc.Classification 04/19/23 1

Troubleshooting TMSP

Marks Shen • Senior Engineer – QAEvan Wang • Engineer - QA


Agenda

Frequent Case

Debug log and information

Troubleshooting

Q&A

Copyright 2009 Trend Micro Inc.

No report was generated

• Logs need to collect on Daemon Server– /opt/TrendMicro/tdss/tdes/log/iae_log.txt– /opt/TrendMicro/tdss/tdes/reports/tdes.log(Since

2.5R3)– /var/log/cron

• Information – Customer expiration date– Device register to TMSP


No report was generated Cont.

• Normal debug log for log correlation and report generation

• Crontab task

cron_iae.sh will be executed at 2:15 am every day


No report was generated Cont.

Report will not be generated if

• Customer service get expired

• Customer without device registered


No incident in report• Logs need to collect on Daemon Server

– /opt/TrendMicro/tdss/tdes/log/iae_log.txt

• Information– Check if TDA log has been uploaded

Latest log time


Cannot access Admin console (err 404)

• Logs that need to collect on Daemon Server– /var/log/httpd/access_log– /var/log/httpd/error_log

• Information – ps –ef | grep httpd– netstat –anp | grep httpd


No Rsync log uploaded

• Logs need to collect on Access Server– Log receiver

• /var/log/messages• /home/tdalog/log/pre-post-exec.log • /home/tdalog/log/db_import_tda.log• /home/tdalog/log/db_import_tdm.log

– Authentication (describe in next sides)

• Information – ps –ef | grep tmsshd– netstat –anp | grep tmsshd

• Listen on port 22


No Rsync log uploaded Cont.

• normal log of tmsshd and rsync

Classification 04/19/23 9


No Rsync log uploaded Cont.

• Normal debug log of TDA log processing– /home/tdalog/log/pre-post-exec.log


CAS server caseProblems caused by CAS failure:

– Device register to TMSP fail– Customer portal login fail (only before R3)– Log uploading fail through RSYNC

Log on Access Server:– /var/log/messages– /var/log/cas_8000.log– /var/log/cas_8001.log– /var/log/cas_8002.log

Information:• ps –ef | grep pound• ps –ef | grep rubcasd

Normal log of CAS authentication– /var/log/cas_8000.log


CAS server case – Service down

• TDA register fail

• Check /var/log/messages

• Recover– If pound or cas service is down

• /etc/init.d/pound start• /etc/init.d/rubcasd start


Data Gateway Case

Problems caused by Data Gateway failure:– OCS Heartbeat / OCS log cannot be handled– T2 / T3 mitigation request cannot be delivered to TMTM– SIC sample cannot be handled

Logs on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/dg/WEB-

INF/logs/dg.log– $APACHE_HOME/logs/ssl_request_log_dg– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/catalina.out

Information:• ps –ef | grep httpd• ps –ef | grep tomcat• netstat –anp | grep 443• netstat –anp | grep 8009• netstat –anp | grep 8080


Data Gateway Case – DB disconnect

• TDA register fail

• Check apache error log: /usr/apache/logs/error_log


Customer portal cannot login

• Logs need to collect on Access Server:– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/webapps/

tms2/WEB-INF/logs/tms.log– $APACHE_HOME/logs/ssl_request_log_portal– $APACHE_HOME/logs/error_log– /opt/TrendMicro/dg/apache-tomcat-6.0.18-1/logs/

catalina.out

• Information• ps –ef | grep httpd• ps –ef | grep tomcat• netstat –anp | grep 443• netstat –anp | grep 8009• netstat –anp | grep 8080


Cannot get eMail notification

• Exclude Mail server problem, collect debug logs:– /root/infomation.log

– /var/log/cron

No Subscription

DB connection fail


23/4/19 17Classification

FAQWhy no daily report can be found from web UI?

1. Check TDES log from #tail -n 100 /opt/TrendMicro/tdss/tdes/log/iae_log.txt

If content like "Daily report: customer_ID, 2009 02 16 JP" cannot be found, that mean the scheduled job has not started so far.

2. Daily report is auto generated at 7:15 am every day, so check the system time of TDES:

3. Report generation need take some time, please check if the report is generating:#ps -ef | grep php

if some php process is running, it means the reports are under generating.4. Check if customer has expired for TMSP service, find the profile “expire time”

from admin console


FAQ

Why there is no data in report?1. Check if the log has been uploaded to TMS and imported

into DatabaseLogin log receiver machine and check the file last modification time#ll /home/tdalog/userdata/USERID/DEVICE_GUID/*.db

If not latest data, that means TDA did not upload logs.2. Log in database and query yesterday's log

Some times, TDA did not detect any events, if so, there will be no yesterday's data in DB

3. Check iae_log.txt, check if “Running TDES 2.1 for XXXX (device=50)(customer_id=30) on date: 2009 2 16” existing, this means TMS run IAE for this customer. if NO this content exist, means there is something wrong when process IAE.


FAQ

How to re-generate report manually?Login TDES machine, change dir to "/opt/TrendMicro/tdss/tdes/";

– 1. Daily Report#php gendailydata.php user_id YEAR MONTH DAYexample: "php gendailydata.php trend 2009 01 04" generate daily report of 2009.01.04 for customer "trend“

– 2. Executive Report (Weekly / Monthly)#php genexecdata.php user_id START_DATE START_DATE yes m/w START_DATE: report start data with format “YYYY-MM-DD” START_DATE: report end data with format “YYYY-MM-DD” yes: if this report will be imported into DB m/w: monthly or weeklyexample: "php genexecdata.php trend 2009-01-01 2009-01-31 yes m" to generate monthly report for customer "trend" of 2009-01

– 3. Upsell Report# php genupselldata.php user_id START_DATE END_DATE noexample: " php genupselldata.php trend 2009-01-01 2009-01-31 no" to generate upsell report for customer "trend" from 2009-01-01 to 2009-01-31Note: Upsell report will not be imported into DB and cannot download from admin console

After execute these command, reports will be re-generated and imported into database for downloading


FAQ

What’s the steps to deploy new report php file?Sometimes, reports generation related php need to be modified and deploy to TDES, here are the steps to do this:1. back up old php file2. replace with new php file3. remove cache_*.php under TDES installation folder4. Re-generate report to verify new php files if necessary


Q&A


THANK YOU!

Documents

Copyright 2009 Trend Micro Inc. Classification 9/23/2015 1 Troubleshooting TMSP Marks Shen Senior Engineer – QA Evan Wang Engineer - QA