スタディサプリを支えるデータ分析基盤～設計の勘所と利活用事例～

(C) Recruit Marketing Partners Co.,Ltd. All rights reserved.





























• ( https://ring.education/ ) -

- https://ring.education/research/2016_hs_al_experiment_report_02/

https://ring.education/

https://ring.education/research/2016_hs_al_experiment_report_02/












id grade_id gender school_id

AAA 001 female 0000X

BBB 002 male 0000Y

CCC 005 male 0000Z

id label

001

002

003

id label

0000X X

0000Y Y

0000Z Z

id grade gender school

AAA female X

BBB male Y

CCC male Z








Rails Kinesis

Lambda

TD SDK web JS app native

web Single Page Application



/(^o^)\






embulk-input-mongodb by @hakobera

embulk-input-postgresql

embulk-parser-json

embulk-filter-expand_json

embulk-filter-hash by @kamatama41

embulk-filter-mask by @beniyama

embulk-output-td





Embulk Hive / Presto



class DailyVideoRatingReport(luigi_td.Query): type = 'hive' database = LuigiConfig().main_db report_database = LuigiConfig().report_db source = ‘queries/hive/daily/history/daily_video_rating_report.sql’ jst_date = luigi.DateParameter(default=DateTimeUtil.get_jst_date())

def requires(self): return [ DailyUserMasterCreate(self.jst_date), DailyContentsMasterCreate(self.jst_date) ]


def run(self): TdUtil.drop_partition(

table=‘video_rating_report’, to_jst_date=self.jst_date )

super(DailyVideoRatingReport, self).run()

def output(self): return luigi_td.ResultTarget( '{0}/DailyVideoRatingReport-{1}.txt'.format( LuigiConfig().output_path, self.jst_date.strftime('%Y-%m-%d') ), 'td://@/reports/video_rating_report?mode=append' )



+video_rating_report:

+partial_delete: td_partial_delete>: video_rating_report database: reports from: ${moment(session_time).utc().add("days", -1).unix()} to: ${moment(session_time).utc().unix()}

+insert_records: td>: queries/reporting/video_rating_report_insert.sql database: reports engine: presto insert_into: video_rating_report


+fetch_records: td>: queries/reporting/video_ratings_fetch.sql database: reports engine: presto store_last_results: true

+slack_notification: http>: https://hooks.slack.com/services/XXXXXXXX method: POST content: text: “Average rating is ${td.last_results.average_ratings}” link_names: true content_format: json







web console














Technology

スタディサプリを支えるデータ分析基盤 ～設計の勘所と利活用事例～

スタディサプリを支えるデータ分析基盤～設計の勘所と利活用事例～