ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã解説ïŒã°ããŒãã«ããžãã¹åãã®ã¹ã±ãŒã©ãã«ã§è²»çšå¯Ÿå¹æã®é«ãããŒã¿ã¹ãã¬ãŒãžã«é¢ããå æ¬çãªã¬ã€ããèšèšãå©ç¹ã課é¡ããã¹ããã©ã¯ãã£ã¹ãç¶²çŸ ã
ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ïŒçŸä»£ã®ãšã³ã¿ãŒãã©ã€ãºåãã®ã¹ã±ãŒã©ãã«ãªããŒã¿ã¹ãã¬ãŒãž
仿¥ã®ããŒã¿é§ååã®äžçã§ã¯ãäžçäžã®çµç¹ãããŒã¿ã®ææ°é¢æ°çãªå¢å ã«èŠæ ®ããŠããŸãã顧客ãšã®ã€ã³ã¿ã©ã¯ã·ã§ã³ãéèååŒãããã»ã³ãµãŒããŒã¿ããœãŒã·ã£ã«ã¡ãã£ã¢ãã£ãŒããŸã§ãããŒã¿ã®éãé床ãããã³å€æ§æ§ã¯åžžã«å¢å ããŠããŸãããã®ããŒã¿ã广çã«ç®¡çããã³æŽ»çšããããã«ãäŒæ¥ã¯ãŸããŸãããŒã¿ã¬ã€ã¯ã«ç®ãåããŠããŸããããŒã¿ã¬ã€ã¯ã¯ã倧éã®çããŒã¿ããã€ãã£ã圢åŒã§ä¿åããããã«èšèšãããéäžãªããžããªã§ãããã®ããã°æçš¿ã§ã¯ãããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã«é¢ããå æ¬çãªã¬ã€ããæäŸãããã®å©ç¹ãèšèšäžã®èæ ®äºé ã課é¡ãããã³ã¹ã±ãŒã©ãã«ã§å¹æçãªããŒã¿ã¹ãã¬ãŒãžãœãªã¥ãŒã·ã§ã³ãæ§ç¯ããããã®ãã¹ããã©ã¯ãã£ã¹ã«ã€ããŠèª¬æããŸãã
ããŒã¿ã¬ã€ã¯ãšã¯ïŒ
ããŒã¿ã¬ã€ã¯ã¯ãæ§é åããŒã¿ãšéæ§é åããŒã¿ã®ãã¹ãŠãããããèŠæš¡ã§ä¿åã§ããéäžãªããžããªã§ãã峿 Œãªã¹ããŒããšããŒã¿å€æãäºåã«èª²ãåŸæ¥ã®ããŒã¿ãŠã§ã¢ããŠã¹ãšã¯ç°ãªããããŒã¿ã¬ã€ã¯ã¯ãã¹ããŒããªã³ãªãŒããã¢ãããŒããæ¡çšããŠããŸããããã¯ãããŒã¿ãäºåã«å®çŸ©ãããã¹ããŒããåºç¯ãªå€æãªãã«ãç圢åŒã§ä¿åãããããšãæå³ããŸãããã®æè»æ§ã«ãããæ¬¡ã®ãããªããã€ãã®çš®é¡ã®ããŒã¿ãä¿åã§ããŸãã
- æ§é åããŒã¿ïŒãªã¬ãŒã·ã§ãã«ããŒã¿ããŒã¹ãCSVãã¡ã€ã«ãªã©
- åæ§é åããŒã¿ïŒJSONãXMLãªã©
- éæ§é åããŒã¿ïŒããã¹ãããã¥ã¡ã³ããç»åããªãŒãã£ãªããããªãªã©
ããŒã¿ã¬ã€ã¯ã¯ãå€ãã®å Žåãã³ã¢ãã£ãã£ããŒããŠã§ã¢ãŸãã¯ã¯ã©ãŠãããŒã¹ã®ãªããžã§ã¯ãã¹ãã¬ãŒãžãµãŒãã¹äžã«æ§ç¯ãããããã倧éã®ããŒã¿ãä¿åããã®ã«è²»çšå¯Ÿå¹æãé«ããªããŸããããŒã¿åæãæ©æ¢°åŠç¿ãããã³ãã®ä»ã®é«åºŠãªãŠãŒã¹ã±ãŒã¹åãã®æè»ã§ã¹ã±ãŒã©ãã«ãªãã©ãããã©ãŒã ãæäŸããŸãã
ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã®äž»ãªå©ç¹
ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ãæ¡çšãããšãããŒã¿è³ç£ã掻çšããããšããŠããçµç¹ã«ããã€ãã®å€§ããªã¡ãªããããããŸãã
- ã¹ã±ãŒã©ããªãã£ïŒããŒã¿ã¬ã€ã¯ã¯ãå€§èŠæš¡ãªããŒã¿ã»ããã«å¯Ÿå¿ããããã«ç°¡åã«æ¡åŒµã§ãããããäŒæ¥ã¯ãã¿ãã€ãåäœã®ããŒã¿ãä¿åããã³åŠçã§ããŸããç¹ã«ã¯ã©ãŠãããŒã¹ã®ããŒã¿ã¬ã€ã¯ã¯ãäºå®äžç¡å¶éã®ã¹ã±ãŒã©ããªãã£ãæäŸããŸãã
- è²»çšå¯Ÿå¹æïŒããŒã¿ã¬ã€ã¯ã¯ãå€ãã®å Žåããªããžã§ã¯ãã¹ãã¬ãŒãžã®ãããªè²»çšå¯Ÿå¹æã®é«ãã¹ãã¬ãŒãžãªãã·ã§ã³ãå©çšãããããåŸæ¥ã®ããŒã¿ãŠã§ã¢ããŠã¹ãšæ¯èŒããŠãããŒã¿ã¹ãã¬ãŒãžã®å šäœçãªã³ã¹ããåæžã§ããŸãã
- æè»æ§ïŒã¹ããŒããªã³ãªãŒãã¢ãããŒãã䜿çšãããšãããŒã¿ãå ã®åœ¢åŒã§ä¿åã§ãããããããŸããŸãªããŒã¿åããŠãŒã¹ã±ãŒã¹ã«æè»ã«å¯Ÿå¿ã§ããŸããåºç¯ãªäºåã®ããŒã¿ã¢ããªã³ã°ãè¡ããã«ãæ°ããããŒã¿ãœãŒã¹ãé²åããããžãã¹èŠä»¶ã«é©å¿ã§ããŸãã
- ä¿ææ§ïŒããŒã¿ã¬ã€ã¯ã«ãããè¿ éãªå®éšãšã€ãããŒã·ã§ã³ãå¯èœã«ãªããŸããããŒã¿ãµã€ãšã³ãã£ã¹ããšã¢ããªã¹ãã¯ã峿 ŒãªããŒã¿æ§é ãETLããã»ã¹ã«å¶çŽãããããšãªããããŒã¿ã«è¿ éã«ã¢ã¯ã»ã¹ããŠåæã§ããŸããããã«ãããã€ã³ãµã€ããåŸããŸã§ã®æéãççž®ãããã¢ãžã£ã€ã«éçºæ¹æ³è«ããµããŒããããŸãã
- é«åºŠãªåæïŒããŒã¿ã¬ã€ã¯ã¯ãæ©æ¢°åŠç¿ã人工ç¥èœãäºæž¬ã¢ããªã³ã°ãªã©ã®é«åºŠãªåæãŠãŒã¹ã±ãŒã¹ã«æé©ã§ãã倿§ãªããŒã¿åãä¿åããè€éãªåŠçææ³ãé©çšã§ãããããæ°ããã€ã³ãµã€ããšæ©äŒãéãããŸãã
- ããŒã¿ã®æ°äž»åïŒããŒã¿ã¬ã€ã¯ã¯ãçµç¹å ã®ããå¹ åºããŠãŒã¶ãŒãããŒã¿ã«ã¢ã¯ã»ã¹ããããããŸããããã«ãããããžãã¹ãŠãŒã¶ãŒã¯ããŒã¿é§ååã®æææ±ºå®ãè¡ããããŒã¿ãªãã©ã·ãŒãšã³ã©ãã¬ãŒã·ã§ã³ã®æåãè²æã§ããŸãã
ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã®èšèšïŒäž»èŠã³ã³ããŒãã³ã
å ç¢ãªããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ãèšèšããã«ã¯ãããŸããŸãªã³ã³ããŒãã³ããšãã®çžäºäœçšãæ éã«æ€èšããå¿ èŠããããŸããäžè¬çãªããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã®äž»èŠãªèŠçŽ ãæ¬¡ã«ç€ºããŸãã
1. ããŒã¿åã蟌ã¿
ããŒã¿åã蟌ã¿ã¯ãããŒã¿ãããŒã¿ã¬ã€ã¯ã«åã蟌ãããã»ã¹ã§ããããã«ã¯ã次ã®ãããªããã€ãã®æ¹æ³ããããŸãã
- ãããåã蟌ã¿ïŒéåžžã¯ããŒã¿ããŒã¹ããã©ãããã¡ã€ã«ããŸãã¯ãã®ä»ã®ããŒã¿ãœãŒã¹ããã倧éã®ããŒã¿ããããã§ããŒãããŸããApache SqoopãApache NiFiãAWS GlueãAzure Data Factoryã®ãããªã¯ã©ãŠãããŒã¹ã®ãµãŒãã¹ããããåã蟌ã¿ã«äœ¿çšã§ããŸãã
- ã¹ããªãŒã åã蟌ã¿ïŒWebãµãŒããŒãã°ãIoTããã€ã¹ããœãŒã·ã£ã«ã¡ãã£ã¢ãã£ãŒããªã©ã®ãœãŒã¹ãããªã¢ã«ã¿ã€ã ã®ããŒã¿ã¹ããªãŒã ããã£ããã£ããŸããApache KafkaãApache FlinkãAWS KinesisãAzure Event Hubsã®ãããªã¯ã©ãŠãããŒã¹ã®ã¹ããªãŒãã³ã°ãµãŒãã¹ãäžè¬çã«äœ¿çšãããŸãã
- APIçµ±åïŒããŸããŸãªã¢ããªã±ãŒã·ã§ã³ããµãŒãã¹ããæäŸãããAPIããããŒã¿ãååŸããŸãã
广çãªããŒã¿åã蟌ã¿ããã»ã¹ã«ãããããŒã¿ãæ£ç¢ºãå¹ççããã€ç¢ºå®ã«ãã£ããã£ãããããšãä¿èšŒãããŸãã
2. ããŒã¿ã¹ãã¬ãŒãž
ããŒã¿ã¹ãã¬ãŒãžã¯ãããŒã¿ã¬ã€ã¯ã®åºç€ã§ããããŒã¿ã¯éåžžãç圢åŒã§è²»çšå¯Ÿå¹æã®é«ãã¹ãã¬ãŒãžãœãªã¥ãŒã·ã§ã³ïŒå€ãã®å Žåãæ¬¡ã®ãããªã¯ã©ãŠãããŒã¹ã®ãªããžã§ã¯ãã¹ãã¬ãŒãžïŒã«ä¿åãããŸãã
- AWS S3ïŒAmazon Simple Storage Service
- Azure Blob StorageïŒMicrosoft Azure Blob Storage
- Google Cloud StorageïŒGoogle Cloud Storage
ãããã®ãµãŒãã¹ã¯ãé«ãèä¹ æ§ãã¹ã±ãŒã©ããªãã£ãããã³å¯çšæ§ãæäŸããŸããã¹ãã¬ãŒãžå±€ã¯ãã¹ãã¬ãŒãžå¹çãšã¯ãšãªããã©ãŒãã³ã¹ãæé©åããããã«ãCSVãParquetãAvroãJSONãªã©ã®ããŸããŸãªããŒã¿åœ¢åŒããµããŒãããå¿ èŠããããŸãã
3. ããŒã¿åŠç
ããŒã¿åŠçã«ã¯ãããŒã¿ã¬ã€ã¯ã«ä¿åãããŠããçããŒã¿ã®å€æãã¯ã¬ã³ãžã³ã°ãããã³ãšã³ãªããã¡ã³ããå«ãŸããŸããäžè¬çãªããŒã¿åŠçã¿ã¹ã¯ã«ã¯ã次ã®ãã®ããããŸãã
- ETLïŒæœåºã倿ãããŒãïŒïŒåŸæ¥ã®ETLããã»ã¹ã§ã¯ããœãŒã¹ã·ã¹ãã ããããŒã¿ãç§»åãã倿ããŠãããŒã¿ãŠã§ã¢ããŠã¹ãŸãã¯ãã®ä»ã®åæã·ã¹ãã ã«ããŒãããŸãã
- ELTïŒæœåºãããŒãã倿ïŒïŒELTããã»ã¹ã§ã¯ãçããŒã¿ãããŒã¿ã¬ã€ã¯ã«ããŒãããæ¬¡ã«ã€ã³ã¬ã€ã¯åŠçãšã³ãžã³ã䜿çšããŠå€æãå®è¡ããŸãã
- ããŒã¿ã¯ã¬ã³ãžã³ã°ãšæ€èšŒïŒããŒã¿å ã®ãšã©ãŒãççŸãããã³æ¬ æå€ãèå¥ããŠä¿®æ£ããŸãã
- ããŒã¿å€æïŒããŒã¿ããã圢åŒããå¥ã®åœ¢åŒã«å€æããããããŒã¿ãéèšããããæ°ããããŒã¿ãã£ãŒã«ããäœæãããããŸãã
- ããŒã¿ãšã³ãªããã¡ã³ãïŒä»ã®ãœãŒã¹ããã®æ å ±ãçµ±åããŠãããŒã¿ã«ã³ã³ããã¹ãã远å ããŸãã
äžè¬çãªããŒã¿åŠçããŒã«ã«ã¯ãApache SparkãApache HiveãApache PigãAWS EMRãAzure DatabricksãGoogle Dataprocã®ãããªã¯ã©ãŠãããŒã¹ã®ãµãŒãã¹ãªã©ããããŸãã
4. ããŒã¿ã«ã¿ãã°ãšã¡ã¿ããŒã¿ç®¡ç
ããŒã¿ã«ã¿ãã°ã¯ãããŒã¿ã¬ã€ã¯å ã®ããŒã¿ãæŽçããã³ç®¡çããããã«äžå¯æ¬ ã§ããããã¯æäŸããŸãïŒ
- ã¡ã¿ããŒã¿ç®¡çïŒã¹ããŒããããŒã¿ãªããŒãžãããŒã¿å質ã¡ããªãã¯ãããŒã¿æææš©ãªã©ãããŒã¿ã«é¢ããæ å ±ãç¶æããŸãã
- ããŒã¿æ€åºïŒãŠãŒã¶ãŒãå¿ èŠãªããŒã¿ãç°¡åã«èŠã€ããŠçè§£ã§ããããã«ããŸãã
- ããŒã¿ã¬ããã³ã¹ïŒããŒã¿å質ã«ãŒã«ãã¢ã¯ã»ã¹å¶åŸ¡ãããã³ã³ã³ãã©ã€ã¢ã³ã¹èŠä»¶ãé©çšããŸãã
äžè¬çãªããŒã¿ã«ã¿ãã°ããŒã«ã«ã¯ãApache AtlasãAWS Glue Data CatalogãAzure Data CatalogãAlationãªã©ããããŸãã
5. ããŒã¿ã»ãã¥ãªãã£ãšã¢ã¯ã»ã¹å¶åŸ¡
ããŒã¿ã»ãã¥ãªãã£ã¯æãéèŠã§ããæ©å¯ããŒã¿ãä¿è·ããããã«ã次ã®ãããªå ç¢ãªã»ãã¥ãªãã£å¯Ÿçãå®è£ ããŸãã
- æå·åïŒä¿åæããã³è»¢éäžã®ããŒã¿ãæå·åããŸãã
- ã¢ã¯ã»ã¹å¶åŸ¡ïŒãŠãŒã¶ãŒã®åœ¹å²ãšæš©éã«åºã¥ããŠããŒã¿ãžã®ã¢ã¯ã»ã¹ãå¶éããããã«ã詳现ãªã¢ã¯ã»ã¹å¶åŸ¡ãå®çŸ©ããŸãã
- èªèšŒãšæ¿èªïŒãŠãŒã¶ãŒIDã確èªããããã«ã匷åãªèªèšŒã¡ã«ããºã ãå®è£ ããŸãã
- ç£æ»ïŒãã¹ãŠã®ããŒã¿ã¢ã¯ã»ã¹ããã³å€æŽã¢ã¯ãã£ããã£ãç£èŠããã³ãã°èšé²ããŸãã
ã¯ã©ãŠããããã€ããŒã¯ãAWS IAMãAzure Active DirectoryãGoogle Cloud IAMãªã©ã®ããŸããŸãªã»ãã¥ãªãã£æ©èœãšãµãŒãã¹ãæäŸããŠãããŒã¿ã¬ã€ã¯ã®ã»ãã¥ãªãã£ã確ä¿ããã®ã«åœ¹ç«ã¡ãŸãã
6. ããŒã¿æ¶è²»ãšåæ
ããŒã¿ã¬ã€ã¯ã¯ãããŸããŸãªåæãŠãŒã¹ã±ãŒã¹ã®åºç€ãšããŠæ©èœããŸããããŒã¿ã³ã³ã·ã¥ãŒããŒã¯ãããŒã«ãšãã¯ãããžãŒãå©çšããŠããŒã¿ããã€ã³ãµã€ããæœåºããŸããããã«ã¯ä»¥äžãå«ãŸããŸãã
- ããŒã¿ãŠã§ã¢ããŠãžã³ã°ïŒAmazon RedshiftãAzure Synapse AnalyticsããŸãã¯Google BigQueryã®ãããªããŒã¿ãŠã§ã¢ããŠã¹ã«ããŒã¿ãããŒãããŸãã
- ããžãã¹ã€ã³ããªãžã§ã³ã¹ïŒBIïŒïŒTableauãPower BIãLookerã®ãããªBIããŒã«ãå©çšããŠãããã·ã¥ããŒããšã¬ããŒããäœæããŸãã
- æ©æ¢°åŠç¿ïŒMLïŒïŒTensorFlowãPyTorchãã¯ã©ãŠãããŒã¹ã®MLãµãŒãã¹ã®ãããªããŒã«ã䜿çšããŠãMLã¢ãã«ããã¬ãŒãã³ã°ããã³ãããã€ããŸãã
- ã¢ãããã¯ã¯ãšãªïŒPrestoãTrinoããŸãã¯Apache Impalaã®ãããªSQLããŒã¹ã®ããŒã«ã䜿çšããŠãããŒã¿ã¬ã€ã¯ããããŒã¿ãçŽæ¥ã¯ãšãªããŸãã
ããŒã¿ã¬ã€ã¯ã®ãããã€ã¢ãã«
ããŒã¿ã¬ã€ã¯ããããã€ããæ¹æ³ã¯ããã€ããããŸãã
- ãªã³ãã¬ãã¹ïŒç¬èªã®ã€ã³ãã©ã¹ãã©ã¯ãã£ã«ããŒã¿ã¬ã€ã¯ããããã€ããŸãããã®ãªãã·ã§ã³ã«ã¯ãããŒããŠã§ã¢ãšã€ã³ãã©ã¹ãã©ã¯ãã£ãžã®å€é¡ã®åææè³ãå¿ èŠã§ãã峿 ŒãªããŒã¿æåšå°èŠä»¶ãŸãã¯æ¢åã®éèŠãªããŒããŠã§ã¢æè³ãããçµç¹ã¯ããããæ€èšããå ŽåããããŸãã
- ã¯ã©ãŠãããŒã¹ïŒã¹ãã¬ãŒãžãåŠçãåæã«ã¯ã©ãŠããµãŒãã¹ïŒAWSãAzureãGCPïŒã掻çšããŸããããã«ãããã¹ã±ãŒã©ããªãã£ãè²»çšå¯Ÿå¹æãããã³ç®¡çã®å®¹æããæäŸãããŸããããã¯ã仿¥ã®æãäžè¬çãªãããã€ã¢ãã«ã§ãã
- ãã€ããªããïŒãªã³ãã¬ãã¹ã³ã³ããŒãã³ããšã¯ã©ãŠãããŒã¹ã®ã³ã³ããŒãã³ããçµã¿åãããŸãããã®ã¢ãããŒãã¯ãèŠå¶äžã®å¶çŽãŸãã¯ã»ãã¥ãªãã£äžã®çç±ã«ãããäžéšã®ããŒã¿ããªã³ãã¬ãã¹ã«ä¿æããå¿ èŠããããã¯ã©ãŠãã®ã¹ã±ãŒã©ããªãã£ãšæè»æ§ã掻çšããå¿ èŠãããçµç¹ã«é©ããŠããŸãã
ããŒã¿ã¬ã€ã¯ã®å®è£ ã«ããã課é¡ãšèæ ®äºé
ããŒã¿ã¬ã€ã¯ã«ã¯å€ãã®å©ç¹ããããŸããã广çã«å®è£ ããã³ç®¡çããã«ã¯ãããã€ãã®èª²é¡ããããŸãã
1. ããŒã¿ã¬ããã³ã¹
å ç¢ãªããŒã¿ã¬ããã³ã¹ããªã·ãŒã確ç«ããããšãéèŠã§ããããã«ã¯ä»¥äžãå«ãŸããŸãã
- ããŒã¿å質ïŒããŒã¿ã®æ£ç¢ºæ§ãå®å šæ§ãããã³äžè²«æ§ãä¿èšŒããŸããããŒã¿æ€èšŒã«ãŒã«ãšå質ãã§ãã¯ãå®è£ ããŸãã
- ããŒã¿ãªããŒãžïŒããŒã¿ã®èµ·æºãšå€æå±¥æŽã远跡ããŸãã
- ããŒã¿ã«ã¿ãã°ïŒã¡ã¿ããŒã¿ã䜿çšããŠããŒã¿è³ç£ãææžåããŸãã
- ããŒã¿ã»ãã¥ãªãã£ãšã³ã³ãã©ã€ã¢ã³ã¹ïŒããŒã¿ãã©ã€ãã·ãŒèŠå¶ïŒäŸïŒGDPRãCCPAïŒãéµå®ããã¢ã¯ã»ã¹å¶åŸ¡ãå®è£ ããŸãã
2. ããŒã¿ã»ãã¥ãªãã£
ããŒã¿ã¬ã€ã¯ã®ã»ãã¥ãªãã£ã確ä¿ããããšãéèŠã§ããããã«ã¯ã匷åãªèªèšŒãæ¿èªãæå·åãããã³ç£æ»ã¡ã«ããºã ã®å®è£ ãå¿ èŠã§ããé²åããè åšã«å¯ŸåŠããããã«ãã»ãã¥ãªãã£ããªã·ãŒã宿çã«èŠçŽããæŽæ°ããŸãã
3. ããŒã¿ããŒãžã§ãã³ã°ãšã¹ããŒãé²å
ããŒã¿ã¹ããŒãã¯æéã®çµéãšãšãã«å€åããå¯èœæ§ããããŸããäžäœäºææ§ãšããŒãžã§ãã³ã°ãåŠçããããã®ããŒã«ãšææ³ã䜿çšããŠãã¹ããŒãã®é²åã广çã«ç®¡çããŸããApache AvroãŸãã¯Apache Parquetã®ãããªã¹ããŒãã¬ãžã¹ããªãœãªã¥ãŒã·ã§ã³ã®äœ¿çšãæ€èšããŠãã ããã
4. ããŒã¿ãµã€ã
ããŒã¿ãµã€ãã®äœæãé²ããŸããããŸããŸãªããŒã ãšéšééã®ã³ã©ãã¬ãŒã·ã§ã³ãšç¥èå ±æã奚å±ããŸããããŒã¿ã¬ã€ã¯å šäœã§äžè²«æ§ãšæŽåæ§ã確ä¿ããããã«ãçµ±åãããããŒã¿ã¬ããã³ã¹ãã¬ãŒã ã¯ãŒã¯ãå®è£ ããŸãã
5. ããŒã¿ã®è€éã
å€§èŠæš¡ã§å€æ§ãªããŒã¿ã»ããã®è€éãã管çããã«ã¯ãç¹å¥ãªã¹ãã«ãšå°éç¥èãå¿ èŠã§ããããŒã¿ãšã³ãžãã¢ãªã³ã°ããŒã ãšããŒã¿ãµã€ãšã³ã¹ããŒã ã®ãã¬ãŒãã³ã°ãšã¹ãã«ã¢ããã«æè³ããŸããããŒã¿ã广çã«æŽçããããã«ãããŒã¿ã¬ããã³ã¹ãã¬ãŒã ã¯ãŒã¯ã®äœ¿çšãæ€èšããŠãã ããã
6. ããã©ãŒãã³ã¹ã®æé©å
ã¿ã€ã ãªãŒãªã€ã³ãµã€ãã確ä¿ããã«ã¯ãã¯ãšãªããã©ãŒãã³ã¹ã®æé©åãäžå¯æ¬ ã§ããããã«ã¯ä»¥äžãå«ãŸããŸãã
- é©åãªããŒã¿åœ¢åŒã®éžæïŒParquetãAvroãORCã¯ãã¯ãšãªããã©ãŒãã³ã¹ãåäžãããã«ã©ã ãã¹ãã¬ãŒãžã«æé©åãããŠããŸãã
- ããŒã¿ã®ããŒãã£ã·ã§ã³åå²ïŒæ¥ä»ãå°åãªã©ã®äž»èŠãªãã£ã¡ã³ã·ã§ã³ã«åºã¥ããŠããŒã¿ãããŒãã£ã·ã§ã³åå²ãããšãã¯ãšãªããã©ãŒãã³ã¹ãå€§å¹ ã«åäžããŸãã
- ã€ã³ããã¯ã¹äœæïŒé »ç¹ã«ã¯ãšãªãããåã«ã€ã³ããã¯ã¹ãäœæããŸãã
- ã¯ãšãªã®æé©åïŒäžŠååŠçæ©èœã掻çšããããã«ã¯ãšãªãæé©åããŸãã
æåããããŒã¿ã¬ã€ã¯ãæ§ç¯ããããã®ãã¹ããã©ã¯ãã£ã¹
ãã¹ããã©ã¯ãã£ã¹ã«åŸãããšã§ãããŒã¿ã¬ã€ã¯ã®å®è£ ãæåããããšãä¿èšŒãããŸãã
- æç¢ºãªããžãã¹ç®æšãå®çŸ©ããïŒããŒã¿ã¬ã€ã¯ã§è§£æ±ºãããç¹å®ã®ããžãã¹äžã®åé¡ãç¹å®ããŸããããã«ãããããŒã¿ã¬ã€ã¯ã®èšèšãšå®è£ ãã¬ã€ããããŸãã
- å°ããå§ããŠå埩ããïŒã¢ãŒããã¯ãã£ãæ€èšŒããã¹ã±ãŒã«ã¢ããããåã«çµéšãç©ãããã«ããã€ããããããžã§ã¯ãããå§ããŸããåŠç¿ããæèšã«åºã¥ããŠããŒã¿ã¬ã€ã¯ãå埩ããã³æ¹è¯ããŸãã
- é©åãªãã¯ãããžãŒãéžæããïŒããžãã¹èŠä»¶ãããŒã¿éãããã³äºç®ã«åèŽãããã¯ãããžãŒãéžæããŸãããªãŒãã³ãœãŒã¹ããŒã«ãã¯ã©ãŠãããŒã¹ã®ãµãŒãã¹ãããã³åçšãœãªã¥ãŒã·ã§ã³ãæ€èšããŠãã ããã
- å ç¢ãªããŒã¿ã¬ããã³ã¹ãã¬ãŒã ã¯ãŒã¯ãå®è£ ããïŒããŒã¿åè³ªæšæºãããŒã¿ãªããŒãžãã¡ã¿ããŒã¿ç®¡çãããã³ã¢ã¯ã»ã¹å¶åŸ¡ã確ç«ããŸãã
- ããŒã¿ã»ãã¥ãªãã£ãåªå ããïŒäžæ£ã¢ã¯ã»ã¹ããããŒã¿ãä¿è·ããããã«ã匷åãªã»ãã¥ãªãã£å¯Ÿçãå®è£ ããŸãã
- ããŒã¿ãã€ãã©ã€ã³ãèªååããïŒããŒã¿åã蟌ã¿ã倿ãããã³ããŒãããã»ã¹ãèªååããŠãå¹çãåäžããããšã©ãŒãæžãããŸããApache Airflowã®ãããªã¯ãŒã¯ãããŒç®¡çã·ã¹ãã ã䜿çšããŸãã
- ããã©ãŒãã³ã¹ãç£èŠããã³æé©åããïŒããŒã¿ã¬ã€ã¯ã®ããã©ãŒãã³ã¹ãç¶ç¶çã«ç£èŠããã¯ãšãªãã¹ãã¬ãŒãžãããã³åŠçãæé©åããŠãæé©ãªããã©ãŒãã³ã¹ã確ä¿ããŸãã
- ã¹ãã«ãšãã¬ãŒãã³ã°ã«æè³ããïŒããŒã¿ãšã³ãžãã¢ãªã³ã°ããŒã ãšããŒã¿ãµã€ãšã³ã¹ããŒã ã«ãã¬ãŒãã³ã°ãæäŸããŠãããŒã¿ã¬ã€ã¯ã广çã«ç®¡çããã³å©çšããããã«å¿ èŠãªã¹ãã«ãšç¥èãç¿åŸãããŸãã
- ããŒã¿é§ååã®æåãæ§ç¯ããïŒããŒã¿ãªãã©ã·ãŒã®æåãè²æããçµç¹å šäœã§ããŒã¿é§ååã®æææ±ºå®ã奚å±ããŸãã
- é©åãªã¹ããŒãé²åæŠç¥ãéžæããïŒå¯èœãªå Žåã¯äžäœäºææ§ãæ€èšããŠãã ããã
ããŒã¿ã¬ã€ã¯ã®ã°ããŒãã«ãªå®è£ äŸ
ããŒã¿ã¬ã€ã¯ã¯ãããŸããŸãªããžãã¹äžã®èª²é¡ã«å¯ŸåŠããããã«ãäžçäžã®çµç¹ã§æ¡çšãããŠããŸããæ¬¡ã«ããã€ãã®äŸã瀺ããŸãã
- éèãµãŒãã¹ïŒéè¡ããã³éèæ©é¢ã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠé¡§å®¢ããŒã¿ãåæããäžæ£è¡çºãæ€åºãããªã¹ã¯ã管çãã顧客äœéšãããŒãœãã©ã€ãºããŠããŸããããšãã°ã倧æåœééè¡ã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠããŸããŸãªåœã®ååŒããŒã¿ãåæããäžæ£è¡çºãç¹å®ããã»ãã¥ãªãã£ãããã³ã«ãæ¹åããå ŽåããããŸãã
- ãã«ã¹ã±ã¢ïŒãã«ã¹ã±ã¢ãããã€ããŒã¯ãããŒã¿ã¬ã€ã¯ã掻çšããŠæ£è ããŒã¿ãä¿åããã³åæããæ£è ã®ã¢ãŠãã«ã ãæ¹åããå»åŠç ç©¶ãå éããŠããŸããããšãã°ããšãŒãããå šäœã®ç é¢ã¯ãæ£è ããŒã¿ãåæããŠç é¢ã®éå¶ãæé©åããæ£è ã®ããŒãºãäºæž¬ã§ããŸãã
- å°å£²ïŒå°å£²æ¥è ã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠé¡§å®¢ã®è¡åãçè§£ããããŒã±ãã£ã³ã°ãã£ã³ããŒã³ãããŒãœãã©ã€ãºãããµãã©ã€ãã§ãŒã³ãæé©åããŠããŸããã°ããŒãã«ãªeã³ããŒã¹äŒæ¥ã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠé¡§å®¢ã®è³Œè²·ãã¿ãŒã³ãåæããããŒãœãã©ã€ãºããã補åã®æšå¥šãè¡ãå ŽåããããŸãã
- 補é ïŒè£œé æ¥è ã¯ãããŒã¿ã¬ã€ã¯ãå©çšããŠè£œé è£ çœ®ããã»ã³ãµãŒããŒã¿ãåéããã³åæããçç£ããã»ã¹ãæé©åããè£ çœ®ã®æ éãäºæž¬ããŠããŸããããšãã°ãæ¥æ¬ãšãã€ãã®äŒæ¥ã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠè£œé è£ çœ®ã®äºç¥ä¿å šãå®è¡ããŠããŸãã
- 黿°éä¿¡ïŒé»æ°éä¿¡äŒç€Ÿã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠãããã¯ãŒã¯ããã©ãŒãã³ã¹ãåæãã顧客ã®è§£çŽã管çãã顧客ãªãã¡ãŒãããŒãœãã©ã€ãºããŠããŸããã€ã³ãã®é»æ°éä¿¡ãããã€ããŒã¯ãããŒã¿ã¬ã€ã¯ã䜿çšããŠãããã¯ãŒã¯ããã©ãŒãã³ã¹ãšé¡§å®¢ã®äœ¿çšç¶æ³ãåæãããããã¯ãŒã¯åè³ªãæ¹åããæé©åãããããŒã¿ãã©ã³ãæäŸããå ŽåããããŸãã
çµè«
ããŒã¿ã¬ã€ã¯ã¢ãŒããã¯ãã£ã¯ãå€§èŠæš¡ã§å€æ§ãªããŒã¿ã»ãããä¿åããã³åŠçããããã®åŒ·åã§æè»ãªãã©ãããã©ãŒã ãæäŸããŸããäž»èŠãªã³ã³ããŒãã³ããå©ç¹ãããã³èª²é¡ãçè§£ããããšã§ãçµç¹ã¯ç¹å®ã®ããŒãºãæºããããŒã¿ã¬ã€ã¯ãèšèšããã³å®è£ ã§ããŸãããã¹ããã©ã¯ãã£ã¹ã«åŸãã匷åãªããŒã¿ã¬ããã³ã¹ãã¬ãŒã ã¯ãŒã¯ã確ç«ããé©åãªãã¯ãããžãŒãšã¹ãã«ã«æè³ããããšãã貎éãªã€ã³ãµã€ããåŒãåºããããžãã¹ã€ãããŒã·ã§ã³ãæšé²ããæåããããŒã¿ã¬ã€ã¯ãæ§ç¯ããããã«äžå¯æ¬ ã§ããããŒã¿ãææ°é¢æ°çã«å¢ãç¶ããã«ã€ããŠãããŒã¿ã¬ã€ã¯ã¯ãäžçäžã®çµç¹ãããŒã¿é§ååã®æä»£ã«ç¹æ ããã®ã«åœ¹ç«ã€äžã§ãŸããŸãéèŠãªåœ¹å²ãæããã§ãããã