Åä¾°

½üÄêÀ´
£¬Ëæ×ÅÈ˹¤ÖÇÄÜÊÖÒÕµÄѸÃÍÉú³¤
£¬»úеѧϰģ×ӵĹæÄ£·ºÆð³ö±¬·¢Ê½ÔöÌí̬ÊÆ¡£ÓÈÆäÔÚÉî¶ÈѧϰÁìÓò
£¬Ä£×Ó¹æÄ£µÄÏÔÖøÌá¸ß³ÉΪÍƶ¯Öî¶àÊÖÒÕÍ»ÆÆÓëÐÔÄÜÌáÉýµÄÒªº¦ÒªËØ¡£µ±Ä£×Ó²ÎÊýÊýÄ¿µÖ´ïÍòÒÚ¼¶±ðʱ
£¬ÊÜÏÞÓÚµ¥Ì¨×°±¸µÄÎïÀí×ÊÔ´¶øÎÞ·¨ÊµÏÖµ¥»úѵÁ·¡£Îª´Ë
£¬ÂþÑÜʽѵÁ·ÊÖÒÕÓ¦Ô˶øÉú
£¬²¢Ñ¸ËÙ³ÉΪѵÁ·³¬´ó¹æģģ×ӵĽ¹µãÊֶΡ£ÂþÑÜʽѵÁ·Äܹ»ÏÔÖøÌá¸ßѵÁ·Ð§ÂÊ
£¬Í»ÆƵ¥»úÄÚ´æºÍÅÌËãÄÜÁ¦µÄÆ¿¾±¡£ÈÚºÏÄ£×Ó²¢ÐÐÓëÊý¾Ý²¢ÐеÈÊÖÒÕµÄÂþÑÜʽ»ìÏý²¢ÐÐѵÁ·Õ½ÂÔ³ÉΪѵÁ·³¬´ó¹æÄ£´óÓïÑÔÄ£×ÓµÄÖ÷ÒªÊÖÒÕ;¾¶
£¬¸ÃÒªÁì²»µ«ÄÜÓÐÓÃʹÓôó¹æÄ£Ó²¼þ×ÊÔ´
£¬»¹ÄÜÔÚÈ·±£ÑµÁ·ÎȹÌÐÔºÍÄ£×ÓÐÔÄܵÄÌõ¼þÏÂ
£¬ÊµÏÖ´óÓïÑÔÄ£×ӵĸßЧѵÁ·Óë°²ÅÅ¡£

Ò»¡¢ÍøÂç¹ÊÕÏ

ÂþÑÜʽѵÁ·ÊÇÖ¸ÔÚ¶à¸öÅÌËã½ÚµãÉÏЭͬ×÷Òµ
£¬ÅäºÏÍê³É»úеѧϰģ×ÓµÄѵÁ·Àú³Ì¡£Í¨¹ý½«ÑµÁ·Ê¹Ãü²ð·ÖÖÁ¶à¸ö×°±¸Éϲ¢ÐÐÖ´ÐÐ
£¬²»µ«ÄܺÏÀí·ÖÅÉÅÌËãºÍ´æ´¢×ÊÔ´
£¬Ò²ÄÜÏÔÖøÌáÉýÑ·üçٶÈ
£¬²¢ÇÒÄܹ»´¦Öóͷ£¸ü´ó¹æÄ£µÄÊý¾ÝºÍÄ£×Ó¡£

È»¶ø
£¬ÑµÁ·Àú³ÌÖзºÆð¹ÊÕϵĿÉÄÜÐÔ»áËæ×ÅѵÁ·¹æÄ£ºÍÒ»Á¬Ê±¼äµÄÔöÌí¶øÉý¸ß¡£Ò»µ©±¬·¢¹ÊÕÏ
£¬½«Ê¹ËùÓмÓÈëµÄ×°±¸´¦ÓÚÏÐÖÃ״̬
£¬Ö±ÖÁ¹ÊÕÏ×°±¸»Ö¸´Õý³£
£¬µ¼Ö´ó×ÚËãÁ¦ÎÞ·¨»ñµÃ³ä·ÖʹÓá£À´×ÔMeta¡¢HuggingFaceºÍLAIONµÄÍŶӾù±¨¸æÁËÔÚѵÁ·´óÐÍÄ£×ÓʱÒòʧ°Üµ¼ÖÂʹÓÃÂÊÑÏÖØϽµµÄÇéÐΡ£

ƾ֤MetaÍŶÓѵÁ·Llama3.1µÄ±¨¸æ
£¬ÔÚѵÁ·Ê±´úƽ¾ùÿÈý¸öСʱ¾Í»á±¬·¢Ò»´Î¹ÊÕÏ
£¬ÆäÖÐÔ¼8.4%µÄ¹ÊÕÏÊÇÓÉÍøÂç×°±¸¹ÊÕϵ¼ÖµÄ¡£²î±ðÓÚÆäËûÓÉÈí¼þÒý·¢µÄ¹ÊÕÏ
£¬ÍøÂç×°±¸¹ÊÕÏ´ó¶àΪӲ¼þ¹ÊÕÏ
£¬ÀýÈçÍøÂçÏßÀ»òÊÇÍø¿¨¹âÄ£¿éµÈÓ²¼þ×°±¸·ºÆð¹ÊÕÏ¡£Ïà½ÏÓÚÈí¼þ¹ÊÕÏ
£¬Ó²¼þ¹ÊÕϵĴ¦Öóͷ£ÄѶȴ󡢻ָ´Ê±¼ä³¤
£¬¿ÉÄÜÔì³É¸ü´óµÄËãÁ¦Ëðʧ¡£

½»Á÷»ú×°±¸¼äÁ´Â·¹ÊÕÏÏÖÔÚÒÑÄܹ»ÊµÏÖºÁÃ뼶´¦Öóͷ£ÓëÊÕÁ²
£¬µ«¶Ë²àÍøÂç×°±¸¹ÊÕÏÏÖÔÚÉÐÎÞ½ÏΪ¸ßЧµÄ´¦Öóͷ£¼Æ»®¡£¶Ë²à×°±¸¹ÊÕÏÖ÷Òª°üÀ¨·þÎñÆ÷Íø¿¨¹ÊÕÏ¡¢·þÎñÆ÷Óë½»Á÷»úµÄÅþÁ¬ÏßÀÂËð»µÒÔ¼°ÏßÀ¹âÄ£¿é¹ÊÕϵÈ¡£Ëæ×ÅÅÌË㼯Ⱥ¹æÄ£Ò»Á¬À©´ó
£¬¶Ë²àÍøÂç×°±¸¹ÊÕÏÒѳÉΪؽ´ý½â¾öµÄÎÊÌâ¡£

¶þ¡¢¹ÊÕÏ´¦Öóͷ£

ÏÖÔÚ
£¬³ýÈí¼þ¹ÊÕÏÍâ
£¬Ó²¼þ¹ÊÕϵĴ¦Öóͷ£Õ½ÂÔͨ³£²»Çø·ÖÏêϸµÄ¹ÊÕÏÀàÐÍ
£¬±¬·¢Ó²¼þ¹ÊÕÏʱ
£¬Í¨³£½ÓÄÉÏàͬµÄ¼Æ»®¾ÙÐд¦Öóͷ£¡£

¼ì²éµã£¨Checkpoint£©ÊÇÒ»ÖÖ³£¼ûµÄ³¤ÆÚ»¯»úÖÆ
£¬ÓÃÓÚÉúÑÄѵÁ·½ø¶È¡£ÔÚ¹ÊÕϱ¬·¢ºó
£¬ÏµÍ³»áÁ¬Ã¦¶Ô¹ÊÕÏ×°±¸¾ÙÐÐÐÞ¸´
£¬»òÕß½«¹ÊÕÏ×°±¸´Ó¼¯ÈºÖÐÌÞ³ý
£¬È»ºó´Ó×î½üµÄ¼ì²éµã¼ÓÔØÄ£×Ó²¢¼ÌÐøѵÁ·¡£ÏÖÔÚ
£¬Ðí¶àÓÅ»¯Õ½ÂÔΧÈƼì²éµã»úÖÆÕö¿ª£º

1.EasyCkpt ½ÓÄÉÒì²½»¯¡¢ÌõÀí»¯µÄÉúÑÄ·½·¨
£¬ÍŽáÖصþÄ£×Ó¿½±´ÓëÅÌËã¡¢ÍøÂç¸ÐÖªµÄÒì²½´æ´¢Õ½ÂÔ
£¬ÊµÏÖÁ˽üºõÁ㿪ÏúµÄÄ£×ÓÉúÑÄ»úÖÆ
£¬²¢°ü¹ÜÁË´óÄ£×ÓѵÁ·Àú³ÌÖÐÄ£×ÓÉúÑÄÓë»Ö¸´µÄ¾«¶ÈÎÞËð¡£

2.Gemini Äܹ»½« Checkpoint ÉúÑÄÔÚ¾ßÓиü´ó¾ÛºÏ´ø¿íµÄ CPU ÄÚ´æÖÐ
£¬²¢Í¨¹ýһϵÁмƻ®ÊµÏÖÁË´óÐÍÄ£×ÓѵÁ·µÄ¿ìËÙ¹ÊÕϻָ´¡£

Ö»¹ÜÔÆÔÆ
£¬ÈÔÎÞ·¨ÍêÈ«×èÖ¹Òò¹ÊÕ϶¨Î»ºÍÄ£×ÓѵÁ·ÖØÆôËù´øÀ´µÄËãÁ¦Ëðʧ¡£

ÈßÓàÅÌË㣨Redundant computation£©¿ÉÒÔ×èÖ¹ÖØÐÂÉèÖúÍÖØÐÂÆô¶¯µÄ¿ªÏú
£¬ÔÚ¾ÙÐÐÄ£×ÓѵÁ·Ê±Ê¹Óòî±ðµÄ½Úµã¾ÙÐÐÈßÓàÅÌËã¡£µ±Ä³¸ö½Úµã±¬·¢¹ÊÕÏʱ
£¬ÆäËû½ÚµãÄܹ»È¡´ú¹ÊÕϽڵã¾ÙÐÐÅÌËã
£¬¿ÉÊÇÕâÑùÒýÈëÁËÀο¿µÄÄڴ濪ÏúºÍÅÌË㿪Ïú
£¬½øÒ»²½ÔöÌíÁËѵÁ·±¾Ç®¡£

Èý¡¢97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉúÊÇÒ»ÖÖ»ùÓÚNCCL£¨NVIDIA Collective Communications Library£©ÊµÏֵĶ˲àÁ´Â·¹ÊÕÏ¿ìËÙ¹æ±Ü¼Æ»®
£¬Äܹ»ÔÚÉϲãѵÁ·¿ò¼ÜÎÞ¸ÐÖªµÄÇéÐÎÏÂ
£¬×Ô¶¯¶Ô¶Ë²àÁ´Â·¹ÊÕϾÙÐÐʶ±ð²¢´¦Öóͷ£
£¬ÔÚÄ£×ÓѵÁ·Àú³ÌÖб¬·¢¶Ë²àÁ´Â·¹ÊÕÏʱÄܹ»¼á³ÖѵÁ·²»ÖÐÖ¹
£¬ÔÚ¹ÊÕÏ×°±¸Íê³ÉÐÞ¸´ºóѵÁ·ÐÔÄÜ×Ô¶¯»Ö¸´¡£

¶Ë²à¹ÊÕÏÌÓÉú¼Æ»®Í¨¹ýÔÚNCCL¼ÓÈë¹ÊÕÏʶ±ðÒÔ¼°¹ÊÕϹæ±Ü»úÖÆ
£¬²¢ÔÚ½¨ÉèÊý¾ÝÁ´Â·Ê±Í¬²½½¨×°±¸·ÝÁ´Â·À´´¦Öóͷ£ÓÉÓÚÓ²¼þ¹ÊÕϵ¼Öµ¥ÌõÁ´Â·ÎÞ·¨Í¨Ñ¶µÄÎÊÌâ¡£

±¸·ÝÁ´Â·ÔÚÖ÷Á´Â·×´Ì¬Õý³£Ê±²»»á¾ÙÐÐÊý¾Ý´«Êä
£¬¶Ô´«ÊäЧÂʲ»±¬·¢ÈκÎÓ°Ïì¡£¶øÔÚ·ºÆð¶Ë²àÁ´Â·¹ÊÕÏÖ®ºó
£¬Í¨¹ýһϵÁÐÇл»»úÖƽ«Ô­Á´Â·ÉϵÄͨѶʹÃüתÒƵ½±¸·ÝÁ´Â·¾ÙÐд«Êä¡£

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¼Æ»®¾ßÓÐÒÔÏÂÖ÷ÒªÌØÕ÷£º

1.·ÇÇÖÈëʽ

¸Ã¼Æ»®²»ÊÜÌض¨ÑµÁ·¿ò¼ÜµÄÏÞÖÆ
£¬Ò²ÎÞÐè¶ÔÉϲã¿ò¼Ü¾ÙÐÐÐ޸ġ£¹ÊÕϱ¬·¢ºó
£¬ÓÉNCCL×ÔÐд¦Öóͷ£
£¬Éϲã¿ò¼ÜÎÞ¸ÐÖª¡£

2.¿ìËÙÇÒ¿É¿¿

Äܹ»¶Ô¶Ë²àÁ´Â·¹ÊÕϾÙÐкÁÃ뼶µÄʶ±ðÓ붨λ
£¬²¢¶Ô¹ÊÕÏÁ´Â·ÉϵÄͨѶʹÃü¾ÙÐпìËÙǨáã
£¬¿ÉʵÏÖÃ뼶ÊÕÁ²
£¬²¢ÄÜ°ü¹ÜͨѶʹÃü²»ÖÐÖ¹¡£

3.¿É»Ö¸´ÐÔ

ʵʱ¼à²â×°±¸×´Ì¬
£¬Èô¹ÊÕÏ×°±¸×´Ì¬»Ö¸´Õý³£
£¬Äܹ»ÔÚ»Ö¸´µÄ×°±¸ÉÏÖØÐÞͨѶÁ´Â·
£¬²¢½«ÆäÖØÐÂÄÉÈëͨѶװ±¸¼¯Èº¡£

97¹ú¼ÊÍøÂçµÄ¶Ë²àÁ´Â·¹ÊÕÏÌÓÉú¼Æ»®¾ßÓÐÒÔϼ¸µãÒªº¦ÊÕÒ棺

1.ÌáÉýϵͳ¿É¿¿ÐÔ

ʹÓö˲à¹ÊÕÏÌÓÉú¼Æ»®Äܹ»ÓÐÓÃ×èÖ¹Òò¶Ë²àÍø¿¨¹âĤÏßÀµÈ×°±¸±¬·¢¹ÊÕ϶øÒýÆðµÄѵÁ·ÓªÒµÖÐÖ¹
£¬ÊµÏÖ¶ÏÁ´Ò»Ö±Ñµ
£¬Í¨Ñ¶µÄ¿É¿¿ÐÔÌáÉý10±¶¡£

2.½µµÍѵÁ·±¾Ç®

Äܹ»ÓÐÓÃ×èֹѵÁ·¼¯ÈºÒò¶Ë²àÁ´Â·¹ÊÕ϶øµ¼ÖµÄËãÁ¦×ÊÔ´ÆÌÕÅ
£¬Í¬Ê±»ùÓÚ×Ô¶¯¹ÊÕϻָ´´¦Öóͷ£»úÖÆÈ·±£ÍøÂç×°±¸ÊµÏÖ×î´óʹÓÃÂÊ
£¬´Ó¶øÏÔÖø½µµÍѵÁ·±¾Ç®¡£ ÒÀ¾ÝMetaÐû²¼µÄѵÁ·ÈÕÖ¾£¨Llama 3.1
£¬405BÄ£×Ó£©
£¬ÔÚΪÆÚ54ÌìµÄԤѵÁ·½×¶Î
£¬¹²·ºÆð419´ÎÒâÍâÖÐÖ¹ÇéÐÎ
£¬ÆäÖÐÔ¼8.4%ÊÇÓÉÍøÂç¹ÊÕÏËùÖ¡£H100µÄËãÁ¦½¨ÉèÓöÈԼΪ10Ôª/¿¨/Сʱ
£¬Èôÿ´ÎÖÐÖ¹»Ö¸´ÐèºÄʱ1Сʱ
£¬ÄÇôÍò¿¨¼¯ÈÔÃÄ¿ÎѵÁ·ÒòÍøÂç¹ÊÕÏÔì³ÉµÄËãÁ¦ËðʧԼΪ350ÍòÔª¡£Í¨¹ý¶Ë²àÁ´Â·¹ÊÕÏÌÓÉúÆøÖÆ
£¬Ã¿Äê¿ÉÍì»ØÉÏÍòÍòÔªµÄËðʧ¡£

×ܽá

´óÄ£×ÓÂþÑÜʽѵÁ·Àú³ÌÖеĶ˲àÍøÂç¹ÊÕÏ
£¬ÕâÊÇÓ°ÏìÄ£×ÓѵÁ·µÄÖØ´ó×è°­¡£¹¹½¨¾«×¼ÇÒ¸ßЧµÄ¹ÊÕÏ´¦Öóͷ£»úÖÆ
£¬ÊǸ÷´óÐÍÄ£×ÓѵÁ·ÍŶӵÄÒ»Á¬×·Çó
£¬Ò²ÊÇ°ü¹Ü´óÄ£×ÓѵÁ·µÄÖ÷Òª»·½Ú¡£

ͨ¹ýÔÚͨѶ¿âÖÐÌí¼Ó¹ÊÕÏ´¦Öóͷ£Ä£¿é
£¬Äܹ»´ó·ù¶ÈïÔÌ­¶Ë²àÁ´Â·¹ÊÕϵ¼ÖµÄËãÁ¦Ëðʧ¡£Ëæ׿ƻ®µÄÒ»Á¬µü´úÓëÍêÉÆ
£¬ÎÒÃǼáÐŶ˲à¹ÊÕÏÌÓÉú¼Æ»®Äܹ»ÔÚ´ó¹æÄ£¼¯ÈºÑµÁ·¹ÊÕÏ´¦Öóͷ£ÖÐÊ©Õ¹¸üΪÖ÷ÒªµÄ×÷ÓÃ
£¬Îª¿Í»§´´Ôì¸ü´óµÄ¼ÛÖµ¡£97¹ú¼ÊÍøÂç
£¬ÖÂÁ¦×ö×¶Ë²àµÄÍøÂ繩ӦÉÌ£¡