Ëæ×ÅÈ˹¤ÖÇÄÜÊÖÒյķÉËÙÉú³¤
£¬´óÓïÑÔÄ£×Ó£¨LLM£©ÒѳÉΪ×ÔÈ»ÓïÑÔ´¦Öóͷ£ÁìÓòµÄÈÈÃÅ»°Ìâ¡£¶øÔÚÕⳡÊÖÒÕˢеÄÀ˳±ÖÐ
£¬MoE£¨Mixture of Experts£©ÊÖÒÕÒÀ¸½ÆäÆæÒìµÄÓÅÊÆ
£¬ÎªLLMѵÁ·×¢ÈëÁËеĻîÁ¦¡£È»¶ø
£¬ÔÚÏÖʵӦÓð²ÅÅÖÐ
£¬MoE ÊÖÒÕ¶ÔÖÇËãÖÐÐÄÍøÂç´øÀ´²»Ð¡µÄÌôÕ½¡£±¾ÎÄÖ÷ÒªÏÈÈÝÁËMoEÊÖÒյĽµÉúÅä¾°Ô­Àí¡¢Ó¦ÓÃÁìÓò
£¬ÌÖÂÛÁËÆä¶ÔÖÇËãÖÐÐÄÍøÂç´øÀ´µÄÌôÕ½¡£

MoE½µÉúÅä¾°

MoEÊÖÒÕ
£¬¼´“Mixture of Experts”£¨×¨¼Ò»ìÏýϵͳ£©
£¬ÆðÔ´ÓÚ1991ÄêMichael I. JordanºÍRobert A. JacobsËù׫дµÄÂÛÎÄ¡¶Hierarchical Mixtures of Experts and the EM Algorithm¡·¡£ÕâƪÔÚÉñ¾­ÐÅÏ¢´¦Öóͷ£ÏµÍ³£¨Neural Information Processing Systems
£¬¼ò³ÆNIPS£©¾Û»áÉϽÒÏþµÄÂÛÎÄ
£¬±»¹«ÒÔΪ»úеѧϰÁìÓòµÄÖ÷ÒªÀï³Ì±®¡£

ÔÚ¸ÃÂÛÎÄÖÐ
£¬JordanºÍJacobsÏÈÈÝÁËÒ»ÖÖÐÂÓ±µÄÉñ¾­ÍøÂç½á¹¹——“ÌõÀí»¯×¨¼Ò»ìÏýϵͳ”Hierarchical Mixtures of Experts£©¡£Æä½¹µãÍ·ÄÔÔÚÓÚ½«ÖØ´óʹÃüϸ·ÖΪ¶à¸ö×ÓʹÃü
£¬²¢»®·Ö½»ÓÉÌض¨µÄ“ר¼ÒÄ£×Ó”¾ÙÐд¦Öóͷ£¡£ÕâÖּܹ¹¼¯³ÉÁ˶à¸öר¼ÒÄ£×ÓÒÔ¼°Ò»¸öÃÅ¿ØÄ£×Ó
£»ÃÅ¿ØÄ£×ÓÈÏÕæƾ֤ÊäÈëµÄÊý¾ÝÌØÕ÷
£¬Ñ¡Ôñ×îÊʺϵÄר¼ÒÄ£×ÓÀ´Ó¦¶ÔÌض¨Ê¹Ãü¡£

MoEÊÖÒÕµÄÎÊÊÀΪ»úеѧϰºÍÉî¶ÈѧϰÁìÓò×¢ÈëÁËеĻîÁ¦
£¬ÌØÊâÊÇÔÚ´¦Öóͷ£°üÀ¨¶à¸ö×ÓÎÊÌâ»òÐèÒªÈں϶àÁìÓò֪ʶµÄÖØ´óʹÃüʱ
£¬MoEÊÖÒÕÕ¹ÏÖ³öÁËÆä׿ԽµÄÐÔÄܺÍÆæÒìµÄÓÅÊÆ¡£

MoEÊÖÒÕÔ­Àí

MoEµÄÊÖÒÕÔ­Àí»ùÓÚÒ»ÖÖ·Ö¶øÖÎÖ®µÄÕ½ÂÔ
£¬½«ÖØ´óÎÊÌâÆÊÎöΪ¶à¸ö½Ï¼òÆÓµÄ×ÓÎÊÌâ
£¬²¢ÎªÃ¿¸ö×ÓÎÊÌâѵÁ·Ò»¸öרÃŵÄÄ£×Ó£¨×¨¼Ò£©¡£ÕâЩר¼ÒÄ£×Ó¸÷×ÔÈÏÕæ´¦Öóͷ£ÊäÈëÊý¾ÝµÄÒ»¸ö×Ó¼¯»òÒ»ÖÖÌض¨µÄÇéÐΡ£ÎªÁ˶¯Ì¬µØÕûºÏÕâЩר¼ÒµÄÊä³ö²¢ÌìÉú×îÖÕÕ¹Íû»ò¾öÒé
£¬MoEÒýÈëÁËÒ»¸öÃÅ¿ØÍøÂ磨Gating Network£©¡£

1£©×¨¼ÒÄ£×Ó£º

È·¶¨×¨¼ÒµÄÊýÄ¿ºÍÀàÐÍ¡£Ã¿¸öר¼Òͨ³£ÊÇÒ»¸ö½ÏСµÄÉñ¾­ÍøÂç
£¬×¨ÃÅÓÃÓÚ´¦Öóͷ£Êý¾ÝµÄÒ»¸ö×Ó¼¯»òÒ»¸öÌض¨µÄʹÃü¡£×¨¼ÒÍøÂçµÄÉè¼Æ¿ÉÒÔƾ֤ʹÃüÐèÇó¾ÙÐж¨ÖÆ
£¬ÀýÈç
£¬Ê¹ÓÃÈ«ÅþÁ¬²ã¡¢¾í»ý²ã¡¢Ñ­»·²ãµÈ¡£

2£©ÃÅ¿ØÄ£×Ó£º

ÃÅ¿ØÄ£×ÓÊÇÒ»¸öÓÃÓÚ¾öÒéÄÄЩר¼ÒÓ¦¸Ã¼ÓÈ뵽Ŀ½ñÊäÈëÊý¾Ý´¦Öóͷ£ÖеĻúÖÆ¡£Ëüͨ³£Ò²ÊÇÒ»¸ö½ÏСµÄÉñ¾­ÍøÂç
£¬ÆäÊä³öÊÇÒ»¸ö¸ÅÂÊÂþÑÜ
£¬ÌåÏÖÿ¸öר¼Ò¶ÔÄ¿½ñÊäÈëµÄÖ÷ÒªÐÔ¡£ÃÅ¿ØÄ£×ÓµÄÊäÈë¿ÉÒÔÊÇԭʼÊäÈëÊý¾ÝµÄÒ»¸ö×Ó¼¯»òËùÓÐ
£¬»òÕßÊÇר¼ÒÄ£×ÓµÄÖÐÐÄÊä³ö¡£

ÊÖÒÕÔ­Àí

µ±Ò»¸öÊäÈëÊý¾Ý½øÈëMoEϵͳʱ
£¬Ê×ÏÈͨ¹ýÃÅ¿ØÍøÂçÅÌËã³öÿ¸öר¼ÒµÄ¼¤»î¸ÅÂÊ¡£È»ºó
£¬Ã¿¸öר¼ÒÍøÂç»ùÓÚÊäÈëÊý¾Ý¾ÙÐÐÇ°ÏòÈö²¥
£¬±¬·¢Êä³ö¡£

½«ËùÓÐר¼ÒµÄÊä³öƾ֤ÃÅ¿ØÍøÂçµÄ¼¤»î¸ÅÂʾÙÐмÓȨ×éºÏ
£¬»ñµÃ×îÖÕµÄÄ£×ÓÊä³ö¡£ÕâÖÖ×éºÏ¿ÉÒÔÊǼòÆӵļÓȨƽ¾ù
£¬Ò²¿ÉÒÔÊǸüÖØ´óµÄ×éºÏÕ½ÂÔ¡£

MoEÊÖÒÕÓ¦ÓÃÁìÓò

ÏÖÔÚMoEÊÖÒÕÔÚÖî¶à³¡¾°ÖлñµÃÁËÆÕ±éµÄÓ¦Óð²ÅÅ£º

1£©×ÔÈ»ÓïÑÔ´¦Öóͷ££ºÔÚ»úе·­Òë¡¢ÇéÐ÷ÆÊÎöµÈ×ÔÈ»ÓïÑÔ´¦Öóͷ£Ê¹ÃüÖÐ
£¬MoE¿ÉÒÔ¼¯³É¶à¸öÄ£×Ó
£¬´Ó¶øÌá¸ßÎı¾Ã÷È·ºÍÌìÉúµÄÖÊÁ¿ºÍ׼ȷÐÔ¡£

2£©Í¼Ïñʶ±ðºÍÅÌËã»úÊÓ¾õ£ºÔÚͼÏñ·ÖÀà¡¢ÎïÌå¼ì²âºÍͼÏñÌìÉúµÈÅÌËã»úÊÓ¾õʹÃüÖÐ
£¬MoEÄܹ»ÍŽá¶à¸öר¼ÒÄ£×ÓµÄÌصã
£¬ÌáÉýÄ£×Ó¶ÔͼÏñµÄ±íÕ÷ºÍÃ÷È·ÄÜÁ¦¡£

3£©ÍƼöϵͳ£ºÔÚ¸öÐÔ»¯ÍƼöºÍ¹ã¸æͶ·ÅµÈÁìÓò
£¬MoE¿ÉÒÔ½«¶à¸öÍƼöÄ£×Ó×éºÏÆðÀ´
£¬Ìṩ¸ü׼ȷºÍ¸öÐÔ»¯µÄÍƼöЧ¹û
£¬Ìá¸ßÓû§Öª×ã¶ÈºÍÉÌÒµ¼ÛÖµ¡£

MoEÊÖÒÕ¶ÔÍøÂçµÄÌôÕ½

×ÝÈ»MoEÊÖÒÕÓÐÖî¶àÊÖÒÕÓÅÊÆ
£¬µ«ÏÖʵӦÓð²ÅÅÆðÀ´¶ÔÖÇËãÖÐÐÄÍøÂçÕվɱ£´æ²»Ð¡µÄÌôÕ½¡£

ÓÉÓÚר¼ÒÄ£×ӵIJ¢Ðл¯ºÍÂþÑÜʽѵÁ·µÄÐèÇó
£¬Ã¿¸öר¼Ò¿ÉÄÜÐèÒª´¦Öóͷ£ËùÓÐÊäÈëÊý¾ÝµÄÒ»²¿·Ö
£¬²¢ÇÒËüÃǵÄÊä³öÐèÒª±»»ã×ÜÒÔÌìÉú×îÖÕЧ¹û
£¬ÕâÖÖÐÅÏ¢½»Á÷µÄģʽ¾Í»áÒýÈë»ú¼äall-to-allµÄͨѶ¡£

ÒýÈëall-to-allͨѶµÄÏêϸԵ¹ÊÔ­ÓÉÓÐÒÔϼ¸µã£º

1£©×¨¼Ò¼äµÄÊý¾Ý½»Á÷£ºÔÚMoEϵͳÖÐ
£¬Ã¿¸öר¼Ò¿ÉÄÜÐèÒª»á¼ûÀ´×ÔÆäËûר¼ÒµÄÐÅÏ¢
£¬ÒÔ±ãÃÅ¿ØÍøÂç¿ÉÒÔ¾öÒéÿ¸öר¼ÒµÄ¼¤»îˮƽ¡£Õâͨ³£Éæ¼°µ½ËùÓÐר¼ÒÖ®¼äµÄÊý¾Ý½»Á÷
£¬¼´all-to-allͨѶ¡£

2£©²¢ÐÐѵÁ·£ºÔÚÂþÑÜʽѵÁ·ÇéÐÎÖÐ
£¬²î±ðµÄר¼Ò¿ÉÄܱ»°²ÅÅÔÚ²î±ðµÄÅÌËã½ÚµãÉÏ¡£ÎªÁËѵÁ·Ð§ÂÊ
£¬Ã¿¸ö½ÚµãÉϵÄר¼ÒÐèÒª¿ìËÙ½»Á÷Êý¾Ý
£¬ÕâÒ²ÐèҪͨ¹ýall-to-allͨѶÀ´ÊµÏÖ¡£

3£©Ä£×Ó²¢ÐÐÐÔ£ºµ±×¨¼ÒÄ£×Ó±äµÃºÜÊÇ´óʱ
£¬¿ÉÄÜÐèÒª½«Ò»¸öר¼ÒÄ£×Ó²ð·Öµ½¶à¸öÅÌËã½ÚµãÉÏ¡£ÕâÑù
£¬Í³Ò»¸öר¼ÒÄ£×ӵĸ÷¸ö²¿·ÖÐèÒªÔÚ½ÚµãÖ®¼ä¾ÙÐÐͨѶ
£¬ÒÔ¼á³ÖÄ£×ÓµÄÒ»ÖÂÐÔºÍ׼ȷÐÔ¡£

4£©ïÔÌ­ÅÌËãÆ¿¾±£ºall-to-allͨѶ¿ÉÒÔ×ÊÖúïÔÌ­µ¥¸ö½ÚµãµÄÅÌËãÆ¿¾±
£¬ÓÉÓÚËüÔÊÐí²¢Ðд¦Öóͷ£ºÍ½»Á÷Êý¾Ý¡£ÕâÖÖͨѶģʽ¿ÉÒÔ×î´ó»¯µØʹÓÃÅÌËã×ÊÔ´
£¬Ìá¸ßѵÁ·Ð§ÂÊ¡£

È»¶ø
£¬all-to-allͨѶҲ´øÀ´ÁËһЩÌôÕ½
£¬ÈçÔöÌíÍøÂç´ø¿íµÄÐèÇó¡¢Í¨Ñ¶ÑÓ³ÙºÍDZÔÚµÄÍøÂçÓµÈû¡£Òò´Ë
£¬ÔÚÏÖʵӦÓÃÖÐ
£¬ÐèÒª×ÐϸÉè¼ÆͨѶսÂÔºÍÓÅ»¯ÍøÂçÍØÆË
£¬ÒÔÈ·±£¸ßЧµÄÂþÑÜʽѵÁ·ºÍÍÆÀí¡£

²»ÒýÈëMoEʱµÄͨѶÊý¾ÝÁ¿

ÒýÈëMoEºóµÄͨѶÊý¾ÝÁ¿

ͨ¹ýÍøÉÏ»ñÈ¡µ½µÄ¹ûÕæÐÅÏ¢¶ÔGPT4¾ÙÐн¨Ä£
£¬ÔÚÒýÈë16¸öMoEר¼Ò²ãºó
£¬»ú¼äͨѶÊý¾ÝÁ¿Óë»úÄÚͨѶÊý¾ÝÁ¿»ù±¾³Öƽ
£»¶øÔÚ²»ÒýÈëMoEר¼Òʱ
£¬»ú¼äͨѶÊý¾ÝÁ¿ÊÇԶСÓÚ»úÄÚͨѶÊý¾ÝÁ¿µÄ¡£ÓÉ´Ë¿ÉÖª£ºMoEÊÖÒÕµÄÒýÈëÔöÌíÁË»ú¼äͨѶµÄ±ÈÀý
£¬»ú¼äͨѶµÄÍøÂç´ø¿íÐÔÄܱäµÃÓú·¢Ö÷Òª
£¬AIGCÖÇËãÖÐÐÄÍøÂçµÄ½¨ÉèÒ²³ÉΪ¸÷È˹Ø×¢µÄÖصã¡££¨¡¶ITÓ°ÏìÖйú2023£º97¹ú¼ÊAI-FlexiForceÖÇËãÖÐÐÄÍøÂç½â¾ö¼Æ»®ÈÙ»ñÓ°ÏìÁ¦½â¾ö¼Æ»®½±¡·£©

×ܽá

MoEר¼Ò»ìÏýϵͳÊÖÒÕÏÖÔÚÒѱ»ÆÕ±éÓ¦ÓÃÓÚLLMѵÁ·³¡¾°ÖÐ
£¬¸ÃÊÖÒÕÄܹ»ÏÔÖøÔöÇ¿Ä£×ӵķº»¯ÄÜÁ¦²¢ÌáÉýÔËËãЧÂÊ¡£È»¶ø
£¬ÕýÈçÿ¸öÊÖÒÕ¶¼ÓÐÆäÁ½ÃæÐÔÒ»Ñù
£¬MoEÊÖÒÕÔÚ´øÀ´Öî¶àÀûÒæµÄͬʱ
£¬Ò²ÒýÈëÁË´ó×Ú»ú¼äall-to-allµÄͨѶÐèÇó
£¬ÕâÎÞÒɶԻú¼äÍøÂçÐÔÄÜ×é³ÉÁËÑÏËàÌôÕ½¡£

ΪÁËÓ¦¶ÔÕâÒ»ÌôÕ½
£¬97¹ú¼ÊÍøÂçͨ¹ý´î½¨¸ßÐÔÄܵÄÖÇËãÖÐÐÄÍøÂç
£¬ÌáÉý»ú¼äÍøÂçµÄ´ø¿íʹÓÃÂÊÀ´È·±£ÓªÒµµÄ¸ßÍÌÍÂÄÜÁ¦
£¬»º½âÓÉMoEÊÖÒÕÒýÈëµÄ´ó×ÚͨѶÐèÇó¶ÔÍøÂçÐÔÄÜÔì³ÉµÄѹÁ¦¡£

ÔÚÈ«Çò»¥ÁªÍøÁ÷Á¿Ò»Ö±ÔöÌíºÍÊý¾ÝÓ¦ÓÃÐèÇóÈÕÒæ¶àÑù»¯µÄÅä¾°ÏÂ
£¬×÷ΪAIGCÈ«Õ»·þÎñר¼Ò
£¬97¹ú¼ÊÍøÂç²»µ«ÖÂÁ¦ÓÚÍƶ¯ÍøÂçÊÖÒÕµÄÇ°½øºÍÉú³¤
£¬¸üÊdzмÌÁ¢ÒìÐԵؽâ¾ö¿Í»§ÎÊÌâ
£¬½ÏÔçÍƳöAIGCÖÇËãÖÐÐÄÍøÂçÕûÌå¼Æ»®·þÎñ¿Í»§¡£Õ¹ÍûδÀ´
£¬Í¨¹ýÒ»Á¬µÄÊÖÒÕÑз¢ºÍ²úÆ·Á¢Òì
£¬97¹ú¼ÊÍøÂ罫¼ÌÐøΪȫÇòµÄÊý¾ÝÖÐÐÄÌṩԽ·¢¸ßЧ¡¢¿É¿¿¡¢ÖÇÄܵÄÍøÂç½â¾ö¼Æ»®
£¬ÔÚAIGCʱ´ú
£¬ÖúÁ¦»¥ÁªÍøÆóÒµ¼°¸÷Ðи÷ÒµµÄ¿ìËÙÉú³¤¡£