{"id":70,"date":"2025-04-22T13:16:04","date_gmt":"2025-04-22T13:16:04","guid":{"rendered":"https:\/\/michaelmaynord.net\/?p=70"},"modified":"2025-04-22T13:34:55","modified_gmt":"2025-04-22T13:34:55","slug":"leap-llm-generation-of-egocentric-action-programs","status":"publish","type":"post","link":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/","title":{"rendered":"LEAP: LLM-Generation of Egocentric Action Programs"},"content":{"rendered":"\n<p>We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual, and structural aspects of action, and consist of sub-actions, pre- and<br>post-conditions, and control flows. LEAP\u2019s action programs are centered on egocentric video and employ recent developments in LLMs both as a source for program knowledge<br>and as an aggregator and assessor of multimodal video information. We apply LEAP over a majority (87%) of thetraining set of the EPIC Kitchens dataset, and release theresulting action programs as a publicly available datasethere. We employ LEAP as a secondary source of supervision, using its action programs in a loss term applied to action recognition and anticipation networks. We demon-<br>strate sizable improvements in performance in both tasks due to training with the LEAP dataset. Our method achieves 1st place on the EPIC Kitchens Action Recognition leaderboard as of November 17 among the networks restricted to RGB-input. <\/p>\n\n\n\n<p>Read more here: <a href=\"https:\/\/arxiv.org\/pdf\/2312.00055\">https:\/\/arxiv.org\/pdf\/2312.00055<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual,&#8230; <\/p>\n","protected":false},"author":1,"featured_media":71,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-70","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-currentpublications"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v25.1 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord\" \/>\n<meta property=\"og:description\" content=\"We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual,...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/\" \/>\n<meta property=\"og:site_name\" content=\"Michael Maynord\" \/>\n<meta property=\"article:published_time\" content=\"2025-04-22T13:16:04+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-04-22T13:34:55+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png\" \/>\n\t<meta property=\"og:image:width\" content=\"436\" \/>\n\t<meta property=\"og:image:height\" content=\"346\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"admin\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"admin\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/\",\"url\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/\",\"name\":\"LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord\",\"isPartOf\":{\"@id\":\"https:\/\/michaelmaynord.net\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png\",\"datePublished\":\"2025-04-22T13:16:04+00:00\",\"dateModified\":\"2025-04-22T13:34:55+00:00\",\"author\":{\"@id\":\"https:\/\/michaelmaynord.net\/#\/schema\/person\/4032622d57ec607730c1f229bb9fe96c\"},\"breadcrumb\":{\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage\",\"url\":\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png\",\"contentUrl\":\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png\",\"width\":436,\"height\":346,\"caption\":\"Leap-LLM\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/michaelmaynord.net\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"LEAP: LLM-Generation of Egocentric Action Programs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/michaelmaynord.net\/#website\",\"url\":\"https:\/\/michaelmaynord.net\/\",\"name\":\"Michael Maynord\",\"description\":\"Research Scientist\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/michaelmaynord.net\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/michaelmaynord.net\/#\/schema\/person\/4032622d57ec607730c1f229bb9fe96c\",\"name\":\"admin\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/michaelmaynord.net\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2026\/03\/cropped-Transparent-Avatar-96x96.png\",\"contentUrl\":\"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2026\/03\/cropped-Transparent-Avatar-96x96.png\",\"caption\":\"admin\"},\"sameAs\":[\"https:\/\/michaelmaynord.net\"],\"url\":\"https:\/\/michaelmaynord.net\/index.php\/author\/robert_4gn3jyiv\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/","og_locale":"en_US","og_type":"article","og_title":"LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord","og_description":"We introduce LEAP (illustrated in Figure 1), a novel method for generating video-grounded action programs through use of a Large Language Model (LLM). These action programs represent the motoric, perceptual,...","og_url":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/","og_site_name":"Michael Maynord","article_published_time":"2025-04-22T13:16:04+00:00","article_modified_time":"2025-04-22T13:34:55+00:00","og_image":[{"width":436,"height":346,"url":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png","type":"image\/png"}],"author":"admin","twitter_card":"summary_large_image","twitter_misc":{"Written by":"admin","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/","url":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/","name":"LEAP: LLM-Generation of Egocentric Action Programs - Michael Maynord","isPartOf":{"@id":"https:\/\/michaelmaynord.net\/#website"},"primaryImageOfPage":{"@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage"},"image":{"@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage"},"thumbnailUrl":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png","datePublished":"2025-04-22T13:16:04+00:00","dateModified":"2025-04-22T13:34:55+00:00","author":{"@id":"https:\/\/michaelmaynord.net\/#\/schema\/person\/4032622d57ec607730c1f229bb9fe96c"},"breadcrumb":{"@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#primaryimage","url":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png","contentUrl":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2025\/04\/LEAP-LLM.png","width":436,"height":346,"caption":"Leap-LLM"},{"@type":"BreadcrumbList","@id":"https:\/\/michaelmaynord.net\/index.php\/2025\/04\/22\/leap-llm-generation-of-egocentric-action-programs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/michaelmaynord.net\/"},{"@type":"ListItem","position":2,"name":"LEAP: LLM-Generation of Egocentric Action Programs"}]},{"@type":"WebSite","@id":"https:\/\/michaelmaynord.net\/#website","url":"https:\/\/michaelmaynord.net\/","name":"Michael Maynord","description":"Research Scientist","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/michaelmaynord.net\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/michaelmaynord.net\/#\/schema\/person\/4032622d57ec607730c1f229bb9fe96c","name":"admin","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/michaelmaynord.net\/#\/schema\/person\/image\/","url":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2026\/03\/cropped-Transparent-Avatar-96x96.png","contentUrl":"https:\/\/michaelmaynord.net\/wp-content\/uploads\/2026\/03\/cropped-Transparent-Avatar-96x96.png","caption":"admin"},"sameAs":["https:\/\/michaelmaynord.net"],"url":"https:\/\/michaelmaynord.net\/index.php\/author\/robert_4gn3jyiv\/"}]}},"_links":{"self":[{"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/posts\/70","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/comments?post=70"}],"version-history":[{"count":1,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/posts\/70\/revisions"}],"predecessor-version":[{"id":72,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/posts\/70\/revisions\/72"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/media\/71"}],"wp:attachment":[{"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/media?parent=70"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/categories?post=70"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/michaelmaynord.net\/index.php\/wp-json\/wp\/v2\/tags?post=70"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}