{"id":64393,"date":"2025-01-19T21:27:43","date_gmt":"2025-01-19T21:27:43","guid":{"rendered":"https:\/\/mailinvest.blog\/index.php\/2025\/01\/19\/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai\/"},"modified":"2025-01-19T21:28:47","modified_gmt":"2025-01-19T21:28:47","slug":"ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai","status":"publish","type":"post","link":"https:\/\/mailinvest.blog\/index.php\/2025\/01\/19\/ai-benchmarking-organization-criticized-for-waiting-to-disclose-funding-from-openai\/","title":{"rendered":"AI benchmarking organization criticized for waiting to disclose funding from OpenAI"},"content":{"rendered":"

<\/a>\n<\/p>\n

\n

A corporation creating math benchmarks for AI didn\u2019t disclose that it had obtained funding from OpenAI till comparatively not too long ago, drawing allegations of impropriety from some within the AI group.<\/p>\n

Epoch AI, a nonprofit primarily funded by Open Philanthropy, a analysis and grantmaking basis, revealed on December 20 that OpenAI had supported the creation of FrontierMath. FrontierMath, a take a look at with expert-level issues designed to measure an AI\u2019s mathematical expertise, was one of many benchmarks OpenAI used to demo its upcoming flagship AI, o3<\/a>.<\/p>\n

In a post<\/a> on the discussion board LessWrong, a contractor for Epoch AI going by the username \u201cMeemi\u201d says that many contributors to the FrontierMath benchmark weren\u2019t knowledgeable of OpenAI\u2019s involvement till it was made public. <\/p>\n

\u201cThe communication about this has been non-transparent,\u201d Meemi wrote. \u201cIn my opinion Epoch AI ought to have disclosed OpenAI funding, and contractors ought to have clear details about the potential of their work getting used for capabilities, when selecting whether or not to work on a benchmark.\u201d<\/p>\n

On social media, some<\/a> users<\/a> raised considerations that the secrecy might erode FrontierMath\u2019s status as an goal benchmark. Along with backing FrontierMath, OpenAI had entry to most of the issues and options within the benchmark \u2014 a reality Epoch AI didn\u2019t disclose previous to December 20, when o3 was introduced.<\/p>\n

In a reply to Meemi\u2019s submit, Tamay Besiroglu, affiliate director of Epoch AI and one of many group\u2019s co-founders, asserted that the integrity of FrontierMath hadn\u2019t been compromised, however admitted that Epoch AI \u201cmade a mistake\u201d in not being extra clear. <\/p>\n

\u201cWe have been restricted from disclosing the partnership till across the time o3 launched, and in hindsight we should always have negotiated tougher for the flexibility to be clear to the benchmark contributors as quickly as attainable,\u201d Besiroglu wrote. \u201cOur mathematicians deserved to know who may need entry to their work. Although we have been contractually restricted in what lets say, we should always have made transparency with our contributors a non-negotiable a part of our settlement with OpenAI.\u201d<\/p>\n

Besiroglu added that whereas OpenAI has entry to FrontierMath, it has a \u201cverbal settlement\u201d with Epoch AI to not use FrontierMath\u2019s downside set to coach its AI. (Coaching an AI on FrontierMath could be akin to teaching to the test<\/a>.) Epoch AI additionally has a \u201cseparate holdout set\u201d that serves as an extra safeguard for unbiased verification of FrontierMath benchmark outcomes, Besiroglu stated.<\/p>\n

\u201cOpenAI has \u2026 been totally supportive of our determination to keep up a separate, unseen holdout set,\u201d Besiroglu wrote. <\/p>\n

Nonetheless, muddying the waters, Epoch AI lead mathematician Ellot Glazer noted in a post on Reddit<\/a> that Epoch AI hasn\u2019t be capable to independently confirm OpenAI\u2019s FrontierMath o3 outcomes.<\/p>\n

\u201cMy private opinion is that [OpenAI\u2019s] rating is legit (i.e., they didn\u2019t prepare on the dataset), and that they haven’t any incentive to lie about inside benchmarking performances,\u201d Glazer stated. \u201cNonetheless, we are able to\u2019t vouch for them till our unbiased analysis is full.\u201d<\/p>\n

The saga is yet<\/a> another<\/a> example<\/a> of the problem of creating empirical benchmarks to judge AI \u2014 and securing the required assets for benchmark growth with out creating the notion of conflicts of\u00a0curiosity.<\/p>\n<\/div>\n