Frontier AI has broken the open CTF format

(kabir.au)

64 points | by frays 1 hour ago

16 comments

  • himata4113 28 minutes ago
    I was writing an obfuscator recently, I just had the model deobfuscate and optimize the code back to original and I kept improving the obfuscator until it couldn't. The funny thing is that after all this I also ended up with a really strong deobfuscator and optimizer which is probably more capable than most commercial tools.

    The solution is just to make CTFs harder, but when do CTFs become too hard? Maybe the problem is that 'hard' CTFs are fundementally too 'simple' where it's just a logic chain and an exhaustive bruteforce towards a solution since there really are limited ways to express a solution in plain sight.

    Or maybe human creativity has been exhausted and we're not so limitless as we thought. Only time will tell.

    I had another idea spring to mind: we could hide two flags, one that could only be found by ai agents and not humans or tools written by humans.

    • koolala 11 minutes ago
      A portion could require astral projection and computers can't do that. Or maybe just a VR mini-game like the 90s always imagined.
      • himata4113 1 minute ago
        bringing CTF solutions into the real world is a really good idea! I didn't even think of this until you mentioned it.
  • amingilani 28 minutes ago
    I don’t think CTFs are dead, they’ll just evolve. The difficulty level will need to be increased or the rules locked down. Just like sports and racing persist despite the existence of performance enhancing drugs and rocket technology.

    I just did a CTF where I was in the top 10. It was the first CTF I completed and I used AI because the rules permitted it. That said, I couldn’t solve all challenges.

    But yes, it was significantly easier now than I last attempted one. Even manually solving with AI assisted assembly interpretation was much easier.

    • mort96 22 minutes ago
      Increasing the difficulty level is a terrible solution. The problem with CTFs isn't that they're too easy. Making them harder just makes them even less accessible to people who don't cheat. It'd be like seeing people who put hidden electric motors in their bikes during Tour de France and conclude, "oh we just need longer distances and steeper hills".
  • rurban 28 minutes ago
    I don't do CTF's but took part at the security workshop for fun ~2 years with my Android phone only. I was first with the first simple challenge, but then couldnt continue because my phone was just too limited. But I watched what the others did. And a young Indian guy did everything with ChatGPT then. I found it silly, but amusing, because he actually got second. There was no Codex nor Claude then. Nowadays it must be dead for real, because I would solve everything with my agents, as I do in the real world.
  • susam 41 minutes ago
    I have normally found any sort of timed technical competition intimidating. Even so, about 6 or 7 years ago, after being persuaded by a colleague, I participated in a few CTFs. I am glad I did, back when this type of thing still meant something. I have kept a screenshot from one of the CTFs that I am quite fond of: https://susam.net/files/blog/ctf-2019.png
  • kevinsimper 29 minutes ago
    You could make it offline and with provided laptops only, just like with the competitive CS2 scene.
    • sheept 4 minutes ago
      Offline CTFs could also incorporate physical security challenges, like lockpicking
    • hsbauauvhabzb 14 minutes ago
      Ctfs need preparation and unconstrained internet, even if you block domains it’s possible to tunnel out
      • sheept 2 minutes ago
        Presumably if you block domains, you wouldn't be able to use AI to find a way around the block. So doing so demonstrates at least some human skill
      • belabartok39 5 minutes ago
        Use jumpbox to access CTF. Disable all wireless for the playing hall.
    • eastbound 13 minutes ago
      Since real-life situations involve AI, banning AI would make CTFs just a simple game, not a demonstration of capabilities and talent.
      • mort96 8 minutes ago
        What do you mean? Solving a CTF challenge demonstrates way more capabilities and talent than just asking a chat bot to solve a CTF challenge.
      • loeg 9 minutes ago
        They always were just a game?
  • raphman 25 minutes ago
    Interesting and well written article that mirrors/foreshadows how LLMs do and will change other scenes.

    As I don't know much about the CTF scene, I looked for other takes on this topic.

    Here's an article from 2015 about how tool-assistance already changed CTFs:

    > Individual skill will undoubtedly be a factor next year. But, I'm left wondering whether next year's DEFCON CTF will tell us anything more than how well-developed each team's tools are (and how well they can interpret the results).

    https://fuzyll.com/2015/ctf-is-dead-long-live-ctf/

    But there are quite a few recent (2026) articles with the same core message as in the original article, e.g., https://blog.includesecurity.com/2026/04/ctfs-in-the-ai-era/ or https://k3ng.xyz/blog/ctf-is-dead

    And here's someone explaining how Claude Max allowed them to win CTFs:

    > I had always been interested in CTF as one of the only ways people could compete and show off their skill in coding/problem solving on a global scale. It was just too difficult and didn't make sense for me to learn the fundamentals as an electrical engineer. As time went on, I got better and better, and it was hard to tell whether it was because of experience or if it was because of improvements in AI.

    > I accomplished my goals, and for that reason I'm quitting CTF, at least for now. [...] I'd like to think I highlighted the problem before it became a bigger issue. So, how do we fix this? Teams and challenge authors losing motivation is not good. CTF dying is not good. AI bad. Or is it?

    https://blog.krauq.com/post/ctf-is-dying-because-of-ai

    The only article that saw LLMs as a non-negative force for CTFs was this one. Fittingly, it sounds like LLM output ("Let's be honest", "This is where things get interesting.") and only contains hallucinated references.

    https://caverav.cl/posts/ctfs-not-dead/ctfs-not-dead/

  • tromp 45 minutes ago
    https://en.wikipedia.org/wiki/Capture_the_flag_(cybersecurit...

    still has no mention of AI, but that will likely change as they increasingly dominate competition.

  • Grimburger 28 minutes ago
    Very impressed that OP has gone from starting university in 2021 to becoming a Senior Security Engineer.

    It's an incredibly exciting time in security research in my humble old man opinion.

    Think the cadence of new exploits is perhaps a good measure of that rather than subjective thoughts by anyone regardless of experience.

  • eecc 32 minutes ago
    “solve”, why not solution? Like “spend” and not expenditure, why use the verb as a noun and not care about grammar?
    • iainmerrick 26 minutes ago
      They’re shorter.

      Why so pedantic?

  • monarx 43 minutes ago
    used to see some really good CTF videos show up on youtube and now nothing like that shows up on the feed
  • walletdrainer 50 minutes ago
    >I started playing CTFs in 2021

    >and the old game is not coming back

    For many people the CTF scene was already dead in 2021 because it had turned into something unrecognisable.

    In reality it’s just different.

    • lukan 46 minutes ago
      Well, I had to google what CTF means (capture the flag, a hacking competition), so surely cannot judge here, but the text indicates that with AI some things are very different today:

      "That makes open CTFs pay-to-win. The more tokens you can throw at a competition, the faster you can burn down the board. Specialised cybersecurity models like alias1 by Alias Robotics are becoming less relevant compared to general frontier LLMs. The competition is turning into "who can afford to run enough agents, with enough context, for long enough.""

      • mock-possum 33 minutes ago
        Isn’t that the bitter lesson in a nutshell? “Specialised cybersecurity models … are becoming less relevant compared to general frontier LLMs.”
    • Grimburger 27 minutes ago
      >Learning about eternal September in May 2026

      Hits different doesn't it

  • vasco 44 minutes ago
    My first ever was Stripe CTF in 2012 I think, I still wear the shirt I got (now super fainted) from passing some challenges. I was a student in portugal and remember receiving the shirt for it and thinking, maybe those Americans aren't any better than me and I can compete at the same level.

    I never got super into security but it gave me the confidence to play in the same field and lose the stupid aura I had that somehow "rich americans" would be better than me at everything because they had better universities or because of Hollywood or something.

    Sad that another cool thing is lost to AI but I guess kids will learn in other ways.

  • deafpolygon 48 minutes ago
    Unrelated, but does anyone find this site incredibly hard to read?
    • walletdrainer 47 minutes ago
      Bizarre font and poor contrast, yep.

      The text itself being exceedingly long for no obvious reason doesn’t help.

      • lukan 43 minutes ago
        Poor contrast? White on black?

        And if you think it was too long, what part would you have shortened? I never knew about the scene and found it interesting to read this personal take on it.

  • utopiah 14 minutes ago
    Right, the same way that car racing has "broken" jogging. This is so dumb. /s

    The whole point of competitions is to provide a safe environment thanks to a set of rules all participants AGREE on in order to progress together.

    If new tools "break" the competition, we change the rules and that's A-OK.

    CTF isn't a natural phenomenon, if tools change, rules change, simple.

  • chvid 44 minutes ago
    What is CTF? And why is the cyber security world filled with silly gaming references?
    • mort96 33 minutes ago
      Capture The Flag is a cybersecurity game where the organizers set up a bunch of intentionally vulnerable computer systems with a "flag" on them, a string that's "supposed to be" secret but is accessible through exploiting the vulnerabilities. This may be a line in /etc/password, a string in memory, a field in a database, whatever. The goal of the game is to hack into the computer systems, find ("capture") the flag, then copy/paste it into the organiser's scoreboard website to prove that you solved that particular challenge.

      It's pretty fun. Or at least it was, back when you had some sense that your competitors were competing on an even playing field and just beat you because they were better than you.

      I wouldn't say the name is a "gaming reference", it's just a descriptive name for a game.

    • throwa356262 37 minutes ago