Looks like a promising way to reduce the training cost of large language models.

OK, this is a pre-print so to take with a truckload of salt. If further nice results get built up on this it could turn out interesting though. This is a much more intellectually satisfying approach than the current arm race of "let's throw bigger models at the problem". This has the potentially of reducing the computational complexity of those models, this is definitely welcome in term of energy and hardware requirements. Let's wait and see...

This was only a matter of time. It'll be interesting to see how this will unfold. Potentially it could turn into lawsuit cases being built up, it could also mean content producers get a cut down the line... of course could be both. Since FOSS code also ends up in training those models I'm even wondering if that could lead to money going back to the authors. We'll see where that goes.

This is impressive results. Clearly much less artifacts than on previous such models.

This is important. We need truly open generator models. This can't be left in the hands of a few with only API access, especially since they lack basic transparency.

Clearly aims to demonstrate the superiority of their specialized hardware for training. That said it's nice to have proper open models available (architecture, training data, weights... it's all in the open).

Now, this starts to become interesting. This is a first example of trying to plug symbolic and sub-symbolic approaches together in the wild. This highlights some limitations of this particular (quite a bit rough) approach, we'll see how far that can go before another finer approach is needed.

Now this is a properly balanced piece which looks beyond the hype. Usable yes, if hallucinations don't have a high impact. Can the hallucinations be solved? To be seen, I personally have my doubts with the current architecture... at least banking it all on human feedback is being very naive about the scale of the task.

This is an excellent piece. Very nice portrait of Emily M. Bender a really gifted computational linguist and really bad ass if you ask me. She's out there asking all the difficult questions about the current moment regarding large language models and so far the answers are (I find) disappointing. We collectively seem to be way too fascinated by the shiny new toy and the business opportunities to pay really attention to the impact on the social fabric of all of this.

Interesting strategy, shows a fascinating blind spot in the typical AIs used for Go nowadays. It kind of hints to the fact that the neural networks abstract knowledge much less than advertised.

Interesting work, trying to get back to the source material used by a generative model. This is definitely necessary as well.

A few interesting points in there. Too much hype and important points are glanced over, we'd all benefit from them being more actively explored.

Very nice summary of the architecture in the latest trend of transformer models. Long but comprehensive, a good way to start diving in the topic.

Such generative models are getting more and more accessible. You can play with them using a few lines of python now.

The human labor behind AI training is still on going. This is clearly gruesome and sent over to other countries... ignoring the price for a minute this is also a good way to hide its consequences I guess.

Interesting reverse engineering job of Copilot's client side to have a better idea at which information it actually feeds to the model. A couple of funny tricks to prepare the prompt are involved. Obviously some telemetry involved as well, again with interesting heuristics to try to figure out if the user kept the suggestion or not.

At least a good balanced post about Generative AI and programming. It's not overestimating abilities of the latest trend in large language models and moves away from the "I'll loose my job, developers will be replaced with AI" stance.

A few months old but a good piece to put things in perspective after the recent craze around large language models in general and GPT in particular. Noteworthy is the "wishful mnemonics" phrase mentioned and how it impacts the debate. Let's have less talks about AIs and more about SALAMIs please?

Nice article, gives a few clues to get a grasp on how GPT-3 works.

Words of caution regarding the use of language models for producing code. This can derail fairly quickly and earlier than you'd expect... without noticing it.