I'm happy to see I'm actually very much aligned with one of the "Attention Is All You Need" co-authors. The current industry trend of "just scale the transformer architecture" is indeed stifling innovation and actual research. That said I find ironic that he talks about freedom to explore... well this is what public labs used to be about, but we decided to drastically reduce their funding and replace that with competition between startups. It's no surprise we have very myopic views on problems.